Automated dialog systems are currently being used in various applications, but it is unclear if they will ever be able to converse as naturally as humans do. One challenge is avoiding breakdowns during conversations due to inappropriate system utterances. Although many studies have focused on dialog breakdown detection, the influence of differences among individual users on dialog breakdowns and breakdown detection has not been sufficiently examined. In this study, we focus on individual differences thought to be related to emotional responses after breakdowns, specifically language, acoustic, and facial features, as well as gender and BigFive personality traits, to analyze differences in user responses to breakdowns. Our results suggest that gender and personality traits influence user responses to dialog breakdowns. For example, users with low Openness scores were more likely to express anger, while women were less likely to do so.