This paper proposes complex emotion recognition system for a specific user, where complex emotion has mingled emotions. In order to show the differences between individuals, we use Self-Organizing Feature Map(SOM) for proposed system. Additionaly, in order that emotion recognition system expresses complex emotion, we propose new method for labeling. We verify proposed system using emotional speech database we recorded. As a result of verifying, this system could express fuzziness emotions such as anger with sadness, and this paper showed the effectiveness for emotion recognition to specify the user.