Although the majority of previous research on music-induced responses has focused on pleasurable experiences and preferences, it is undeniable that music is capable of eliciting strong dislike and aversion as well. To date, only limited research has been carried out to understand the subjective experience of listening to aversive music. This qualitative study explored people’s negative experiences associated with music listening, with the aim to understand what kinds of emotions, affective states, and physical responses are associated with listening to aversive music. One hundred and two participants provided free descriptions of (1) musical features of aversive music; (2) subjective physical sensations, thoughts and mental imagery evoked by aversive music; (3) typical contexts where aversive music is heard; and (4) the similarities and/or differences between music-related aversive experiences and experiences of dislike in other contexts. We found that responses to aversive music are characterized by embodied experiences, perceived loss of agency, and violation of musical identity, as well as social or moral attitudes and values. Furthermore, two “experiencer types” were identified: One reflecting a strong negative attitude toward unpleasant music, and the other reflecting a more neutral attitude. Finally, we discuss the theoretical implications of our findings in the broader context of music and emotion research.