Computational Analysis of Melodic Contour and Body Movement

Abstract

The aim of this dissertation is to understand the role of embodiment in melodic contour perception. In other words, it studies how we move our bodies in response to music. Melodies play an important role in both speech and music. This thesis consists of two parts, the first part being a background section discussing the theoretical motivations and methods used. The second part is a collection of four articles. Each of the articles explores a dimension of melodic contour: verticality, motion metaphors, body use, and multi-feature correlational analysis. The empirical work is based on sound-tracing as an experimental paradigm. This brings together the multimodal mappings of pitched sound, gestural imagery evoked by these sounds, and defining geometries of these contours. Two sound-tracing experiments were conducted, resulting in three datasets that have been used in the analyses. In the experiments, participants listened to 16 melodies from four different genres: operatic vocalise, jazz scatting, North-Indian singing, and Sámi joik. The participants listened to each melody twice, the first time standing still, and the second time ``drawing'' the sounds in the air. Infrared motion capture was used to record the participants' body movement, and the analysis is focused primarily on the movement of their hands. The sound analysis is based on signal processing algorithms for pitch detection and methods for contour representation. Cross-correlation of the data is performed using a range of methods from statistical hypothesis testing to canonical correlation analysis. The analysis reveals that although there is a natural propensity to describe pitches in terms of the vertical dimension, the experimental data do not clearly show such an association. Average profiles of movement responses to melodies have an arch-like representation, regardless of the contour of the melody. Spatial height is associated more with relative pitch in the melodic context traced, rather than an absolute pitch scale. In addition, movement indicating metaphoric representation of sound was used more often by participants.