There’s Information in Music and Music in Information

“A painter paints pictures on canvas but musicians paint their pictures on silence” – Leopold Stokowski

Sonification is defined as the use of non-speech audio to convey information. It takes advantage of people’s innate ability to detect subtle difference in sounds and perceive rhythms, patterns and short events simply by listening.

The idea is not a new one. When we think about it, we find that the concept is familiar. Sound has always conveyed information. The sound of a car’s horn or the growl of a dog alerts us to possible danger while the sound of waves lapping on the shore imparts a feeling of peace and relaxation. I write this sitting, literally, in the middle of an orchestra rehearsing the soundtrack to The Dark Knight. Through the composer’s choice of musical elements, his music conveys tension, mystery, danger and a feeling of dread.

Like visualization, in which elements of data are represented by the shapes and colours, sonification uses elements of music to represent data. These are: pitch, the quality that makes it possible to judge sounds as higher or lower; rhythm, the pattern of strong and weak beats and timbre, the quality of sound that allows us to identify the sound of a flute and distinguish it from a drum or a trumpet.

In the process of mapping that data to sound, we as humans get to make choices about how musical we want our sonification to be and how pleasing or displeasing we would like it to be to the listener. We can, through our choices of pitch, timbre and rhythm produce sonifications that, while still conveying information, can be as musical or un-musical as we choose.

Let’s use a simple example to illustrate this, the sonification of a fruit salad.

A fruit salad can be thought of as an arrangement of a selection of fruits in some particular order. This sonification was created by mapping the elements of the salad 1 apple, 1 banana, 9 cherries, and 18 grapes and 3 pears to notes played on a marimba. The duration and pitch of the notes represent the type of fruit involved with smaller fruit having shorter note lengths. Thus the apple is represented by the C above middle C a half note in 4/4 time duration, the banana is a whole note F above middle C, the 9 cherries are eighth note Gs above middle C, the 18 grapes are sixteenth note Bs above middle C and the 3 pears are a quarter note duration D above middle C. By choosing notes which will cover a medium range all within a C major scale a relatively cheerful tune emerges.

Rearranging those notes produces yet another, different, cheerful little tune representing another arrangement of these same ‘musical’ fruit.

Rearranging them again produces a third variation. Played together they sound like this:

Imagine time passing though and the fruit starting to rot … a cheerful little melody would no longer perhaps be the appropriate way of portraying the now not so appetizing fruit arrangement. By representing the amount of time passing by slowing the tempo, transposing the melody down a third and adding a clashing note to the cherries tone we can create a sonification that lets the listener hear the decaying of the fruit. Simply by listening you can know how fresh the fruit in that salad is.

Here is the sound of the not as fresh fruit salad:

There are endless possibilities for sonification. You could sonify the weather report and hear how cold or warm it is outside. You could sonify the traffic report to hear how busy the highway is before you leave to drive home. The possibilities are almost endless. However, if we expect people to embrace sonification as a legitimate mode of conveying information, we will have to exercise some control over our musical choices when we map the data and choose sonifications that both convey information as well as remaining ‘listenable’.

Katie Legere has degrees in music from Queen’s University and the University of Toronto as well as a Bachelor of Education and Master’s degrees in Computing Science and Business from Queen’s where she is a member of the Adjunct Faculty at the School of Music. Katie is a member of the Kingston Symphony playing bassoon and contrabassoon and has performed as part of the Toronto Symphony, National Arts Centre Orchestra, Violins Du Roy, and the Ohio Light Opera Company. She is a teacher and performer in Kingston and works as a Business Analyst on enterprise level projects for the Queen’s Project Portfolio Office.