My intentions in this paper is to take the auditory hierarchy used in film and animation and apply it to instructional message design to determine if it can enhance narrative comprehension and presentation for the learner. My research will evaluate and quote various articles on instructional message design, the auditory hierarchy, and general principles of sound design used in film and animation.
Before any unification of the auditory hierarchy and instructional message design can exist, we must define the skill set of the sound designer, quoted by Elisabeth Weis in the article: Sync Tanks: The Art and Technique of Postproduction Sound (Cineaste, 1995) as:
“Sound designer – are an artist who are brought on staff during the planning stages of a film, along with the set and costume designers, and who do their own mixing. The sound designer works with the director to shape an overall, consistent soundtrack that exploits the expressive possibilities of the sound medium.” (Cineaste, 1995)
I wish to remove the idea of the skills of a sound designer connecting only with film and add multimedia and instructional message design to the definition. A modern day sound designer is someone who could use “sounds to enhance the narrative for traditional film, multimedia and instructional message design to enhance the learning experience.
The auditory hierarchy is a common film term used by sound designers to describe the importance of three major levels of sound mixed with visuals: dialogue, sound effects and music: “Sound is traditionally divided into three elements: dialog, music, and effects (any auditory information that isn’t speech or music). Although much of the dialog can be recorded during principal photography, if it needs fine tuning later.” (Cineaste, 1995)
Sound designers have to make many decisions to find an appropriate mix in the auditory hierarchy to enhance meaning and narrative between the various levels of sound. My research sets out to determine if use of the auditory hierarchy could enhance narrative comprehension and presentation value for the learner in instructional message design. My research draws on multiple sources of film, animation, and human interface design.
Instructional message design is defined and referenced in Introduction to Instructional Message Design (Sept.2008) by Ellen Rose as: “Instructional message design is a fundamental component of the overall instructional design process. It is concerned with presenting and structuring instructional content in a way that will promote learning.” Further dissection of the words instructional, message and design from Introduction to Instructional Message Design:
“Instruction refers to “a way of organizing and sequencing information for the learner” (Grabowski, 1995, p. 224), in order to promote comprehension and retention. According to David Merrill (1999), “Information that does not include presentation, practice, and learner guidance is information but not instruction” (p. 401)
Message refers to a pattern of signs (words, images, gestures, sounds) used to communicate.
Design refers to a deliberate process of planning and organizing for a particular purpose. The dictionary definition of design is “to prepare the plans for; to plan an object, work of art, etc.” Design is both a strategic undertaking involving careful thought and an artistic process; it therefore involves the informed application of research as well as intuition and creativity based on the designer’s experience.”
The first obvious connection with sound appears in the definition of “message”, but “instruction” could be greatly impacted with the use of sound to present an enhanced instructional message for the learner. The way an instructional message is presented can give all three levels of the auditory hierarchy a chance to impact the final design. In this article I will build a fictional instructional message and list ways the audio hierarchy can play an integral part in the presentation and narrative:
If I created an instructional message on the lions of Africa with the auditory hierarchy in mind; I would look at balancing sound effects, narration and music throughout the message to make sure it benefits the intended outcome for the learner. If the instructional message started as a text-based sheet that needed to become a multimedia instructional message, I could design a Keynote (or PowerPoint) slide presentation with lion pictures and text-based slides that appear between the pictures. I could export the presentation as a QuickTime file to have it as a self-contained movie for desktop or online use.
One of the best ways to bring a sense of temporal and spatial depth to a two-dimensional image is by including a well-mixed auditory hierarchy. By hearing sound effects of the lion’s roar, an instructional narration, and environmental music, the instructional message can become more engaging for the learner and help aid in comprehension. In creating an instructional message with these sounds the sound designer will have to decide the mix of sounds throughout the slide show, so that it helps draw attention to the part of the message intended for the learner.
If the presentation where six-minutes in length and ran four slides in total, you could have each slide appearing for one-minute and thirty-seconds:
o The first slide would be a picture of a lion in the African jungle,
o The second slide would include a list of facts on the lion,
o The third slide has a different picture of a lion in the jungle,
o The fourth slide has more facts on the lion and an activity requesting the learner research further on the African jungle habitat of the lion.
If the six-minute movie of slides seems self-sufficient and engaging without the use of sound, then what could using the film audio hierarchy add to the message? Lets start with the first level of the auditory hierarchy, sound effects: The article Introduction to Animation Sound from The Animation Book: A Complete Guide to Animated Filmmaking, from Flip- Books to Sound Cartoons by Kit Laybournean describes how sound effects can add to the design of animation:
The idea of sound effects creating metaphor:
“The combination of fast moving animated visuals and unrealistic sound effects create audiovisual metaphors that often have humorous aspects. For example…slow moving footsteps synched to cymbal crashes …or .. fast moving hands digging to the sound of a roaring engine..”
The application of sound metaphor to an instructional message could help imply meaning that normally would require further text, or include more visual information that increases the development time for the designer and the participation time of the learner. If we wanted to imply that the lion is the king of the jungle we could play music associated with the entrance of a king in high court commonly found in animations or period-piece films. This helps imply the metaphor “the lion wanders as a courtly king.”
The idea of sound effects creating kinetic energy:
“In animation synchronized sound effects are used as a source of kinetic energy. In the Classic Hollywood Studio Cartoon the sound track are mostly constructed from boinks, petangs, pings, and kerthuds to energize a scene. “
Trying to add kinetic energy to the instructional message could mean layers of sound effects mixed together at different dynamic ranges to produce a sense of spatial and temporal depth for the images. Even though we have a still picture of a lion in a jungle, we could have a mix of quiet and loud roars played at various time intervals, giving the illusion of the lion in movement. The roar of the lion could have a slight degree of reverberation placed on it to increase a sense of spatial depth to match the sound of the lion depending on how close, or how far the lion is framed in the shot. If the lion is in a close-up shot, then less reverberation would be needed then if the lion is further away. If the lion is off in the distance of the jungle, you would want to increase the reverberation effect to make it seem like the lion is as sonically distant as it is visually.
Beyond just the roar of the lion, it might be interesting to have quieter sounds of the lion moving around in the background if it appears to be standing, or maybe subtle sounds of the lion brushing its tail on the ground if it’s laying down. All of these additional sound effects low in the mix, help to give a sense of life, or kinetic energy to the two-dimensional picture: “Recent laboratory studies focusing on cross-modal interactions have found that sound can indeed induce the illusion of seeing a visual event when there is none.” from Filling-in visual motion with sounds by A. Väljamäe a,b,*, S. Soto-Faraco c,d,e. Although the lion is static in the picture, the subtle sound effects can help keep the viewer engaged in narrative aspects of the message and add realism and enjoyment to the presentation.
A more detailed look at the aspects of sound effects in the audio hierarchy comes from edited excerpts from Kerner, Marvin M: The Art of the Sound Effects Editor (1989), where he states “sound effects have the ability to simulate reality, create illusion and define mood.”
In reference to sound effects simulating reality:
“In gun battles the weapon is actually loaded with blanks and what is called quarter loads which means one-fourth of normal amount of gunpowder contained in a real bullet. The actual sound is just slightly louder than a cap pistol until the sound editor has completed work.“
How could such an obvious film reference about “reality” work in the context of instructional messaging? If the creator of the instructional message for lions in Africa had no other option then using still images and was intent on combing sound effects to heighten comprehension about the surroundings, then the most effective way of creating this illusion is with sound that creates a sense of spatial and temporal dimension.
In reference to creating illusion with sound effects:
“Creating illusion was one of the biggest additions to the art of film by sound. By adding the sound of the off-scene diners the audience is convinced that they are still in the café. Obviously, the producer does not want to pay a group of extras to sit off camera. The sound editor places them there with his crowd walla for the sound (Walla is an industry term for the sound of people talking without hearing specific words)”
Creating an illusion in an instructional message could be as important as it is in film. One reason is the financial aspects of placing something in the message that might not be financially viable given the project budget. Simulating the roar of a lion in an African jungle is fairly easily to do with sound effects, but trying to record these sounds for real could cost the instructional designer thousands of dollars to setup and record.
In reference to creating a “mood” for film, the sound designer has several options like the ones listed here: “Add an off-scene owl and it becomes lonesome. Add a wolf howling in the distance and it perhaps harken danger. Cut a gunshot and you are telling the audience that another human is nearby. Is he friend or foe?”
Creating a mood might seem more important in film then it would be in instructional design, but a designer can use these mood techniques to change the tone of a message for different types of learners. Sound effects that are unrealistic and cartoon sounding can help create a message that would be appropriate for a youth audience. Particular sound effects could help create a fantastical world that makes the “reality” of the message well received by children. If the message was for researchers partaking in an African safari, a realistic and serious mood would be most beneficial to their training. The auditory hierarchy positioned in an instructional message can be a powerful tool to assist a designer in creating a message specific to a learners needs.
The second level of the auditory hierarchy used in film is dialogue. David Sonnenschien introduces the concept of “Textual speech” in his text Sound Design: The expressive power of music, Voice, and Sound Effects in Cinema:
“Textual speech: This is generally produced as voiceover commentary to change a setting or call upon a memory, characters, or place, at any moment. The diegetic sound and images are at its mercy, as this voiceover status is reserved for certain privileged characters for a specific moment of entry.”
If dialogue is included in instructional message design, its strongest role could be as voiceover describing instructions, message content, or as a general guide for navigating through activities. If textual speech is added to the instructional message of lions of Africa, we could have an audio tour guide that introduces the content and explains the outcomes of participating. The narrator could welcome the learner, talk about the lion images and guide the learner through the slides of text, by either reading the text, or by asking the learner to think and reflect on specific questions.
Sonnenschien states that sound and film are under the mercy of any narrative power that presents itself. This could work to the advantage of an instructional designer who is looking to tailor their message to a specific audience. Depending on if the message is for young children, or for a field group-researching lions, a humorous and fun sounding voice could introduce the instructional message, or a more dramatic and serious voice could be used for the researchers. Just like visuals and text can be learner specific, sounds can be constructed to suit a specific audience by modifying their delivery.
Further reference to dialogue in the auditory hierarchy by Sonnenschien states:
“Although exceptions exist, dialogue is usually the most significant sound, so all other tracks will work around the verbal information. Music typically lies low until the last word, then rises at a moment of decisive rhythmic or emotional change in the scene.”
In film studies a sound designer will generally find that dialogue takes center stage in the auditory hierarchy. The main function of speech being either to tell the story through the dialogue of characters, or by voiceover that describes narrative to the viewer. Sound effects and music always seem to take the supportive role for dialogue, but film viewers may find themselves involved in an experimental film that tells a story with the auditory hierarchy being driven by music or effects. Examples of these types of films are more common today with the introduction of video and lower cost equipment readily available for experimentation. The need for videos to be a marketable commodity because of a large investment from a corporate company is disappearing and experimental variations of the auditory hierarchy is common.
Here is a list of some main functions of dialogue described in the article Functions of Dialogue in Narrative Film – author not listed (2008 – Advameg Inc.):
o The identification of the fictional location and characters.
o Providing “realistic” verbal wallpaper.
o Guiding the viewer. Filmmakers accomplish this by using dialogue to control pacing or atmosphere.
All of these points are an important tool-set for dialogue in film, but it could appear that verbal wallpaper might be a useless addition to the world of instruction. Depending on the complexity of the message being created for the audience, verbal wallpaper could be a well-added tool for selectively adding attention to a character in isolation, or by making the viewer connect with the emotions of the character. A good example in animation is the use of unintelligible verbal wallpaper in the Peanuts cartoons when adults speak. An instructional designer could replicate sounds similar to this for de-emphasizing a particular character or situation.
The third level of the auditory hierarchy for a sound designer in film is music. Roy A. Prendergast, from The Aesthetics of Film Music explains:
“ Music can tie together a visual medium that is, by its very nature, continually in danger of falling apart. A film editor is probably most conscious of this particular attribute of music in films. In a montage, particularly, music can serve an almost indispensable function: it can hold the montage together with some sort of unifying musical idea. Without music the montage can, in some instances, become merely chaotic. Music can also develop this sense of continuity on the level of the film as a whole.”
Application of this quote can be applied to instructional design with one key word: “continuity.” In reference to the instructional lion message, by having a soundtrack that plays throughout the entire slideshow it can help create seamless transitions from the lion images to simple text slides. Just as music can help disarm the radical juxtaposition of cutting from shot to shot in film, the same technique could help soften any radical transitions in the visual design of an instructional message.
Another design option of music is the ability to create authenticity in an environment. A sound designer in film wants to compose a musical soundtrack to help place the viewer in an authentic context of time and place, and a sound designer in instructional design should be looking to create a specific time and place for the learner. Depending on the intended audience the musical composition can be changed to cater to the “mood” of the message and the intended audience. Sonnenschein adds further to music in film by adding his idea of music as “emotional signifier”:
“Music helps to hypnotize us into the make-believe world of the film, making plausible all that constitutes such genres as fantasy, horror, and science fiction. In all types of films, rather than supporting the realistic image on screen, the music allows us to sense the invisible and inaudible, the spiritual and emotional process of the characters portrayed.”
One of the greatest aspects of music in film is the ability to empathize, or share a subjective experience with the characters on-screen. Sonnenschien states about the power of emotional connections and the ability to find an entrance into the seemingly invisible, or even spiritual aspects of a character. If we used this technique in the design of the lion instructional message for researchers, we could have the music playing exemplify the absolute seriousness and dramatic aspect of sharing a live environment with lions.
The power of music to enhance presentation and narrative comprehension in an instructional messages seems almost unlimited. It might not seem plausible that an instructional message would always need to cause an emotional connection, but even placing a musical intro and outro of ten to fifteen seconds could help set the mood of the message for learners. Some examples can be heard with the music that introduces even the most serious of topics in news and science on television and in podcasts.
Instructional designers need to take advantage of the auditory hierarchy in film to expand presentation and narrative comprehension in their messages. Writings by David Sonnenschien apply the idea of the auditory hierarchy working with Gestalt principles of design and how sound has the ability to become a cognitive tool to enhance learning. Further research on the presentation and narrative power of sound needs to be conducted to better inform instructional designers and multimedia developers. Like any sensory output for engagement, sound has the ability to be distracting, but with the right placement and construction it has the power to create new ways of connecting instructional messages with a learner.
Sync Tanks: The Art and Technique of Postproduction Sound by Elisabeth Weir (Cineaste, 1995).
Introduction to Instructional Message Design by Ellen Rose (Sept.2008).
The Animation Book: A Complete Guide to Animated Filmmaking, from Flip- Books to Sound Cartoons by Kit Laybournean.
Filling-in visual motion with sounds by A. Väljamäe a,b,*, S. Soto-Faraco c,d,e.
The Art of the Sound Effects Editor by Kerner, Marvin M: (1989).
Sound Design: The expressive power of music, Voice, and Sound Effects in Cinema by David Sonnenschien (2001).
Functions of Dialogue in Narrative Film – author not listed (2008 – Advameg Inc.).
The Aesthetics of Film Music by Roy A. Prendergast.