Multimodal meaning denotes the ways that semiotic resources of a multimodal text are used by people in semiotic production (see also entry on multimodal meaning-making).
Drawing on Halliday’s concept of metafunctions (1978), multimodal meaning may be described as organized in three interrelated ways: as representation of some kind of human experience (called ideational meaning), as social interaction between people involved in the communicative act (called interpersonal meaning) and as flow of information that shapes and organizes human experience and interpersonal relations into a text (called textual meaning). Multiple semiotic resources can contribute to all three dimensions of meaning.
In the social semiotic and cognitive literature, multimodal meaning is often understood either from a perspective of interaction or integration. In an interaction perspective, multimodal meaning is an interaction between modes; this interaction gives rise to multimodal meaning, which is a kind of extension of monomodal meaning(s), i.e. meaning created in single semiotic systems. The multimodal meaning in itself is metaphorically described as a layer on top of the layer of monomodal meaning, or a bridge across modes (see e.g. Lim 2004, van Leeuwen 2005; Forceville & Urios-Aparisi 2009).
From an integration perspective, multimodal meaning is rather thought of as a meaningful whole that is more than its parts (i.e. the resources of different semiotic systems). Metaphors of fusion, compression, chemical reaction and emergence are used as descriptive analogies (see e.g. Baldry & Thibault 2006; Kvåle 2012; Steen & Turner 2012; Poulsen 2014).
While both the interaction and integration perspectives are complementary and present in current multimodal research, they also mark a historical shift over the last 15 years, from seeing multimodality mostly as interaction between modes to viewing it as an integration of multiple resources, the latter perspective being pursued, among others, by the CMC-group in Odense.
Kress and van Leeuwen (2001) and van Leeuwen (2005) propose a set of general semiotic principles at work in relation to multimodal meaning. These principles are cohesion, time-space, genre and dialog. The theoretical and empirical work on these principles holds great promise, but further research needs to be done.
Citing this entry
Poulsen, Søren Vigild. 2015. “Multimodal meaning.” In Key Terms in Multimodality: Definitions, Issues, Discussions, edited by Nina Nørgaard. www.sdu.dk/multimodalkeyterms. Retrieved dd.mm.yyyy.
References
Baldry, A. & Thibault, P. J. (2006). Multimodal transcription and text analysis: A multimedia toolkit and coursebook. London: Equinox.
Forceville, C. & Urios-Aparisi, E. (Eds.). (2009). Multimodal Metaphor. Berlin: Mouton de Gruyter.
Halliday, M. A. K. (1978). Language as social semiotic: the social interpretation of language and meaning. London: Edward Arnold.
Kress, G. & Leeuwen, T. v. (2001). Multimodal discourse: the modes and media of contemporary communication. London: Arnold.
Kvåle, G. (2012). Multimodalt samspill i bildeskriftkomplekser – En sosialsemiotisk undersøkelse av relasjoner mellom skrift og bilde. (Ph.D. Dissertation), Universitetet i Agder, Kristiansand.
Lim, F. V. (2004). Developing an integrative multi-semiotic model. In K. O’Halloran (Ed.), Multimodal discourse analysis: systemic-functional perspectives (pp. 220-246). London: Continuum.
Poulsen, S. V. (2014). Mod en analysemetode for webstedet som multimodal tekst. Ph.d.-thesis, SDU.
Steen, F. & Turner, M. (2012). Multimodal Construction Grammar. In M. Borkent, B. Dancygier & J. Hinnell (Eds.), Language and the Creative Mind. Stanford, CA: CSLI Publications.
van Leeuwen, T. (2005). Introducing social semiotics. London: Routledge.