Multimodal integration refers to the ways semiotic resources function as a multimodal whole. For a resource to enter into semiosis it depends on and thus must be integrated with other resources to do semiotic work in a textual unit. This text may be further integrated into other larger functional units (Boeriis 2015).
Baldry and Thibault (2006: 3) describe integration as a key principle of multimodal meaning-making: “Multimodal texts integrate selections from different semiotic resources to their principles of organisation. (…) These resources are not simply juxtaposed as separate modes of meaning-making but are combined and integrated to form a complex whole which cannot be reduced to, or explained in terms of the mere sum of its separate parts.”
Van Leeuwen (2015) outlines a number of concepts that could be useful to address in relation to multimodal integration:
Cohesion
The study of cohesion has focused on conjunction (text-image relations) and reference while less attention has been given to ‘lexical cohesion’ and cross-modal grammatical relations (e.g. verbal participants, visual processes). The study of cohesion could also be extended to include ‘formal’ cohesion (e.g. colour grading, style sheets, etc.) (van Leeuwen 2005; Holsanova 2012).
Prosody
So far, studies of prosody have focused on annotating verbal text transcriptions with prosodic diacritics or interspersed markers of intonation and non-verbal communication, thus focusing on forms of multimodality where language is not necessarily the central mode. Such studies naturally focus on time-based textual organization as a key element of multimodal integration (Cyrstal 1969; Halliday 1970; van Leeuwen 2016).
Composition and rhythm
To some extent, multimodal integration has been addressed in the study of composition and rhythm with its focus on compositional integration of space-based texts and the rhythmic integration of the elements in a time-based text. This has included studies of how time-based structure is generated from space-based structure, e.g. reading paths (van Leeuwen 2005).
Genre
An important aspect of integration relates to genre. The existing literature has focused on the way the stages of a genre can be realized by different modes and/or combinations of modes. But it does not deal with the integration of modes in multimodal stages. Further study in this field is much needed. (see entry on Genre)
Supra-modal concepts
“Supra-modal concepts” refers to analytical methods that can be applied to different modes (including genre). A supra-modal concept does not in itself model multimodal integration, but can provide a basis for comparison and integration (e.g. transitivity in different modes) (Johannessen 2017). However, operating with supra-modal concepts involves the risk of not recognizing differences between modes (e.g. meanings that can be realized in one mode but not in another).
Blended parameters
Approaches to multimodal integration as a kind of blending have focused on the parameters that operate simultaneously in modes and claim that each parameter, in its specific proportionality, contributes to the overall meaning, but have no principled approach to analysing that overall meaning. Blending theory has not (yet) been applied across modes, though parameters may have analogies in different modes (synaesthesia).
Besides the need to further develop each of these different approaches individually, van Leeuwen points out that the research needs to look into the advantages and disadvantages of the different approaches and to consider the theoretical and analytical contributions of each approach and the extent to which they may be combined.
According to Poulsen (2014), multimodality has been understood historically from either an interaction perspective or an integration perspective, though these perspectives are not mutually exclusive. These two perspectives are also seen in the last 20 years of multimodal research which has moved from understanding and analysing multimodal texts as interaction between modes towards analysing them as integration of resources. Taking inspiration from Bache’s (2005) elaboration of cognitive blending theory (Fauconnier & Turner 2002), Poulsen finds that it is relevant to discuss multimodal integration in relation to disintegration, meaning that although resources create meaning in an integrated unit, single resources may be disintegrated in order to perform some specific semiotic work. Bache also suggests blending at different levels, i.e. expression, form and semantics. A similar distinction between integration of semiotic resources between levels and across levels would be interesting to explore.
Citing this entry
Poulsen, Søren Vigild. 2017. “Multimodal integration.” In Key Terms in Multimodality: Definitions, Issues, Discussions, edited by Nina Nørgaard. www.sdu.dk/multimodalkeyterms. Retrieved dd.mm.yyyy.
References
Bache, C. (2005). “Constraining conceptual integration theory: Levels of blending and disintegration”, Journal of Pragmatics, vol. 37 (10), pp. 1615-1635.
Baldry, A. and Thibault, P. (2006). Multimodal transcription and text analysis: A multimedia toolkit and coursebook. London: Equinox.
Boeriis, M. (2015). ”Det analytiske nærbillede: om multimodal analyse af levende billeder i kontekst.” In Kvåle, G., Maagerø, E. and Veum, A. (Eds.). Kontekst språk multimodalitet: Nyere sosialsemiotiske perspektiver. Bergen, Norway: Fagbokforlaget Vigmostad og Bjørke, pp. 103-122.
Crystal, D. (1969). Prosodic Systems and Intonation in English. London: Cambridge University Press.
Fauconnier, G. and Turner, M. (2002). The way we think:conceptual blending and the mind’s hidden complexities. New York: Basic Books.
Halliday, M.A.K. (1970). Course in Spoken English: Intonation. London: Oxford University Press.
Holsanova, J. (2012). “New methods for studying visual communication and multimodal integration,” Visual Communication, 11(3), 251-257.
Johannessen, C. M. (2017). ” Experiential meaning potential in the Topaz Energy logo: a framework for graphemic and graphetic analysis of graphic logo design,” Social Semiotics vol.27 (1): 1-20.
Poulsen, S. V. (2014). Toward an analytical model for web sites as multimodal texts. Unpublished Ph.D. Thesis. Odense: University of Southern Denmark.
van Leeuwen, T. (2015). “Introduction to Multimodal integration.” Paper presentation given at DNC#1: DiscourseNet International Congress, 25 Sept., Bremen University, Germany.
van Leeuwen, T. (2016). “The Meaning of Manner: Change and Continuity in the Vocal Style of News Reading and Information Announcements,” in Mortensen, J., Coupland, N., and Thogersen, J. (Eds.) Style, Mediation and Change – Sociolinguistic Perspectives on Talking Media. Oxford/New York: Oxford University Press.
van Leeuwen, T. (2005). Introducing social semiotics. London: Routledge.