Abstract
In this paper, we present a social semiotic theoretical basis for studying social media as semiotic technology. We argue that social semiotics provides a relevant platform for social media studies, in particular, because it allows for detailed analytical descriptions of multimodal meaning potentials and connecting these to the social context, but that it also needs to be expanded on to cope with this kind of semiotic technology. Our approach is organized around seven dimensions: i) multimodality, ii) practice, iii) the social, iv) medium, v) the material, vi) the historical and vii) the critical. We will illustrate the viability of our arguments with examples from Instagram.In general, social semioticians are interested in the semiotic resources people use for making meaning in multimodal artifacts and events as part of some social practice, and in how the use of these resources are socially regulated (van Leeuwen, 2005). Seen through the lenses of traditional social semiotic studies, the central objects of study is, therefore, firstly, the multimodal artifacts of the given social media e.g. Instagram images along with caption, emojis, hashtags, and secondly, the social practice that the multimodal artefact is part and parcel of.However, in social media (and other digital media) it’s difficult to demarcate the artefacts, traditionally glossed “texts”, as they appear inseparable from the media for design, production, distribution (Kress & van Leeuwen, 2001). For instance, creating an Instagram post not only involves taking and/or uploading an image, but also deploying semiotic resources such as image filters, lighting, location, hashtags, etc., i.e. to engage in a process in which meaning is continuously made by selecting and negotiating with a broad range of resources provided by the social medium. Similarly, responding to an Instagram post involves the use of resources like buttons and sharing functions, which contribute to the design of the multimodal artefact. We will, therefore, argue that it is neither possible nor desirable to distinguish clearly between the multimodal “texts,” e.g. posts and comments, and the text-making devices provided by the social medium used in the creation and sharing of these “texts.” A viable social semiotic multimodal approach therefore also needs to engage with the tools with which the semiotic artefacts are created and which are regulating and regulated by social practice. The tools are part of the ‘semiotic technology’ of a given social medium, along with other dimensions, such as feed algorithms and automatized image content limitations.In social semiotics, the field of semiotic technology has been emerging in the past years to cope better with the role of technology for meaning-making, in particular, digital technologies. The term refers to the whole array of technologies people use for making and managing meaning as part of some social practice, and has so far mainly been developed in relation to PowerPoint (Djonov & van Leeuwen, 2011; 2012; 2013a; 2013b; van Leeuwen, Djonov & O ́Halloran, 2013; Zhao, Djonov & van Leeuwen, 2014), Microsoft Word (Kvåle, 2016) and technologies related to specific fields, such as mathematics (O ́Halloran, 2009). There have been some social semiotic studies of social media, but so far included social media technology only to a limited degree (some exceptions are Jones 2009, Adami 2014; Zappavigna, 2013, 2016; Zhao & Zappavigna, in prep).
In our paper, we want to apply and develop the social semiotic technology approach to social media. In our view, the semiotic technologies of social media provide people with digital environments of preselected semiotic resources and designs for making, enacting and managing meanings. The object of a social semiotic technology study should therefore not be limited to the meaning potentials of social media posts or the “use” of social media i.e. on the semiotic resources that are present on the surface or display; the research attention should also be directed towards the technologically designed spaces of possibilities and constraints and the semiotic actions these spaces allow for and invite to. Technology is not only a matter of physical and electronic products and processes, or a question of functionality and user friendliness, but is in itself inscribed with sociality and cultural norms in codes, algorithms, templates, and designs. Semiotic technologies are thus not seen as neutral technical devices, but as historically developed social, cultural and semiotic constructs that play a crucial role in changing and creating new social practices.
What we strive at in this article is providing a viable theoretical-analytical approach that serves to underscore our research interest: the relationship between multimodal meaning potentials of semiotic surfaces, various dimensions of the technology “below” these surfaces, and the social practices in which a given social media is embedded in and contributing to shaping. We have found that the following seven dimensions are the most crucial dimensions in our thinking, and we will, therefore, structure our paper around it: i) practice, ii) multimodality, iii) the social, iv) medium, v) the material, vi) the historical, and vii) the critical. Other dimensions may be added, and the order and significance of each of the seven dimensions will vary in an analysis of specific social media technologies. But as a whole, the dimensions provide an apt perspective for the multimodal social semiotic study of social media.
References
Adami, Elisabetta (2014) Retwitting, Reposting, Repinning; Reshaping Identities Online: Towards a Social Semiotic Multimodal Analysis of Digital Remediation. LEA : Lingue e Letterature d’Oriente e d’Occidente 3 (3): 223-243.
Djonov, E. & Van Leeuwen, T. (2011). The semiotics of texture: from tactile to visual. Visual Communication, 10 (4), p. 541–564.
Djonov, E. & Van Leeuwen, T. (2012). Normativity and software: A multimodal social semiotic approach. In S. Norris (ed.), Multimodality and Practice: Investigating Theory-in-Practice-through-Method (p. 119–137). New York: Routledge.
Djonov, E. & Van Leeuwen, T. (2013a). Between the grid and composition: Layout in PowerPoint’s design and use. Semiotica (197), p. 1–34.
Djonov E. & van Leeuwen, T. (2013b): Bullet points, New Writing and the Marketization of Public Discourse. A critical multimodal perspective. In E. Djonov & Z. Sumin (eds.) Critical Multimodal Studies of Popular Discourse (p. 232–250). London: Routledge.
Jones, R. H. (2009). Technology and sites of display. In Jewitt, Carey (ed.) The Routledge Handbook of multimodal analysis. (pp. 114-126). London: Routledge.
Kress, G. & van Leeuwen, T. (2001). Multimodal Discourse. The Modes and Media of Contemporary Communication. London: Arnold.
Kvåle, G. (2016). Software as Ideology. A Multimodal Critical Discourse Analysis of Microsoft Word and SmartArt. Journal of Language and Politics – Special Issue on Multimodality, Politics and Ideology 15 (3), p. 259–273.
O’Halloran, K. (2009). Historical changes in the semiotic landscape: from calculation to computation. In C. Jewitt (ed.) The Routledge Handbook of Multimodal Analysis (p. 98–113). London: Routledge.
van Leeuwen, T. (2005). Introducing Social Semiotics. London: Routledge.
van Leeuwen, T. & Djonov, E. (2013). Multimodality and software. In C. Chapelle (ed.), Encyclopedia of Applied Linguistics. (unpaginated). Oxford: Wiley-Blackwell.
van Leeuwen, T., Djonov, E., & O’Halloran, K. L. (2013). “David Byrne really does love PowerPoint”: art as research on semiotics and semiotic technology. Social Semiotics 23 (3), p. 409-423.
Zappavigna, M. (2013) “Enacting identity in microblogging through ambient affiliation”. Discourse & Communication.
Zappavigna, M. (2016), `Social media photography: Construing subjectivity in Instagram images’, Visual Communication 15(3), 271-292.
Zhao, S., Djonov, E., & van Leeuwen, T. (2014). Semiotic technology and practice: a multimodal social semiotic approach to PowerPoint. In Text & Talk 34 (3), p. 349-375.
Zhao, S & Zappavigna, M (in pre) Intersubjectivity and the Selife: a social semiotic analysis of social photography from Instagram to “hipster” magazines New Media and Society