Resources

Unit 6: Five more approaches to multimodality

Chapter overview

Learning objectives

This study unit will help you to:

  • Engage critically with the possibilities for mixing multimodality with other methods
  • Familiarize yourself with approaches that have mixed multimodality with other methods and theories
  • Consider the potentials and limitations of these combined approaches

Chapter overview

Topics

  • Research frameworks combining multimodality with other theories and methods
  • The potentialities and challenges of these frameworks and their contribution to multimodal research

Summary

In Chapter 6 we explored the analytical potential of bringing approaches to multimodality into contact with other methodological or theoretical frameworks. We presented five approaches that have combined multimodal concepts with other approaches, that have some settled conventions and practices and an (emergent) research community, and that have made a distinct contribution to extending the scope of multimodality. The five frameworks were:

  • Geo-semiotics (or discourses in place) combines linguistic anthropology, and place semiotics with a social semiotic approach to foreground the placement of semiotic artefacts and interactions in the wider spatial environment.
  • Multimodal (inter)action analysis combines interactional social-linguistics and a social semiotic approach to foreground identity formation through interaction.
  • Multimodal ethnography uses social semiotic theory in ethnographic research to produce accounts of situated artefacts and interactions and the relations between them.
  • Corpus-based multimodal analysis combines corpus analysis methods with SFL and social semiotic approaches to empirically evaluate, critique and validate the hypotheses and theories of multimodal meaning making through the analysis of artefacts.
  • Multimodal reception analysis combines psychological concepts and methods to investigate the cognitive processes involved in the ‘perception’ of textual artefacts and to ‘test’ semiotic principles proposed in systemic functional linguistics and social semiotics.

We outlined the scope, aims and research questions of each approach, as well as their theory of meaning, concepts and methods, and highlighted their contribution to multimodality. We illustrated the potential of connecting multimodality with other theories and taking it in new directions, specifically to foreground space and place, identity, insider perspectives, and to test theories of meaning. We also discussed the need to ensure such combinations achieve an internal coherence and the challenges of doing so.

Study questions

  1. When might it be necessary to combine multimodality with another approach?
  2. Do these five combined approaches change the multimodal concepts that they use. If yes, how?
  3. What do you think are the main benefits and challenges of incorporating multimodal concepts into another research framework?
  4. In your view, are there any methods or theories that are ‘incompatible’ with multimodality? If not, why not? If so, which ones would be incompatible?

Exercises

Exercise 6.1: Assessing the benefits and challenges of combining multimodality with other methods

  1. In Chapter 6 we quoted the following statement by Rosie Flewitt:
    • ‘A focus on semiosis could show what choices children made, and what modes were made available to them through diverse traditional and “new” literacy texts in diverse media, but this constituted “thin” descriptions which situated the production and reception of meanings within physical and material social settings, but did not make accessible more holistic insights into their literacy learning. …For this deeper level of understanding, the study was dependent on ethnographic data, which constituted a rich backstory of how networks of cultural and social values permeated the children’s homes and nursery. While multimodal analysis captured something of the communicative complexity of the studied field, ethnographic approaches to data collection and interpretation helped to situate that complexity in particular social, cultural and historical contexts.’ (Flewitt, 2011: 302)
  2. What tensions can you see in combining a social semiotic and ethnographic approach? How does Flewitt resolve these tensions?

(You may want to engage with the full paper (see further readings) to help you with this exercise.)

Tip: If you combine multimodality with other theories and methods you will need to be explicit about the benefits and challenges of doing so.

Exercise 6.2 Evaluating the contribution to multimodal research

  1. Select one of the five approaches presented in Chapter 6.
  2. Choose a paper written by the named scholar who has led its development from the further readings provided below or the references at the end of this book.
  3. Read the paper and address the following questions:
  4. Is it clear how the authors have used multimodal concepts in their research framework?
  5. In your view, does the approach extend the multimodal approach that it draws on? If so, how? If not, why not?

Tip: It might be useful to be explicit about what contribution is made to methods by using multimodality in a research framework with other methods in your study.

6.3: To mix or not to mix?

  1. Select one of the multimodal artefacts or short video clips of interaction that you engaged with to undertake the exercises in Units 3 to 5.
  2. Choose a research framework that you think is the most apt for your materials from those we discussed in Chapter 6.
  3. How would the selected framework change the scope of materials that you would need to collect?
  4. In your view, would there be any benefits in doing so? If yes, what are they? If not, why not?
  5. What new tensions and challenges might this framework give rise to?

Suggested resources

Bateman, J. (2008). Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. Basingstoke, Hampshire; New York: Palgrave Macmillan.

Dicks, B., Flewitt, R., Lancaster, L. & Pahl, K. (Eds.) (2011). Special issue: Multimodality and ethnography: Working at the intersection. Qualitative Research 11, 3.

Flewitt, R. (2011). Bringing ethnography to a multimodal investigation of early literacy in a digital age. Qualitative Research 11, 293–310.

Holsanova, J. (2014). Reception of multimodality: Applying eyetracking methodology in multimodal research. In Jewitt, C. (Ed.) The Routledge Handbook of Multimodal Analysis (2nd ed.) (pp. 287–98). Abingdon, Oxon: Routledge.

Norris, S. (2004). Analyzing Multimodal Interaction: A Methodological Framework. New York, NY: Routledge.

Scollon, R., & Wong Scollon, S. (2003). Discourses in Place: Language in the Material World. London: Routledge.

Online resources

John Bateman’s website on multimodal documents and film

Sigrid Norris’s website on multimodality

Reference

Flewitt, R. (2011). Bringing ethnography to a multimodal investigation of early literacy in a digital age. Qualitative Research 11, 293–310.