Resources

Unit 7: Designing a multimodal study

Chapter overview

Learning objectives

This study unit will help you to:

  • Engage with the process of designing a multimodal study
  • Reflect on which approach is most apt approach for your study
  • Identify a research focus and questions for your study
  • Identify your empirical focus
  • Explore and assess methods for transcribing and managing your research materials
  • Engage with the ethical dimensions of multimodal research

Overview of Chapter 7

Topics

  • Choosing an apt research approach
  • Formulating a research focus and questions
  • Selecting an empirical focus
  • Collecting research materials
  • Transcribing and managing your research materials
  • Ethical dimensions of multimodal research

Summary

In Chapter 7 we discussed how to design a multimodal study. We started the chapter by discussing the two routes into multimodality, doing multimodality or adopting multimodal concepts, and pointed to the need to assess how central multimodality is to your study and the need to choose an approach accordingly.

  • If you are doing multimodality your first design decision will be to choose an apt approach for your study (e.g. CA, SFL, or social semiotics and the frameworks in Chapter 6). Each of these approaches will provide you with a framework to guide your study, formulate research questions and choose an empirical focus.
  • If you are adopting multimodal concepts your starting point will be different. You will likely be coming to multimodality with a clear research interest or question formulated from a different theoretical position. You will need to select a research framework and reframe your research interest and questions to account for a multimodal perspective.

We then presented and discussed the key elements that need to be considered when designing a multimodal study (see the above list of Chapter 7 topics).

Study questions

  1. How does the route to multimodality, doing multimodality or adopting multimodal concepts, affect the process of designing a multimodal study?
  2. How does the choice of an approach shape the design of a study more generally?
  3. In your view, what are the challenges of focusing on either artefacts or interaction or both in a multimodal study?
  4. What benefits and challenges can transcribing research materials present for a multimodal study?
  5. How do you know when you have collected enough research materials?
  6. What are examples of ethical issues that need to be addressed in a multimodal study?

Exercises

Exercise 7.1: Justifying an approach

  1. Select a paper that uses a multimodal approach in an area of interest to you.
  2. Do you think it the most apt approach for their study?

Tip: Whatever your route to multimodality, you need to be clear on your theoretical orientation to provide a solid grounding for the design of your study.

Exercise 7.2: Focusing your research

A second challenge you will encounter when you design a multimodal study is the need to clarify your research focus. As we discussed in Chapter 7 this challenge will vary depending on whether you are doing multimodality or adopting multimodal concepts.

If you are doing multimodal research the requirements of the particular approach that you select will frame the area and guide the questions of your study.

Alternatively, if you are adopting multimodal concepts for your study you will need to bring a research area, topic or question from another theoretical context.

In both cases you will need to make some significant choices and decisions within this frame regarding research area, topic and questions for your study design.

  1. Use the following ways to generate, refine and focus your research area and questions:
    • Collect some artefacts similar to those that you will research and spend time exploring them.
    • Read papers in your area of interest that use the multimodal approach you have chosen for your study. Do they suggest new directions for research? Do they reveal a gap your study could address?
    • Talk to other students?
    • Brainstorm ideas.
  2. Ask yourself what your overarching research questions and sub-questions are. Are they:
    • Orientated to multimodality?
    • Related to previous research on the topic that you are studying?
    • Clear and answerable?
    • Significant? i.e. does the question matter to anyone, and to whom?
    • Feasible to undertake within the limits of time and other resources that are available to you?

Tip: A clear and answerable question that connects your study to multimodality and a body of previous research, that is, interesting and significant, as well as feasible will provide a useful anchor for your study.

Exercise 7.3: Collecting artefacts as research materials

In Chapter 7 we made a distinction between two broad categories of research materials: artefacts produced by the people you want to study and recordings of their interaction produced by a researcher. We noted the qualities of the materials used across the different approaches discussed in this book (i.e. ‘naturalistic’, fine-grained, and socially situated).

We discussed the specific requirements of different approaches and how these steer the type of materials needed for a study and their collection. The collection of artefacts as research materials was explored in Chapters 4 through to 6 and Units 4 through to 6 of this study guide.

The following exercise is a general one that needs to be considered in conjunction with your approach.

  1. Identify the type(s) of artefact that you might need to collect for your study (e.g. children’s drawings, leaflets, advertisements, building or street environment)
    • Can you collect the original artefact or do you need to reproduce it (e.g. in a photograph, or a scan)?
    • Will any aspects of the artefact be lost during the process of collecting it (e.g. its scale, textual qualities, dynamic elements)? How could you compensate for that loss?
    • Are there any ethical or copyright concerns? How can you respond to these?
  2. How will you set the boundary of the artefact (e.g. a section, a page, a book)?
  3. How much and what kind of contextual information will you need to analyse your materials? Do you need to collect materials on how they were made? Where they are circulated? How people engage with them?
  4. How many artefacts will you need to address your research question?
  5. Do you need to collect a range of artefacts or not?

Tip: You will need to consider the losses involved in recording artefacts and how that might affect your analysis.

Tip: You will need to consider what type and how much contextual information you need about the artefacts you collect for your study.

Exercise 7.4: Recording materials to study interaction

The same caveats outlined for Exercise 7.3 apply to the recording of material to study interaction, discussed in Chapters 3, 5 and 6. The following exercise is a general one that needs to be considered in conjunction with your approach.

As we discussed elsewhere in this book (i.e. Chapters 3 and 7) video is a medium for representing the world in sound and moving image, offering distinct potentialities and challenges for recording interaction. However, the use of video to collect materials for multimodal materials often involves ‘trade-offs’ between detail and the bigger picture or context. How you set up your video to collect data will shape the data you collect, which in turn will shape your analysis.

  1. Imagine you want to video record a whole-class activity, involving some 20 students and a teacher in a classroom.
  2. How would the recordings of their interaction be shaped by each of the following three set ups?
    • Fixed video camera with a wide lens recording the whole class
    • Fixed video camera focused on small groups of students, changing the group every 15 minutes
    • Fixed camera zooming and panning
    • Roaming video camera focused on (and moving with) with the teacher
    • Two video cameras, one focused on the teacher, the other on the students
  3. What might be the challenges of the video recordings produced using a fixed, roaming, or multiple cameras?
  4. What other materials might you collect to supplement the video recordings (e.g. field notes, (recordings of) artefacts such as documents)?
  5. What are your (provisional) guiding questions? What subject or type of activity, or phenomenon, are you interested in? This will shape your filming decisions.
  6. How many hours of footage will you need?

Tip: You will need to have a clear idea of the kind of video recordings that you need to produce for your study. Do you need ‘minimally edited’ video, with minimal ‘zooming and panning’ giving a relatively fixed view of participants interacting (gathered using a wide-frame and long-shot)? Do you need video to capture facial expressions, hand movements et cetera? (Which requires a tight-frame close-up camera.) Or do you need to capture both detail and the bigger picture by changing the camera position during video recording, using a roaming camera or more than one camera?

Exercise 7.5: Transcribing research materials

The same caveats outlined for Exercise 7.4 apply to the transcription of material, which is discussed in Chapters 3, 5 and 6 and Units 3, 5 and 6 of this study guide. The following exercise is a general one that needs to be considered in conjunction with your approach.

  1. Look at the three transcripts below (each of these is discussed in more detail elsewhere in this book: see pages 76, 99, and 117).
    Fig 4.1 - Multimodal transcript of the teacher's framing of the task (page 146). Re-printed from Mavers, D. (2009) 'Student text-making as semiotic work' Journal of Early Childhood Literacy, 9(2): 145–155. Reprinted with permission from SAGE Publications and the author
    Fig 4.1 - Multimodal transcript of the teacher’s framing of the task (p. 146) Re-printed from Mavers, D. (2009) ‘Student text-making as semiotic work’ Journal of Early Childhood Literacy 9(2), 145–155. Reprinted with permission from SAGE Publications and the author.
    Fig 5.4 - Transcript of interaction between a surgical trainee and his supervisor
    Fig 5.4 - Transcript of interaction between a surgical trainee and his supervisor
    Fig 6.1 - Transcript of six simultaneous higher-level actions (reprinted from Sigrid Norris, Analyzing Multimodal Interaction (2004), p. 102)
    Fig 6.1 - Transcript of six simultaneous higher-level actions (reprinted from Sigrid Norris, Analyzing Multimodal Interaction (2004), p.102)
  2. Compare the three transcripts using the dimensions outlined in Chapter 7 (pages 146 to 149):
    • Principles of framing, selecting and highlighting
    • Types of transduction and use of modes: writing, image, layout, time
    • Time scale
    • Unit of analysis
  3. What aspects of interaction does each transcript foreground or background?
  4. Which aspects of interaction will be important to foreground in the transcripts for your study, and why?
  5. What model of transcription and conventions will you draw on in your study?

Tip: You will need to select a model of transcription that aligns with your approach and the questions you want to address.

Exercise 7.6 Considering ethical dimensions

As we discuss in Chapter 7, the planning of a study of multimodal interaction will usually involve gaining access to a site where you can video record and obtaining ethical approval. The question is whether the use of materials for multimodal analysis, especially video materials, throws up any specific challenges in this regard. Some highly experienced video researchers commented:

‘One reason that is often raised for not using video in qualitative research is that it will be impossible to gain permission to make recordings from participants, or more generally, an organisation. Surprisingly perhaps, gaining access to undertake video recording rarely proves a major difficulty, as long as you are sensitive to the demands of the setting and address the concerns of the participants themselves. In recent years, video-based studies have been undertaken in such diverse settings and activities as medical consultations, management consultancy, counselling interviews, banking and financial management, surgical operations, air traffic control, hairdressing, surveillance, nursing, television and radio production, home life, shops and business meetings. Each of these settings pose different challenges for data collection and demand different techniques for securing consent.’ (Heath, Hindmarsh and Luff 2010: 15)

  1. Imagine you want to do video-ethnographic fieldwork in a work place.
  2. In your view, what, if any, unique ethical issues does the collection and use of video recordings in such a context throw up.

    Note: We can make a distinction between anonymity and confidentiality(Derry et al. 2010). Anonymity cannot be ensured, unless you edit the video such that people’s faces and voices can no longer be recognized (software such as Windows Movie Maker allows you to do that). In doing so, the video also loses some of its potential for making visible the phenomenon of interest. However, confidentiality can be protected, by omitting personal information and information about the research site.

  3. With reference to the strategies discussed in Chapter 7 (pages 150–153) how might you be able to manage those ethical issues?
  4. What ethical issues might the use of video recording in your intended study site raise? What measures could you take to deal with them?

Study design tip: Before you approach your participants, think through the concerns that your use of photography and/or video for the study might raise for them and how you can respond to their concerns.

Study design tip: When you design your consent form, give participants different options to opt in and out of (e.g. the use anonymized clips and transcripts and/or un-anonymized clips and transcripts for research or training or publication purposes).

Suggested resources

Further reading

Reading the publications of the many multimodal studies cited in the different chapters of this book provide you with a good starting point for designing your multimodal study.

Transcription

Bezemer, J. & Mavers, D. (2011). Multimodal transcription as academic practice: A social semiotic perspective. International Journal of Social Research Methodology 14, 191–206.

Mavers (2012) discusses examples illustrating different ways of making relatively detailed transcripts of a small strip of interaction, accounting for bodily conduct alongside speech. NCRM working paper

Working with video

Jewitt, C. (2012) An introduction to using video for research. NCRM working paper.

Heath, C., Luff, P. & Hindmarsh, J. (2010). Video in Qualitative Research. London: Sage.

Derry, S. J., Pea, R. D., Barron, B., Engle, R. A., Erickson, F., Goldman, R., Hall, R., Koschmann, T., Lemke, J. L., Sherin, M. G. & Sherin, B. L. (2010) Conducting video research in the learning sciences: Guidance on selection, analysis, technology, and ethics. Journal of the Learning Sciences 19(1), 3–53.

Online resources

The MODE transcription bank is a collection of reflections on transcripts made from video recordings. Authors present transcripts they used in their research and provide an account of why and how they made them

Appendix A to Guidelines for conducting video research in education (Derry 2007).

ELAN Transcription software, a professional tool for the creation of complex annotations on video and audio resources

The CAQDAS network offers practical support, training and information in the use of a range of software programs designed to assist qualitative data analysis including multimodal data

References

Derry, S. J., Pea, R. D., Barron, B., Engle, R. A., Erickson, F., Goldman, R., Hall, R., Koschmann, T., Lemke, J. L., Sherin, M. G., & Sherin, B. L. (2010). Conducting video research in the learning sciences: Guidance on selection, analysis, technology, and ethics. Journal of the Learning Sciences 19(1), 3–53.

Heath, C., Hindmarsh, J., & Luff, P. (2010). Video in Qualitative Research. London: Sage.