Glossary

Adjacency pair

is a term used in conversation analysis to describe a two-part interactional structure, in which the first part is an invitation to the other to respond (e.g. a greeting followed by a greeting response). This structure was first identified in talk. Recent multimodal studies have shown that often only one or none of the two parts are realized in talk (e.g. an exchange of gestural greetings between two acquaintances who are not within earshot).

Affordance

is a social semiotic term used to refer to the idea that different modes offer different potentials for making meaning. Modal affordances affect the kinds of semiotic work a mode can be used for, the ease with which it can be done, and the different ways in which modes can be used to achieve broadly similar semiotic work. Modal affordances are connected both to a mode’s material and social histories, that is, the social purposes that it has been used for in a specific context.

Artefact

is a term used to refer to anything that bears the traces of semiotic work: a building, an inscription, a video recording or film, a landscape, and so forth. In this book we use the term specifically to refer to artefacts produced by the peoplewhose practices we want to study, and to make a methodological distinction between using such artefacts as an empirical ground for multimodal analysis (e.g. of ‘text’, which is common in systemic functional grammar and social semiotics) from using artefacts made by a researcher (such as a video recording of ‘interactions’) as an empirical ground for multimodal analysis (which is common in conversation analysis).

Communication

see meaning making

Conversation analysis (CA)

is an approach to the study of social interaction. In its early days the focus was on analysing ‘talk’ and ‘conversations’ (hence its name). The analytical scope has since been expanded to include attention to a much broader range of semiotic resources used in interaction. Central to a CA approach is the systematic and in-depth analysis of video recordings of multimodal interaction, with a focus on the sequential organization of action and the coordination of action.

Coordination of action

is a term used in conversation analysis to describe an analytical concern with the ways in which people who are engaged in ‘collaborative work’ coordinate their actions with the actions of others and accomplish their work through ‘concerted action’.

Critical discourse analysis (CDA)

is a branch of linguistics that investigates how linguistic choices connote broader discourses. It asks what kind of world, social values, ideas and identities are being represented in language. The aim is to reveal the social and power relations that are realized linguistically and to show what kinds of inequalities and interests they seek to create or perpetuate.

Design

is a social semiotic term used to refer to the situated process in which a sign maker chooses and arranges semiotic resources to realize a particular social function or purpose. Design is conceived of as a starting point for making meaning and both the original producer of a sign and its interpreter are understood as making meaning. It is always socially located and regulated, for example by the types of resources that are made available and to whom, as well as the regimes that regulate and shape how these resources are used to create various norms and expectations.

Genre

is a term used in systemic functional linguistics (SFL) to describe how social processes unfold in a given culture. The concept, which is related to register theory, is conceived as the system of staged goal-oriented social processes through which social subjects in a given culture live their lives.

Geo-semiotics

is a research framework that combines concepts and methods from linguistic anthropology, place semiotics and social semiotics in order to foreground the emplacement of semiotic artefacts and interaction.

Interest

is a term used in a social semiotic approach to refer to a momentary condensation of all the social experiences that have shaped an individual’s subjectivity – a condensation prompted by the social environment (of which the available modes are a significant element) that a sign is made within. A person’s interest connects their choice of one resource over another with the social context of sign production.

Meaning making

is a general term used in this book to recognize that meaning always involves a social actor. Both ‘expressing’ and ‘interpreting’ or ‘understanding’ are acts of making meaning. Meaning making is a more generic term than communication, which is usually defined as involving ‘expression’ by one actor and ‘interpretation’ by another. When there is ‘expression’ and/or ‘interpretation’ by one person only (say when someone makes a sketch or some notes of what s/he observes for her/himself), we can still speak of meaning making, but not of communication.

Metafunction

is the term that Michael Halliday developed to describe the functions of language in systemic functional linguistics: experiential meaning (to construct our experience of the world), logical meaning (to make logical connections in that world), interpersonal meaning (to enact social relations and create a stance to the world) and textual meaning (to organize messages). The metafunctional principle is adopted in systemic functional multimodal discourse analysis (SF-MDA) to understand the functionalities and underlying organization of semiotic resources and to investigate the ways in which semiotic choices interact to create meaning in multimodal texts and processes.

Mode

is a term that is used within systemic functional linguistics and social semiotics to refer to a socially organized set of semiotic resources for making meaning. Examples of modes include image, writing, layout, and speech among others. For something to count as a mode it needs to have a set of semiotic resources and organizing principles that are recognized within a community as realizing meaning. For example, the resources of gesture have been semiotically shaped into communicative modes to serve a diverse range of communities (e.g. hearing-impaired communities, visually and hearing impaired people, ballet dancers).

Motivated sign

is a social semiotic term used to reference that meaning (‘signified’) and form (‘signifier’) are brought together in a relation motivated by the aptness of fit between the interest of the sign maker and the affordances of a semiotic resource.

Multimodal

see multimodality; multimodal approach

Multimodal approach

refers to a distinct theoretical and methodological framework for the study of multimodality (see multimodality).

Multimodal corpus analysis

is a research framework that combines corpus analysis methods with SFL and social semiotics. It is used to empirically test/validate the hypotheses and advance concepts and theories of multimodal meaning making through the systematic analysis of a corpus of artefacts.

Multimodal ethnography

is a research framework that uses social semiotic theory in ethnographic research to produce accounts of situated artefacts and interactions and the relations between them. These ethnographic accounts are also based on interviews with the members of the community of practice under study and on the ethnographer’s field notes, documenting insights obtained through prolonged observation within the community.

Multimodal (inter)action analysis

is a research framework that combines interactional sociolinguistics and a social semiotic approach to foreground identity formation through interaction.

Multimodal reception analysis

describes a research framework that uses psychological concepts and methods to investigate the cognitive processes involved in the ‘perception’ of textual artefacts and to ‘test’ semiotic principles proposed in systemic functional linguistics and social semiotics.

Multimodality

in a broad definition highlights that people draw on distinctly different sets of resources for meaning making (e.g. gaze, speech, gesture). The narrower definition adopted in this book also stresses that in actual instances of meaning making these resources are used in conjunction to form multimodal wholes. (See mutual elaboration and affordance.)

Multisemiotic

see multimodality

Mutual elaboration (of semiotic resources)

is a term used in conversation analysis to recognize that sets of semiotic resources (e.g. gesture) are partial and incomplete and that when joined together diverse semiotic resources mutually elaborate each other. The analyst’s task is to account for the meaning of the ‘sum’ of which they are a part, not for the meaning of a verbal utterance, or a gesture, in isolation. This notion resonates with the social semiotic notion of affordance.

Register

is a term used in systemic functional linguistics to describe the configurations of meaning across three key dimensions: the nature of the activity (field), the social relations in terms of the dimensions of power and solidarity (tenor), and the composition and information flow of the message (mode). These register variables are realized through the ideational (experiential and logical meanings), interpersonal and textual meanings in systemic functional theory (SFT).

Resemiotisation

is the re-construal of semiotic choices within and across multimodal processes and texts. Resemiotisation provides means for understanding how semiotic systems are called into play as social processes unfold. This is a key concept in systemic functional multimodal discourse analysis (SF-MDA) (see also transduction)

Semiotic principle

is a term that is used to refer to principles for and features of meaning making that apply across modes. For instance, all modes have resources for producing intensity. In the mode of speech that is realized by the intensity of sound – ‘loudness’; it is also realized lexically, e.g. as ‘very’. In the mode of gesture intensity might be realized by the speed of movement of the hand, or by the extent of the movement. In the mode of colour it might be done through degrees of saturation, et cetera.

Semiotic resource

is a term used to refer to the meaning potential of material resources, which developed and accumulated over time through their use in a particular community and in response to certain social requirements of that community.

Sequential organization of action

is a term used in conversation analysis to describe the principle that interaction unfolds in time, one action after another. Actions (which can be realized using different semiotic resources) are ‘context-shaped’ in that they serve as a response to the preceding action, and at the same time they are ‘context-renewing’, in that they shape and raise expectations about what is to happen next. The key to understanding the interlocutor’s understanding of an action lies in their response to that action.

Social semiotics

is an approach concerned with how the processes of meaning making (signification and interpretation or ‘semiosis’) shape and are shaped by individuals and societies to realize power and ideologies. It stresses the relationship between modes and their affordances and the social needs they are used to serve, the agency of the sign-maker, and the context of meaning making. It is related to three main strands of influence, systemic functional linguistics, semiotics, and critical linguistics/critical discourse analysis (CDA).

System

is used in systemic functional linguistics to explain how semiotic resources are organized to create meaning. The systems are described in terms of systemic choices (the paradigm) that are mapped onto the structure of language (the syntagm or chain). The notion of system is adopted in systemic functional multimodal discourse analysis (SF-MDA) to describe the meaning potential of semiotic resources and their interactions.

System network

is used to refer to a taxonomic representation of the systematic, semiotic options that are possible within a semiotic or lexicogrammatical system or sub-system, e.g. the system of gaze. The options are usually of the either or type (usually indicated by square brackets), although some semiotic relations are better described as scaled, marked by double-edged dotted arrows (e.g. that between high involvement potential with, or detachment from, a person or an object).

Systemic functional grammar (SFG)

is the grammatical descriptions of language developed by Michael Halliday and colleagues to explain how language is organized to make meaning. The grammatical descriptions are organized as systems of meaning, which are formulated in terms of systemic choices (the paradigm) that are mapped onto the structure of language (the syntagm or chain). These dimensions of language are called the paradigmatic and syntagmatic axes respectively, following Ferdinand de Saussure.

Systemic functional linguistics (SFL)

is the theory of language developed by Michael Halliday and extended by colleagues. Language is viewed as a social semiotic system: that is, a resource for making meaning. The functions that language has evolved to serve in society are reflected in its underlying organization. From this perspective, a major goal of SFL is to develop a systemic functional grammar (SFG) to account for the meaning-making potential of language and to apply that model to understand actual language use. SFL is thus concerned with ‘language as system’ and ‘language as text’. These same two dimensions are used in systemic functional multimodal discourse analysis (SF-MDA).

Systemic functional multimodal discourse analysis (SF-MDA)

is the systemic functional approach to multimodality based on Michael Halliday’s systemic functional theory (SFT). The approach aims to understand and describe the functions of different semiotic resources as systems of meaning, and to analyse the meanings that arise when semiotic choices combine in multimodal phenomena over space and time.

Systemic functional theory (SFT)

is the theory of meaning developed by Michael Halliday for the study of language, which resulted in systemic functional linguistics (SFL). The higher order principles of SFT provide the foundations for systemic functional multimodal discourse analysis (SF-MDA), where social semiotic resources are seen to be tools for creating meaning.

Transduction

is a social semiotic term used to refer to the re-making of meaning across modes – a process in which ‘meaning material’ is moved from one mode to another, for example, something written might be remade as a diagram, or something said might be remade as an action. This compares with the re-making of meaning by changes within the same mode, referred to as transformation. A shift across modes demands a choice of fresh semiotic resources in an endeavour to retain constancy of meaning; however this is complex as specific resources are often not modally shared (e.g. words, spelling, letter case and punctuation do not exist in image). Transduction has profound implications for meaning as it changes the resources that are available for making meaning.

Transformation

see transduction.