Further information

An extended discussion of the concept of ecological validity

In Chapter 4 there is a discussion of the much misused and poorly understood concept of ecological validity. This is the original discussion which I trimmed down for the book.

Ecological validity

In this edition I decided to fully discuss the meaning of this enigmatic and catch-all term ‘ecological validity’ because its widespread and over-generalised use has become somewhat pointless. Hammond (1998) refers to its use as ‘casual’ and ‘corrupted’ and refers to the robbing of its meaning (away from those who continue to use its original sense) as ‘bad science, bad scholarship and bad manners’.

There are three relatively distinct and well used uses of the term, which I shall call ‘the original technical’, ‘the external validity version’ and ‘the pop version’, the latter term to signify that this use I would consider to be unsustainable since it has little to do with validity and its indiscriminate use will not survive close scrutiny.

1. The original technical meaning  

Brunswik (e.g., 1947) introduced the term ecological validity to psychology as an aspect of his work in perception ‘to indicate the degree of correlation between a proximal (e.g., retinal) cue and the distal (e.g., object) variable to which it is related’ (Hammond, 1998). This is a very technical use. The proximal stimulus is the information received directly by the senses – for instance two lines of differing lengths on our retinas. The distal stimulus is the nature of that actual object in the environment that we are receiving information from. If we know that the two lines are from two telegraph poles at different distances from us we might interpret the two poles as the same size but one further away than the other. The two lines have ecological validity in so far as we know how to usefully interpret them in an environment that we have learned to interpret in terms of perspective cues. The two lines do not appear to us as having different lengths because we interpret them in the context of other cues that tell us how far away the two poles are. In that context their ecological validity is high in predicting that we are seeing telegraph poles. More crudely, brown patches on an apple are ecologically valuable predictors of rottenness; a blue trade label on the apple tells us very little about rot.  

2. The external validity meaning

Many textbooks, including this one, have taken the position that ecological validity is an aspect of external validity and refers to the degree of generalisation that is possible from results in one specific study setting to other different settings. This has usually had an undertone of comparing the paucity of the experimental environment with the greater complexity of a ‘real’ setting outside the laboratory. In other words researchers asked ‘how far will the results of this laboratory experiment generalise to life outside it?’ The general definition, however, has concerned the extent of generalisation of findings from one setting to another and has allowed for the possibility that a study in a ‘real life’ setting may produce low ecological validity because its results do not generalise to any other setting – see the Hofling study below. Most texts refer to Bracht and Glass (1968) as the originators of this sense of the term and the seminal work by Cook and Campbell (1979) also supported this interpretation.

On this view effects can be said to have demonstrated ecological validity the more they generalise to different settings and this can be established empirically by replicating studies in different research contexts.

3. The ‘pop’ version     

The pop version is the definition very often taught on basic psychology courses. It takes the view that a study has (high) ecological validity so long as the setting in which it is conducted is ‘realistic’, or the materials used are ‘realistic’, or indeed if the study itself is naturalistic or in a ‘natural’ setting (e.g., Howitt, 2013). The idea is that we are likely to find out more about ‘real life’ if the study is in some way close to ‘real life’, begging the question of whether the laboratory is not ‘real life’.   

The problem with the pop version is that it has become a knee-jerk mantra – the more realistic the more ecological validity. There is, however, no way to gauge the extent of this validity. It is just assumed, so much so that even A-level students are asked to judge the degree of ecological validity of fictitious studies with no information given about successful replications or otherwise.

Teaching students that ecological validity refers to the realism of studies or their materials simply adds a new ‘floating term’ to the psychological glossary that is completely unnecessary since we already have the terminology. The word to use is ‘realism’. As it is, students taught the pop version simply have to learn to substitute ‘realism’ when they see ‘ecological validity’ in an examination question. 

For those concerned about the realism of experimental designs Hammond (1998) points out that Brunswick (1947) introduced another perfectly suitable term. He used representative design to refer to the need to design experiments so that they sample materials from among those to which the experimenter wants to generalise effects. He asked that experimenters specify in their design the circumstances to which they wished to generalise. For instance, in a study on person perception, in the same way as we try to use a representative sample of participants, we should sample a representative sample of stimulus persons (those whom participants will be asked to make a judgment about) in order to be able to generalise effects to a wider set of perceived people. Hammond is not the only psychologist worried about the misuse of Brunswik’s term. Araújo, Davids and Passos (2007) argue that the popular ‘realism’ definition of ecological validity is a confusion of the term with representative design:

‘… ecological validity, as Brunswik (1956) conceived it, refers to the validity of a cue (i.e., perceptual variable) in predicting a criterion state of the environment. Like other psychologists in the past, Rogers et al. (2005) confused this term with another of Brunswik’s terms: representative design.’ (p.69)

This article by Araújo et al is a good place to start understanding what Brunswik actually meant by ecological validity and demonstrates that arguments to haul its meaning back to the original are contemporary and not old-fashioned. The term is in regular use in its original meaning by many cognitive psychologists. They are not clinging to a ‘dinosaur’ interpretation in the face of unstoppable changes in the evolution of human language. 

Milgram v. Hofling – which is more ‘ecologically valid’?

Another problem with the pop version is that it doesn’t teach students anything at all about validity as a general concept. It simply teaches them to spot when material or settings are not realistic and encourages them to claim that this is a ‘bad thing’. It leads to confusion with the laboratory–field distinction and a clichéd positive evaluation of the latter over the former. For example, let’s compare Milgram’s famous laboratory studies of obedience with another obedience study by Hoflinget al (1966), where nurses working in a hospital, unaware of any experimental procedure, were telephoned by an unknown doctor and broke several hospital regulations by starting to administer, at the doctor’s request, a potentially lethal dose of an unknown medicine. The pop version would describe Hofling’s study as more ‘ecologically valid’ because it was carried out in a naturalistic hospital setting on real nurses at work. In fact, this would be quite wrong in terms of external validity since the effect has never been replicated. The finding seems to have been limited to that hospital at that time with those staff members. A partial replication of Hofling’s procedures failed to produce the original obedience effect (Rank and Jacobson, 1977¹), whereas Milgram’s study has been successfully replicated in several different countries using a variety of settings and materials. In one of Milgram’s variations, validity was demonstrated when it was shown that shifting the entire experiment away from the university laboratory and into a ‘seedy’ downtown office, apparently run by independent commercial researchers, did not significantly reduce obedience levels. Here, following the pop version, we seem to be in the ludicrous situation of saying that Hofling’s effect is valid even though there is absolutely no replication of it, while Milgram’s is not, simply because he used a laboratory! In fact Milgram’s study does demonstrate ecological validity on the generalisation criterion. The real problem is that there is no sense of ‘validity’ in the pop notion of ecological validity. 

In a thorough discussion of ecological validity Kvavilashvili and Ellis (2004) bring the original and external validity usages together by arguing that both representativeness and generalisation are involved, with generalisation appearing to be the more dominant concept. Generalisation improves the more that representativeness is dealt with. However, they argue that a highly artificial and unrealistic experiment can still demonstrate an ecologically valid effect. They cite as an example Ebbinghaus’s memory tasks with nonsense syllables. His materials and task were quite unlike everyday memory tasks but the effects Ebbinghaus demonstrated could be shown to operate in everyday life, though they were confounded by many other factors. The same is true of research in medicine or biology; we observe a phenomenon, make highly artificial experiments in the laboratory (e.g., by growing cultures on a dish) then re-interpret results in the everyday world by extending our overall knowledge of the operation of diseases and producing new treatments. In psychology, though, it is felt that by making tasks and settings more realistic we have a good chance of increasing ecological validity. Nevertheless, ecological validity must always be assessed using research outcomes and not guessed at because a study is ‘natural’. 

I think the conclusions that emerge from this review of the uses of ecological validity are that:

• Examiners (public or institutional) should certainly not assess the term unless they are prepared to state and justify explicitly the specific use they have in mind prior to any examinations.
• The pop version tells us nothing about formal validity and is a conceptual dead end; ‘realism’ can be used instead and ‘ecological validity’ takes us no further.
• Rather than ‘ecological validity’ it might be more accurate to use the term ‘external validity concerning settings’. Although in 1979 Cook and Campbell identified ecological validity with generalisation to other settings (i.e., external validity), in their update of their 1979 classic, Shadish, Cook and Campbell (2002) talk of external validity with regard to settings. They seem to pass the original term back to Brunswik saying that external validity is often ‘confused with’ ecological validity. By contrast, Kvavilashvili and Ellis (2004) argue that ‘the difference between the two concepts is really small’. Obviously we cannot hope for pure agreement among academics!
• It is ridiculous to assume that on the sole basis that a study is carried out in a natural setting or with realistic materials it must be in some way more valid than a laboratory study using more ‘artificial’ materials. Validity is about whether the effect demonstrated is genuinely causal and universal. An effect apparently demonstrated in the field can easily be non-genuine and/or extremely limited, as was Hofling’s.
• The pop version cannot be sustained scientifically and is not of much use beyond being a technical sounding substitute for the term ‘realism’. The original version is still used correctly by those working in perception and related fields. The external validity (generalising) version is favoured by Kvavilashvili (over representativeness), and directs attention to a useful aspect of validity in the design of research. However, the external validity version is challenged by authors such as Hammond (1998) and Araújo et al (2007), who claim that this is not at all what Brunswik meant nor is it the way cognitive psychologists use the term. Perhaps it’s better to lie low, use alternative terminology, and see how the term evolves. I rather sense that the pop version will hang around, as will complete misunderstanding of the terms ‘null hypothesis’ and ‘negative reinforcement’.

_{1. Unlike in Hofling’s study, nurses were familiar with the drug and were able to communicate freely with peers.}

Festinger's end-of-the-world study 'Under Cover'

In a famous participant observation case study, Leon Festinger, Henry W. Reiken and Stanley Schachter (1956) studied a woman they called Mrs Keech who had predicted the end of the world by a mighty flood. After reading about Mrs Keech in a newspaper, Festinger’s group joined her followers to see what would happen when the world did not, in fact, come to an end. 

Mrs Keech claimed to have been receiving messages from a group called the Omnipotent Guardians from the planet Clarion. They had sent her messages through a combination of telepathy, automatic writing and crystal ball gazing to indicate that at midnight on a particular December evening the world would be destroyed, killing all humanity except for Mrs Keech’s group. They were to be rescued by flying saucers sent from Clarion to Mrs Keech’s home. During the weeks before the ‘end of the world’, several of the group members quit their jobs and spent their savings in preparation for the end. Messages continued to arrive daily to Mrs Keech. At meetings Festinger and his associates frequently would excuse themselves and write down their notes while in the bathroom. At one meeting, the members were asked to look into a crystal ball and report any new pieces of information. One member of Festinger’s group was forced to participate, even though he was hesitant to take a vocal role in meetings. After he remained silent for a time, Mrs Keech demanded that he report what he saw. Choosing a single word response, he truthfully announced, ‘Nothing’. Mrs Keech reacted theatrically, ‘That’s not nothing. That’s the void.’

On the final evening, members of the group waited for midnight. During the evening other instructions arrived. Cultists were told to remove their shoelaces and belt buckles since these items were unsafe aboard flying saucers. When midnight passed without any end of the world in sight – and without any flying saucers visible – members of the group began questioning whether they had misunderstood the instructions. Mrs Keech began to cry and whimpered that none of the group believed in her. A few of the group comforted her and reasserted their belief in her. Some members re-read past messages, and many others sat silently with stony expressions on their faces. Finally, in the wee hours of the morning, Mrs Keech returned to the group with a new ‘automatically written’ message from the Omnipotent Guardians. Because of the faith of the group, the Earth had been spared. The cult members were exuberant and during the weeks that followed actually attempted to secure additional converts. 

One can find both insight and some entertainment in such a participant observation study. To Festinger’s group of scholars, the experience permitted them to provide a field study examination (in the form of a case study) of the theory of cognitive dissonance that they were trying to develop at the time – a theory that became a major force in communication and social psychology. 

Excerpted from John C. Reinard, Introduction to Communication Research. Boston, McGraw-Hill, 2001, pp 186–8.

Analysing qualitative data

The Jefferson transcription system

Chapter 12 of the book promises to provide some details of the Jefferson transcription system, used to transcribe recorded conversations into text. Here they are. First of all though a few points about the system.

1. The system is used thoroughly in conversation analysis (see Hutchby and Wooffitt, 1998 – reference is in the book). In this approach there are two major concerns – those of turn-taking and of the characteristics of speech delivery. As you can see from the symbols below, many of these concern how a speech partner overlaps or take over from another (e.g., simply the fact of transcription line numbers) and how speech is presented (emphasis, rising tone, pauses and so on). As an exercise listen to a conversation and observe how people use pauses in order to ‘keep the floor, i.e., indicate that they want to go on speaking and not be interrupted. In some Australian and Californian accents there is a marked up turn at the end of an utterance when the speaker indicates the end of their speech and invites a reply.

2. The most important point is that the system is intricate and time hogging so check first that you really do need to use it! Most grounded theory approaches would not transcribe in this kind of detail. The system is used mainly in conversation analysis where researchers are interested in the analysis of para-linguistics, in a nutshell, not what is said but how it is said, for instance, emphasis, rising and falling tones, hesitations, etc. Such hesitation might indicate, for instance, that the speaker is embarrassed or in some way has a problem with what they are saying. You can of course indicate that there was such hesitation without resorting to the details of Jefferson.

3. Any two transcribers will come up with slightly different transcriptions. There is no clear criterion as to what counts as a marked or ‘less marked’ fall in pitch, for instance.

4. Each new utterance (according to the researcher’s hearing) is given a new line number in the transcript.

5. You would need a recording system that permits you to constantly rewind very short sections of speech. Researchers have traditionally used transcription machines although computer programmes can now do the job quite well.

Largely based on Hutchby and Woffitt (1998) p. vi

A few of the symbols can be seen in this conversation sequence.

1  A: gmornin:. j’sleep O­K
2  B: huh
3  A: was that a yes
4  B: hhh° yer °
5  A: HALL¯O:: (0.5) are we awake.
6  B: <just about>
7  A: >look we really need to talk [about the holiday<
8  B:                                              [I know (.) the holiday

If you’d like to get practice specifically with conversation analysis or at least with this kind of transcription you could visit Charles Antaki’s Conversation Analysis website at:

http://homepages.lboro.ac.uk/~ssca1/sitemenu.htm

Where you can go through a script and look at various stages of transcription ending with a version using the notation system above.

Sod's Law – or Murphy’s law as the Americans more delicately put it

A discussion of Sod’s law – a BBC spoof documentary about testing the notion that toast always falls butter side down and other issues.

The sociologist’s chip shop

An attempt to exemplify the concepts of the null hypothesis and significance in an everyday homely tale of chips.

Please, sir, may we use a one-tailed test, sir?

A discussion of the arguments for and against the use of one-tailed tests in statistical analysis in psychology. 

Calculating effect sizes and power in a 2-way ANOVA

In Chapter 21 of the book we calculated a two-way ANOVA on the data that are provided here in Exercise 1. The book tells you that you can obtain effect size and power using SPSS or G*Power, but that the by-hand calculations are rather complex. For the sake of completeness though I will give the detail here.

Remember we are dealing with an experiment where participants consume either strong coffee, decaffeinated coffee or nothing. These conditions are referred to as caffeine, decaff and none. Two groups of participants are tested, those who have just had five hours' sleep and those who have been awake for a full 24 hours. We therefore have a 2 (sleep) x 3 (caffeine) design. The results Table 21.2 from the book is:

Skill scores

Caffeine factor means x¯c1 = 5.125 x¯c2 = 4.25 x¯c3 = 4.0625 x¯g = 4.48 (grand mean)
Σx = 215
Σx2 = 1081
(Σx)2 = 46225
(Σx)2/N = 963.02 = C

* Overlong decimal figures are used here in order that our figures come close to those given by SPSS. With sensible rounding, our ANOVA results would be more different from the SPSS result than they are.

Table 21.2: Driving skill scores by caffeine and sleep conditions

Main effects

The general rules for calculating effect size and power for main effects in a two-way design are as follows, where we will refer to one factor with the general term A and the other with the term B.

For factor A, = where αi is the difference between the grand mean and each mean for factor A ignoring any difference across factor B. Ideally, these means would be the true population means if known but in calculating power after an experiment we use the sample means and assume these are good estimates of the population means. For our caffeine/sleep study, then, the grand mean is 4.48 and the α values would be the differences between 4.48 and each value, that is, 5.13 (caffeine), 4.25 (decaff) and 4.06 (none). Each of these differences is squared, the results added and this result for the top of the equation is divided by a, the number of levels of A (three in the caffeine case) multiplied by the mean square for error (MSE) from the ANOVA calculation.

Calculating for the caffeine effect we get:

= 0.321

Our effect size, also referred to (by Cohen, 1988) as f, is 0.321. If we want to consult tables then we now need Φ which = in general terms but with a factorial ANOVA we substitute n' for n where =1 ¹. Our df are broken down like this:

Total df = N – 1 = 47
Main effect (caffeine) df = 2
Main effect (sleep) df = 1
Interaction df = 2 x 1 = 2
Error df = 47 – 2 – 1 – 2 = 42

Hence and Φ = 0.321√15 = 1.24

We go to appendix Table 13 with Φ = 1.24, df1 = 2 and dfe = 42 and α = 0.05. With a bit of extrapolation, we find that power is around 0.48 (don't forget that power = 1 – β). This is close enough to SPSS and G*Power which both agree on 0.471.

As explained in the book SPSS provides effect size and power if you select this before your analysis and in G*Power select F tests, ANOVA: Fixed effects, special, main effects and interactions and Post hoc: Compute achieved power - given α, sample size and effect size. The values to enter are 0.321 for effect size, .05 for α err probability, 48 for total sample size, 2 for numerator df and 3 for number of groups.

Interaction effects

For the interaction things are a bit tricky. The formula is .

The calculation of ab is carried out for each individual cell of the data table. The calculation is: where is the cell mean (5 hours/caffeine if working from the top left in table 1 above), is the mean of the caffeine condition which that cell is in (caffeine and 5.125 in this case), is the mean for the sleep condition of that cell (5 hours and 4.7917) and finally is the grand mean. Each of these six values (one for each of the cells in the table) is squared and the results are added together. This sum is divided by n' times the MSE where n' is calculated as above.

Put perhaps more simply for each cell you subtract from the cell mean the mean of the row it is in and the mean of the column it is in and then add the grand mean. Square the result and divide by n' times the MSE. Let's do this now. The calculation may look horrific but if you stick to the rule just stated you should be able to follow each step:

1. top line of the fraction inside the square root sign is:

(6.25 – 5.125 – 4.7917 + 4.48)2 + (4 – 5.125 – 4.1667 + 4.48)2 + (4.125 – 4.25 – 4.7917 + 4.48)2 +
(4.375 – 4.25 – 4.1667 + 4.48)2 + (4 – 4.0625 – 4.7917 + 4.48)2 + (4.125 – 4.0625 – 4.1667 + 4.48)2

Which comes to 1.983!

Dividing this by we get: 1.983/(2 x 3 x 2.074) = 0.159

The square root of 0.159 is 0.399. This is our value for Φ' (or f)

We need to calculate n' and we use as above but here is for the interaction and hence dfcaffeine x dfsleep which is 2 x 1 = 2. Hence n' will again be 15.

Φ then is Φ' x √n' = 0.399 x √15 = 1.54

Going to appendix Table 13 with Φ = 1.54, df for the interaction being 2, dferror = 42 and α = 0.05 we need (again) to do some extrapolation between the table values but I get a table value of around 0.35 and, remembering that power = 1 – β this gives power at 0.65 which is pretty close to the SPSS and G*Power values of 0.662. To use G*Power proceed exactly as given above but for the number of groups enter 6.