«Abstract One of the reasons that research is conducted is to build the evidence base to inform strategic or policy directions. In this context, the ...»
Generalising from Qualitative Research: Case studies from VET in
Ian Falk & John Guenther, Charles Darwin University
One of the reasons that research is conducted is to build the evidence base to inform
strategic or policy directions. In this context, the value of qualitative research is often
questioned because ‘you cannot make generalisations from results when the sample is
not statistically representative of the whole population in question’. However, a scan
of the research literature in the field of Vocational Education and Training (VET) reveals a considerable amount of qualitative research which is used for this very purpose even though much of the headline data is in the form of numerical statistics based on sampling regimes. Can findings from qualitative research legitimately be generalised and applied beyond the frame of a particular case or even a set of 100 semi-structured interviews on a particular topic? Are there features within qualitative methods that justify generalisable inferences?
The paper stems from the research experience of the authors over the last two decades, during which time we have, as it turns out, been living with a dichotomy. On the one hand, we were taught in our research training that you can’t generalise much from qualitative research, if at all. On the other hand, what has emerged for us is that, first of all, people do generalise from qualitative research; and second, we suggest that we may well have good reason to be able to do so. By ‘good reason’, we mean that the generalised decisions that are made on the basis of the findings of qualitative research are sound, that the findings have indeed been generalised successfully. That is, when the findings have been applied more generally, it has been found that the generalising has proved valid and reliable.
Introduction One of the reasons that research is conducted is to build the evidence base to inform strategic or policy directions. In this context, the value of qualitative research is often questioned because ‘you cannot make generalisations from results when the sample is not statistically representative of the whole population in question’. However, a scan of the research literature in the field of VET reveals a considerable amount of qualitative research which is used for this very purpose even though much of the headline data is in the form of numerical statistics based on sampling regimes. Can findings from qualitative research legitimately be generalised and applied beyond the frame of a particular case or even a set of 100 semi-structured interviews on a particular topic? Are there features within qualitative methods that justify generalisable inferences?
We would like to thank the Australian National Training Authority (ANTA), the National Council for Vocational Education Research (NCVER), the Northern Territory Council of Social Service (NTCOSS) and the Northern Territory Department of Employment Education and Training (NTDEET) who funded the research on which this paper draws.
-1This paper stems from the authors’ experiences and defences of generalisations made from findings of qualitative and mixed methods research. It draws on an extensive literature review conducted for an earlier paper (Falk and Guenther 2006) and explores some examples from research carried out in the field of vocational learning over a number of years, that highlight the issues.
Posing the question: can we generalise from qualitative research?
There seem to be three sometimes overlapping views in the research literature about qualitative research with regard to its generalisability or not. (a) One is a more-or-less acceptance that generalisability is not the main purpose of qualitative research, but there are plenty of other good reasons for employing it (e.g. Myers 2000:2); (b) The second view is that, yes, you can generalise, but if you do, you have to issue cautions about the limited capacity to do so based on the limited numbers (e.g., Benz and Newman 1998), a view which in part inherently accepts the scientific paradigm’s rules and constructs about ‘good research’; and (c) The third view, named by Stake (1980) in reference to case study research, is one of formalising the idea that qualitative research is generalisable.
Generalisability defined and debated Generalisability refers to the degree to which research findings are applicable to other populations or samples (Polit and Hungler 1991; Ryan and Bernard 2000). It involves ‘the usefulness of one set of findings in explaining other similar situations’ (Grbich 1999:66). Generalising is ‘central to the definition and creation of valid public knowledge’ (Metcalfe 2005). It is sometimes equated with terms of ‘transferability’ and ‘external validity’ (Tashakkori and Teddlie 2003).
Since the 1990s, qualitative methods have become more common in disciplines such as education, social work, health services and evaluation research, with an increase in qualitative research studies in professional journals (Schofield 1993; Boulton and Fitzpatrick 1994; Blaxter 1996; Mays and Pope 2000). There is however considerable debate over the nature of the knowledge produced by qualitative methods and whether a term such as generalisability, derived from the quantitative paradigm, can mean the same when used to judge the rigour of qualitative research design, or whether a completely different term should be applied.
Some authors doubt that generalisability can be achieved in qualitative research.
Lincoln and Guba (1985:110) say: ‘The only generalization is: there is no generalization.’ Others emphasise the context-specificity of qualitative research (Wainwright 1997), which limits generalization to other situations (Creswell 1998).
Hammersley (1990:108) says that ethnographers are generally ‘not very effective in establishing the typicality of what they report. And in the absence of such information we must often suspend judgement about the generalisability of their claims’. The literature review now outlines the main strands of this debate on generalisability.
Reconceptualising generalisability It has been argued that qualitative research represents a distinctive paradigm and as such should not be judged by conventional measures of generalisability, or validity and reliability (Hammersley 1990). In qualitative research the focus, stemming from
-2research traditions in the social sciences and the arts, is on discovery (Hamberg et al.
1994). Cronbach (1975:124) concludes that social phenomena are too context-specific to permit generalisability. He suggests the priority of qualitative research is to ‘appraise a practice or proposition… in context’. Denzin (1983:133) also rejects generalisability as a goal: ‘every instance of social interaction, if thickly described, represents a slice from the life world’ and is thus a proper subject matter.
Donmoyer (1990) looks to schema theory and its concepts of assimilation, accommodation, integration and differentiation (Piaget 1971) for language to characterise how generalisability occurs in experiential learning. He believes that applying this language to qualitative research gives it ‘far more utility for applied fields... than was traditionally believed’ (p. 198). Patton (2002) finds another term instead of generalisability: ‘extrapolations... modest speculations on the likely applicability of findings to other situations’ (p. 584), which may be made from qualitative research. Creswell (2005:48) also uses another term: ‘In qualitative research, the... interpretation consists of stating the larger meaning of the findings’.
Metcalfe (2005) says priorities for generalising knowledge differ across the qualitative paradigm, and authors need to make their priorities explicit. In this way, he believes that ‘the debate on the quality of qualitative research might be both improved and better appreciated relative to other epistemologies’.
‘Naturalistic’ generalisation: reader interpretation, researcher participation Qualitative research is ‘very much influenced by the researcher’s individual attributes and perspectives’ (Schofield 1993:202). Stake (1980:64) suggests that qualitative methods may provide a vicarious link with the reader’s experience and thus be a natural basis for generalization. As Lincoln and Guba (1985:217) say: ‘the final judgment... is … vested in the person seeking to make the transfer’. This process involves reciprocity as the researcher, too, ‘is always a subject in qualitative research’ (Hamberg et al. 1994:177). The grounded theory approach to data analysis (Glaser and Strauss 1967) suggests that all explanations or theories are derived from the dataset rather than from a researcher’s viewpoint, but elsewhere Strauss stresses the importance of researchers’ taking advantage of earlier experiences for enhancing ‘theoretical sensitivity’ (Strauss 1987: 21). Enhanced knowledge is gained through the active participation of the researcher in a process which has been described as a participating-inductive model (Hamberg et al. 1994). Other authors agree that all research involves subjective perception and that different methods produce different perspectives, but argue that there is still an underlying reality which can be studied (Kirk and Miller 1986; Hammersley 1992).
Design and validity Maxwell (1992) identifies generalisability as one of five types of validity emerging from qualitative research methodology. Generalisability aligns with other features, which are: descriptive validity (factual accuracy), interpretive validity (understanding of the perspective of the group under study), theoretical validity (the “fit” of data and theoretical explanation), evaluative validity (application of an evaluation framework.
Maxwell identifies an internal and external generalisability. Internal generalisability applies within the setting or group studied; external generalisability applies beyond the group, setting, context, or time (Onwuegbuzie and Leech 2005). It is elsewhere called external reliability (Kincheloe and McLaren 2000). Patton (2002:230) advises
-3selecting information-rich study sites and participants: ‘those from which one can learn a great deal about issues of central importance to the purpose of the inquiry’.
Sampling and describing A degree of generalisability can be achieved by ensuring that the research report is sufficiently detailed for the reader to be able to judge whether or not the findings apply in similar settings (Mays and Pope 2000). Detailed description should reveal the social relations that underpin it (Wainwright 1997). Generalisability may be enhanced by choosing a research site on the basis of typicality, or by using a multi-site methodology, but thick or rich description is vital (Schofield 1993)—it shows ‘that the researcher was immersed in the setting and [gives] the reader enough detail to ‘make sense’ of the situation’ (Firestone 1987:16).
Some authors (e.g., Firestone 1987; Mays and Pope 2000; Silverman 2001;
Onwuegbuzie and Leech 2005) advocate combining qualitative research with quantitative measures of populations, purposive sampling and theoretical sampling.
Combining sampling strategies may be used within a single method or mixed method research design (Kemper et al. 2003).
Mixed methods This combination of methods—often referred to as ‘mixed methods’—does a lot more than ‘fill in the gaps’ of one method or the other. Methods can be combined in a variety of ways: a) through the ‘quantitization’ (Tashakkori and Teddlie 1998) of qualitative data (for example collating and counting recurrent themes in the qualitative data) in order to add ‘legitimacy to the researchers’ conclusions’ (Onwuegbuzie and Teddlie 2003:356); b) by accessing complementary quantitative data from within the same sample (for example through use of quantitative survey instruments complementing interview data) in what could be described as a ‘concurrent triangulation strategy’ (Creswell 2003) and may incorporate ‘multilevel mixed sampling’ (Kemper et al. 2003:287) and c) by drawing on data that comes from outside the purposive sample frame (for example using national or large sample surveys on related topics) to compare the ‘accessible population’ with a ‘target population’ possibly for the purpose of ‘identifying the population to which a finding can and cannot be made’ (Johnson and Christensen 2004:244-245). This approach uses what is sometimes referred to as ‘sequential mixed methods sampling’ (Teddlie and Yu 2007). While this may be an oversimplification of their uses—certainly the literature describes several other ways of looking at different mixed methods approaches (e.g. Tashakkori and Teddlie 1998; Miller 2003; Tashakkori and Creswell 2007), mixed methods allow researchers to on the one hand make deductions from empirical data (most often the quantitative data) while at the same time testing these deductions with the inferences that emerge (most often from the qualitative data)— and vice versa—to both test hypothesis and build theory (Erzberger and Kelle 2003).
This combination effectively validates the findings of both data sources.
The role of theory Generalization is closely involved with theory. Johnson and Christensen (2004) say, ‘A well-developed theory explains how something operates in general... and it enables one to move beyond the findings of any single research study (p. 19). Yin (2003b) says analysts should generalise findings to theory, ‘analogous to the way a scientist generalizes from experimental results to theory’ (p. 38). Indeed Johnson and
-4Christensen suggest that the only difference between qualitative and quantitative researchers is the starting point of the research on a ‘research wheel’. Figure 1 explains this diagrammatically.
Figure 1. The ‘research wheel’, adapted from Johnson and Christensen (2004:18)
The theory then becomes the vehicle for examining other cases. Yin (2003b:32) calls this role of theory ‘analytic generalization’ (in contrast to statistical generalization).
Maxwell (1992) also believes the generalisability of qualitative data occurs through the development of theory from the data—a theory that can be applied to similar persons in similar situations.