Beyond the Numbers: Why Qualitative Data Should Not Be Used to Make Quantifying Claims--Understanding the Real Value of Qualitative Research
Lario Viljoen (PhD student, Stellenbosch University)
Editorial note: Lario is a Stellenbosch University PhD student and SASH Fellow in the UCT Division of Social and Behavioural Studies. Her current work focuses on the ways in which young women make decisions around love, sex and romance in the context of increased access to HIV treatment in South Africa and Zambia. In this piece, she reflects on the common misuse and misunderstanding of qualitative data (particularly in health sciences research) and describes some of the strengths of qualitative research.
Introduction
Working as part of an HIV research interdisciplinary team, I’ve become increasingly aware of how some types of data and approaches are valued more than others. When it comes to working with data, quantitative data is often assumed to carry more scientific value and, in many cases, is thought of as more rigorous, objective and relevant to the world of public health than its qualitative counterpart. In contexts like these, it seems that this hierarchy is partially based on “the default assumption [that] scientifically sound research is … based [only] on randomized controlled trials” (1). The number of academic publications in health research that use qualitative data has been steadily increasing since the 1960s. However, there remains a question as to how seriously this type of work is being taken in an environment where researchers working with qualitative data feel pressure to either legitimise their data, or to try and quantitatively verify their data to make claims that are more acceptable to the broader scientific and policy environment. This seems to be a common risk for qualitative research introduced into disciplinary spheres initially ‘reserved’ for quantitative research, such as public health. The path to recognition of qualitative research as valuable in its own right has not been an easy one and the challenge is intensified when researchers who are more familiar with qualitative work want to engage with colleagues across disciplines.
The lack of detailed explanation can make the analysis process seem obscure or mystical, which further alienates researchers accustomed to quantitative data analysis and can contribute to the general mistrust in qualitative analysis and uncertainty about the value of qualitative studies.
The critique of qualitative data
While qualitative research may be seen as speaking the language of experience and subjectivity, the language of quantitative research makes claims to more finite, concrete conclusions and a higher degree of objectivity. Arguments against, and critiques of, qualitative approaches are often framed with respect to quantitative concerns. The focus of these critiques are mostly centred on: smaller sample size sizes (and related concerns about representativeness and generalisability), data reliability and validity, questions around scientific rigour, a lack of clear explanations of the analytical process, and the more visible position of the researcher or apparent lack of ‘objectivity’ (2, 3).
Generalisability
Compared to quantitative research, the smaller samples used in qualitative research are often an area of considerable critique, as are attempts to generalise the findings to larger populations. This is particularly the case in instances where researchers feel pressure to satisfy quantitative criteria and generalise qualitative findings to settings other than those studied (2). The aim of qualitative research is not to be generalisable in the same way as quantitative research seeks to be. The case for qualitative data can also be weakened when results and findings are used incorrectly. For example, recommendations from good qualitative research are too often transposed by policy makers into other sometimes inappropriate health settings, largely because they do not have proper training in the use and application of qualitative findings. Qualitative data collection methods are not designed with the aim of extrapolating results to larger populations, should not claim to do so and should not be used in this manner. Much of the value of qualitative research actually lies in its context specificity, its ability to answer questions about how and why phenomena occur, and the combined results of case-to-case examples rather than sample-to-population extrapolation (2). In addition, the analytic or theoretical transferability of qualitative research makes a significant contribution to the wider scientific knowledge base (2). If researchers are only accustomed to conventional quantitative methods though, they might not be familiar with the different strengths of qualitative research and will continue to critique its lack of generalisability.
Analysis
There are many examples of qualitative studies that note that, ‘thematic analysis was conducted’ and that ‘findings or themes emerged’ without explicitly detailing what this process involves. The lack of detailed explanation can make the analysis process seem obscure or mystical, which further alienates researchers accustomed to quantitative data analysis and can contribute to the general mistrust in qualitative analysis and uncertainty about the value of qualitative studies. Of course, qualitative analysis can also be poorly done, and even when well done, it can be poorly explained. When researchers fail to describe the analytical process thoroughly, the process can seem superficial and without scientific merit. It is imperative that researchers provide detailed accounts of the steps taken to analyse their data, as is the expectation with quantitative analysis.
Reliability/Validity
Reliability and validity are considered cornerstones of methodological rigour. Reliability is generally thought of as the consistency of evidence, and validity is the accuracy of evidence (4). With qualitative data we cannot, and should not attempt to, match these types of standards. The data are suited for different aims and need to be judged accordingly. When alternative and appropriate steps (see below) are not taken – and explicitly described when reporting results – to ensure good quality data, arguments against the use of qualitative data may be justified.
Positionality
Qualitative data collection methods have been critiqued for the lack of perceived objectivity often associated with quantitative research and the apparently standardized, neutral and non-biased tools used for data collection and reporting (5). Qualitative data collection methods and researchers tend to readily acknowledge their positionality and involvement in the research. However quantitative data collection and reporting is not actually as value-free as it is often believed to be. Sterne and Smith (6) emphasise the over-reporting of positive associations and the under reporting of the null hypothesis, the arbitrariness of significance testing and the intentional selection of associations in questionnaires consisting of hundreds of items. Further, as Temple (7) noted, “the researcher is the ultimate research 'tool' and is neither standardized, neutral nor non-biased… there is no way to remove these influences on perspective [and] all researchers can do is specify their position”. Thus, claiming objectivity or neutrality in research instruments is impossible in any kind of research, but the (intentional and acknowledged) lack of objectivity is often considered problematic for quantitative researchers.
Legitimising Qualitative Data: Strategies and Challenges
In their efforts to address the critique against qualitative research, scientists have employed several strategies. While some of these efforts have been aimed at improved quality and rigour in qualitative research (3, 4, 8), others have attempted to include quantitative data, translate their findings to fit quantitative criteria, or have adapted their research to be more in line with quantitative scientific language.
As researchers working in the field of health sciences are especially likely to communicate and work with colleagues from different disciplines, it seems that there is a need to be clear of the limitations and importantly, the strengths of the data available.
Quantifying Qualitative Data
In some cases, researchers have grouped open-ended responses and counted frequencies to produce a quantifiable number. While there are appropriate and scientifically valuable opportunities to employ quantitative analytical processes (9), these should not be considered a means for translating qualitative data for audiences who are more receptive to quantitative research claims. Attempting to transform qualitative data into quantifiable measures does little to acknowledge the value and power of in-depth qualitative data and how it can be used to make meaningful contributions in health research. Qualitative data allows for a detailed understanding of the why and the how of any issue but these rich insights are lost when they are reduced to binaries and frequencies.
Combining Quantitative and Qualitative Data Collection: Mixed Methods
Another way in which researchers have attempted to validate their research results is by using mixed methods. While some researchers have done this to strengthen qualitative findings, the process is not as simple as just combining two methodologically different data sets. Rather, it should be treated as an independent third paradigm (10) and should produce a multifaceted description of an event or issue which would reveal things the researcher could never have predicted from a survey. Again, the use of mixed data has a place in research when there is a clear connection between distinct quantitative and qualitative components, the correct analysis methods are used, and all results can be integrated comprehensively and meaningfully into more significant conclusions than could be produced from using just one method (11). Mixed methods research is valuable as a standalone paradigm and should not be used as a quick measure to appease critiques against the use of qualitative data only.
Quantitative Strategies, Including Quantitative Scientific Language
In attempts to better ‘fit’ or speak to quantitative research, health researchers in particular have employed strategies such as using quantitative terminology, implementing quantitative research criteria and adopting writing styles that are in line with quantitative research findings. For example, some researchers invoke terminology borrowed from quantitative research methods like generalisability, validity, and variables (see critiques from 2, 7, and 12 respectively) in order to make their claims seem more translatable. While these terms are well established and suitable for the quantitative, they are usually inappropriate when applied to qualitative research. In attempting to appear objective and mimicking quantitative empirical reporting, researchers may also adopt the passive voice (13) but this implicit denial of involvement serves to cast doubt on the researcher rather than rendering their research claims objective. The need to ‘speak the language’ is related to the perceived hierarchy of data and the pressure that qualitative researchers feel to establish their credentials and be convincing to those working in the dominant (quantitative) paradigm. However, qualitative data and analyses are subject to alternative standards to ensure quality and rigour (see below). Simply using quantitative terminology does not render qualitative data more relatable and instead, subverts the standards in place for good qualitative research.
Suggested Guidelines and Principles for Qualitative Data, With a Focus on Transdisciplinary Health and Development Studies
As researchers working in the field of health sciences are especially likely to communicate and work with colleagues from different disciplines, it seems that there is a need to be clear of the limitations and importantly, the strengths of the data available. Researchers need to ensure that when making use of qualitative data, their methods, analysis and questions are aligned. In addition, there should be a broader appreciation for the specific strengths and importance of qualitative research in answering questions that quantitative research cannot (14). I suggest a list of guidelines for good quality qualitative research below:
-
Pre-specified and relevant research questions: research processes should not simply happen, but should be described, planned and investigated accordingly. Despite changing plans, purposefulness is key to good research. Research questions need to be appropriate and specific to avoid non-directed data collection, superficial analysis and nondescript findings (15).
-
Using appropriate terminology: Researchers using qualitative data should make use of the correct terminology when describing methods and analysis including for example, confirmability instead of objectivity, and transferability in the place of generalisability (16) in order to avoid mistrust of scientific claims.
-
Describing the processes: The reliability of the data and analysis need to be demonstrated by researchers through properly supported argument and persuasion, which gives others confidence in the research (7).
-
Embracing the strengths of qualitative research: Rather than presenting data to fit with quantitative moulds, researchers should embrace qualitative data and analysis and the in-depth detail about beliefs and perceptions that it provides. These “thick” descriptions contribute greatly to context-specific case-to-case reasoning (2).
-
Actual excerpts or examples of data being provided: Data should be presented and described clearly and thoroughly so that inferences can be made on the basis of these results (11). Without access to some representation of the data, readers are unable to assert whether the analysis is rigorous or reliable.
In conclusion, instead of attempting to legitimise qualitative data through quantitative criteria, researchers should conduct good quality research in all methodologies and communicate this well, so that research claims can be translated across health disciplines. Rather than seeing quantitative and qualitative data or research methodologies as competing, the strengths of each method should be embraced and valued within its appropriate context.
References
1. Alasuutari, P. 2010. The rise and relevance of qualitative research. International journal of social research methodology, 13(2): 139-155.
2. Firestone, W.A. 1993. Alternative arguments for generalizing from data as applied to qualitative research. Educational researcher, 22(4): 16-23.
3. Antaki, C., Billig, M., Edwards, D. and Potter, J. 2003. Discourse analysis means doing analysis: a critique of six analytic shortcomings. Discourse Analysis Online, 1.
Available at: http://www.shu.ac.uk/daol/articles/v1/n1/a1/antaki2002002-t.html.
4. Boulton, M., Fitzpatrick, R. and Swinburn, C. 1996. Qualitative research in health care: a structured review and evaluation of studies. Journal of Evaluation in Clinical Practice. 2: 171- 179.
5. Mason, J. 1996. Qualitative Researching. London: Sage Publications.
6. Sterne, J.A. and Smith, G.D. 2001. Sifting the evidence—what's wrong with significance tests? Physical Therapy, 81(8): 1464-1469.
7. Temple, B. 1998. A fair trial? Judging quality in qualitative research. International Journal of Social Research Methodology, 1(3): 205-215.
8. Tracy, S.J. 2010. Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qualitative inquiry, 16(10): 837-851.
9. Scherp, H. 2013. Quantifying qualitative data using cognitive maps. International Journal of Research & Method in Education, 36(1): 67–81.
10. Johnson, R.B., Onwuegbuzie, A.J. and Turner, L.A. 2007. Toward a definition of mixed methods research. Journal of mixed methods research, 1(2): 112-133.
11. Creswell, J. W. and Tashakkori, A. 2007. Developing publishable mixed methods manuscripts. Journal of Mixed Methods Research, 1: 107-111.
12. Adcock, R. and Collier, D. 2001. Measurement validity: A shared standard for qualitative and quantitative research. American political science review, 95(3): 529-546.
13. Douglas, M., 2011. In the Active Voice. Oxon: Routledge.
14. Daniels, K., Loewenson, R., George, A., Howard, N., Koleva, G., Lewin, S., Marchal, B., Nambiar, D., Paina, L., Sacks, E. and Sheikh, K. 2016. Fair publication of qualitative research in health systems: a call by health policy and systems researchers. International journal for equity in health, 15(1): 98.
15. Cohen, D. J. and Crabtree, B. F. 2008. Evaluative criteria for qualitative research in health care: Controversies and recommendations. Annals of Family Medicine, 6(4).
16. Malterud, K. 2001. Qualitative research: standards, challenges, and guidelines. The Lancet, 358(9280): 483-488.
Author Biography
Email: lario@sun.ac.za