Scientific knowledge is generally disseminated in peer-reviewed journals. However, this does not mean that everything that gets published is necessarily meeting the highest standard of scientific rigor. We need to be able to sieve through articles and decide for ourselves whether the paper’s topic is relevant and the findings are trustworthy. There are certain strategies that can help when critiquing research, our own or that of others.
Since quantitative and qualitative studies involve two different types of research processes with dissimilar research philosophies and methodologies, the two are often evaluated against different criteria. For example, while a single participant can sometimes suffice for a good qualitative research project, the sample needs to be a lot larger for a quantitative study to be deemed valid and publishable. To clarify the difference between the two methodologies, you can read this post.
Here we outline a quick checklist you can use when doing an appraisal of a qualitative paper:
- Is the article addressing an important clinical or theoretical problem?
The literature review (and/or the background of the study) should provide you with enough information to convince you that the research is important. Simply stating that ‘there is a knowledge gap’ is not sufficient. The article needs to include a comprehensive review of the field of study so that the reader knows where this research fits, and what it helps to address.
- Is the purpose of the study clear?
The purpose of the study is usually presented already in the abstract. It can then be elaborated upon in the introduction section, for example, via a research question.
- Is qualitative methodology best suited for the problem?
This is an important consideration. Qualitative methodology shouldn’t be used just because the researcher is familiar with it; there should be a clear connection between the problem and the research methodology. Qualitative methodology is usually applied when we do not know a lot about a certain phenomenon. Qualitative methods are good when we want to elicit and understand people’s experiences with what we are studying. It is not meant for studies of cause-and-effect relationships. Therefore, if the purpose of a study was to establish the effects of a certain intervention, qualitative methodology would not be appropriate.
- Is the research methodology congruent with data collection methods?
Similar to the previous point, the article needs to show congruency between its different components. If the methodology is qualitative, you expect to find data collection methods such as interviews, focus groups, observations. On the other hand, using a questionnaire could raise concerns that the participants’ voices were not really heard and the reader is not going to get the rich descriptions one would expect from a qualitative study.
- Has a rich description of the study been provided?
It is very important for a qualitative report to describe the setting, participants and data collection in detail. Since the findings cannot be generalized to the general public, we need to know exactly who (and how) they relate to the project, as well as what kind of a process took place.
- Is the analysis well described?
It should be clear how the text was analyzed and who performed the analysis. If technology was used, what kind?
- Does the findings section reflect the experiences of the participants?
The findings write up should demonstrate that the conclusions of the study were primarily reached based on what the participants expressed, showed and shared. Usually, this section will include direct quotations from data collection to substantiate the points made by the researcher.
- Do the findings go beyond the descriptive level?
Too often, novice qualitative researchers stay at the descriptive level when they present their findings. Qualitative research should reveal more abstract aspects of the phenomenon and dive deep to help the reader understand them.
- Do the findings answer the research question(s)?
At the end, the reader should be certain that the purpose of the study was fulfilled.
- Have the findings been discussed against other research in the field?
There should be a discussion section that meaningfully summarizes the findings and juxtaposes them against other studies, as well as demonstrates the study’s contribution to the field. Alternative explanations should be reviewed. The conclusion of the article needs to include the study’s limitations.
These points should also be considered when we are writing up our own qualitative research. It is best to be our own devil’s advocate and read our own work as if we were a potential reviewer appraising the quality of our writing.