Tools for Critical Appraisal of Studies
“The purpose of critical appraisal is to determine the scientific merit of a research report and its applicability to clinical decision making.”1 Conducting a critical appraisal of a study is imperative to any well executed evidence review, but the process can be time consuming and difficult.2 The critical appraisal process requires “a methodological approach coupled with the right tools and skills to match these methods is essential for finding meaningful results.”3 In short, it is a method of differentiating good research from bad research.
Critical Appraisal by Study Design (featured tools)
- AMSTAR 2The original AMSTAR was developed to assess the risk of bias in systematic reviews that included only randomized controlled trials. AMSTAR 2 was published in 2017 and allows researchers to “identify high quality systematic reviews, including those based on non-randomised studies of healthcare interventions.”4
- ROBISROBIS is a tool designed specifically to assess the risk of bias in systematic reviews. “The tool is completed in three phases: (1) assess relevance(optional), (2) identify concerns with the review process, and (3) judge risk of bias in the review. Signaling questions are included to help assess specific concerns about potential biases with the review.”5
- BMJ Framework for Assessing Systematic ReviewsThis framework provides a checklist that is used to evaluate the quality of a systematic review.
- CASP Checklist for Systematic ReviewsThis CASP checklist is not a scoring system, but rather a method of appraising systematic reviews by considering:
1. Are the results of the study valid?
2. What are the results?
3. Will the results help locally? - CEBM Systematic Reviews Critical Appraisal SheetThe CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence.
- JBI Critical Appraisal Tools, Checklist for Systematic ReviewsJBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
- NHLBI Study Quality Assessment of Systematic Reviews and Meta-AnalysesThe NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study.
- CASP Randomised Controlled Trials ChecklistThis CASP checklist considers various aspects of an RCT that require critical appraisal:
1. Is the basic study design valid for a randomized controlled trial?
2. Was the study methodologically sound?
3. What are the results?
4. Will the results help locally? - CONSORT StatementThe CONSORT checklist includes 25 items to determine the quality of randomized controlled trials. “Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in the report.”7
- NHLBI Study Quality Assessment of Controlled Intervention StudiesThe NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study.
- JBI Critical Appraisal Tools Checklist for Randomized Controlled TrialsJBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
- NHLBI Study Quality Assessment Tools for Non-Randomized StudiesThe NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study.
• Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies
• Quality Assessment of Case-Control Studies
• Quality Assessment Tool for Before-After (Pre-Post) Studies With No Control Group
• Quality Assessment Tool for Case Series Studies - Case Series Studies Quality Appraisal ChecklistDeveloped by the Institute of Health Economics (Canada), the checklist is comprised of 20 questions to assess “the robustness of the evidence of uncontrolled, [case series] studies.”10
- Methodological Quality and Synthesis of Case Series and Case ReportsIn this paper, Dr. Murad and colleagues “present a framework for appraisal, synthesis and application of evidence derived from case reports and case series.”11
- JBI Critical Appraisal Tools for Non-Randomized TrialsJBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• Checklist for Analytical Cross Sectional Studies
• Checklist for Case Control Studies
• Checklist for Case Reports
• Checklist for Case Series
• Checklist for Cohort Studies
- JBI Critical Appraisal Tools Checklist for Diagnostic Test Accuracy StudiesJBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
- STARD 2015The authors of the standards note that “[e]ssential elements of [diagnostic accuracy] study methods are often poorly described and sometimes completely omitted, making both critical appraisal and replication difficult, if not impossible.”10 The Standards for the Reporting of Diagnostic Accuracy Studies was developed “to help… improve completeness and transparency in reporting of diagnostic accuracy studies.”14
- CASP Diagnostic Study ChecklistThis CASP checklist considers various aspects of diagnostic test studies including:
1. Are the results of the study valid?
2. What were the results?
3. Will the results help locally? - CEBM Diagnostic Critical Appraisal SheetThe CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence.
- JBI Critical Appraisal Tools Checklist for Diagnostic Test Accuracy StudiesJBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
- SYRCLE’s RoB“[I]mplementation of [SYRCLE’s RoB tool] will facilitate and improve critical appraisal of evidence from animal studies. This may… enhance the efficiency of translating animal research into clinical practice and increase awareness of the necessity of improving the methodological quality of animal studies.”15
- ARRIVE 2.0“The [ARRIVE 2.0] guidelines are a checklist of information to include in a manuscript to ensure that publications [on in vivo animal studies] contain enough information to add to the knowledge base.”16
- Critical Appraisal of Studies Using Laboratory Animal ModelsThis article provides “an approach to critically appraising papers based on the results of laboratory animal experiments,” and discusses various “bias domains” in the literature that critical appraisal can identify.17
- CEBM Critical Appraisal of Qualitative Studies SheetThe CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance and applicability of clinical evidence.
- CASP Qualitative Studies ChecklistThis CASP checklist considers various aspects of qualitative research studies including:
1. Are the results of the study valid?
2. What were the results?
3. Will the results help locally?
- Quality Assessment and Risk of Bias Tool RepositoryCreated by librarians at Duke University, this extensive listing contains over 100 commonly used risk of bias tools that may be sorted by study type.
- Latitudes NetworkA library of risk of bias tools for use in evidence syntheses that provides selection help and training videos.
References & Recommended Reading
1. Kolaski, K., Logan, L. R., & Ioannidis, J. P. (2024). Guidance to best tools and practices for systematic reviews. British Journal of Pharmacology, 181(1), 180-210
2. Portney LG. Foundations of clinical research : applications to evidence-based practice. Fourth edition. ed. Philadelphia: F A Davis; 2020.
3. Fowkes FG, Fulton PM. Critical appraisal of published research: introductory guidelines. BMJ (Clinical research ed). 1991;302(6785):1136-1140.
4. Singh S. Critical appraisal skills programme. Journal of Pharmacology and Pharmacotherapeutics. 2013;4(1):76-77.
5. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Clinical research ed). 2017;358:j4008.
6. Whiting P, Savovic J, Higgins JPT, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of clinical epidemiology. 2016;69:225-234.
7. Sterne JAC, Savovic J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ (Clinical research ed). 2019;366:l4898.
8. Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials. Journal of clinical epidemiology. 2010;63(8):e1-37.
9. Sterne JA, Hernan MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ (Clinical research ed). 2016;355:i4919.
10. Downes MJ, Brennan ML, Williams HC, Dean RS. Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS). BMJ open. 2016;6(12):e011458.
11. Guo B, Moga C, Harstall C, Schopflocher D. A principal component analysis is conducted for a case series quality appraisal checklist. Journal of clinical epidemiology. 2016;69:199-207.e192.
12. Murad MH, Sultan S, Haffar S, Bazerbachi F. Methodological quality and synthesis of case series and case reports. BMJ evidence-based medicine. 2018;23(2):60-63.
13. Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J. Methodological index for non-randomized studies (MINORS): development and validation of a new instrument. ANZ journal of surgery. 2003;73(9):712-716.
14. Whiting PF, Rutjes AWS, Westwood ME, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Annals of internal medicine. 2011;155(8):529-536.
15. Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ (Clinical research ed). 2015;351:h5527.
16. Hooijmans CR, Rovers MM, de Vries RBM, Leenaars M, Ritskes-Hoitinga M, Langendam MW. SYRCLE's risk of bias tool for animal studies. BMC medical research methodology. 2014;14:43.
17. Percie du Sert N, Ahluwalia A, Alam S, et al. Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0. PLoS biology. 2020;18(7):e3000411.
18. O'Connor AM, Sargeant JM. Critical appraisal of studies using laboratory animal models. ILAR journal. 2014;55(3):405-417.