Skip to main content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

LIBRARY 

Systematic Reviews: Appraising Pool of Citations

This guide is for the clinician undertaking a systematic review.

Overview

You've finally narrowed the thousands of citations down to a smaller, more relevant group of articles. Now what do you do to make sense of this information? Take a look below at the various tools and guides to help you determine the quality of and risk of bias for each article.

"Critical appraisal skills enable you to assess the trustworthiness, relevance and results of published papers so that you can decide if they are believable and useful."  (http://www.casp-uk.net/criticalappraisal)

Critical Appraisal Checklists by specific Study Design type

Randomized Controlled Trials

 

RoB 2.0 Risk Of Bias Tool

New as of July 2019

RoB 2 Tool : A revised Cochrane risk of bias tool for randomized trials.

https://sites.google.com/site/riskofbiastool/welcome/rob-2-0-tool?authuser=0

Archived: There are 3 variations of the tool based on trial design.  Choose the appropriate version.

1.  Individually randomized, parallel group trials (traditional RCT)

2.  Cluster randomized, parallel group trials (like traditional but randomized by groups)

3.  Individual randomized, cross-over trials (randomization occurs to determine the order the patient receives both treatments)

http://www.bristol.ac.uk/population-health-sciences/centres/cresyda/barr/riskofbias/rob2-0/​


CASP: Randomised Controlled Trial Checklist
Summary: Critical Appraisal Skills Program (CASP): RCT CAT is a methodological checklist which provides key criteria relevant to randomised controlled trials.
Authors: Public Health Resource Unit, NHS, England

https://casp-uk.net/wp-content/uploads/2018/01/CASP-Randomised-Controlled-Trial-Checklist-2018.pdf


The CONSORT Statement
Summary: The Consolidated Standards of Reporting Trials (CONSORT) Statement is a detailed document which outlines an explanation and elaboration of the CONSORT statement for reporting randomized controlled trials. It also includes (in table two) the critical appraisal tool.
Authors: The CONSORT Group, Canada 

http://www.consort-statement.org/consort-statement/

 


Non-randomized Studies or Observational Studies

 

ROBINS-I

Risk of Bias for non-randomized (observational) studies or cohorts of Interventions

http://www.bristol.ac.uk/population-health-sciences/centres/cresyda/barr/riskofbias/robins-i/​

 

ROBINS-E

Risk of Bias for non-randomized (observational) studies or cohorts of Exposures other than interventions, including environmental and occupational exposures

http://www.bristol.ac.uk/population-health-sciences/centres/cresyda/barr/riskofbias/robins-e/​

 

Newcastle-Ottawa scale (NOS) -most widely used for case control or cohort studies

>http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp

 

AXIS-To assess the quality of cross-sectional (prevalence) studies

https://bmjopen.bmj.com/content/6/12/e011458

 

NIH – NHLBI – Study Quality Assessment Tools

Tools for Controlled Intervention Studies, Systematic Reviews & Meta-Analyses, Observational Cohort and Cross-Sectional Studies, Case-Control Studies, Before-After (Pre-Post) Studies With No Control Group, Case Series Studies.

https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools

 

IHE Case Series Studies Critical Appraisal Checklist

https://www.ihe.ca/research-programs/rmd/cssqac/cssqac-about

 

Methodological Index for Non-Randomized Studies (MINOR) is a tool for assessing non-randomized interventional studies  

http://cobe.paginas.ufsc.br/files/2014/10/MINORS.pdf

 

Joanna Briggs Institute

Critical Appraisal Tools

https://joannabriggs.org/critical-appraisal-tools

 

McMaster Critical Review Form - Quantitative Studies

http://srs-mcmaster.ca/research/evidence-based-practice-research-group/

 


Systematic Reviews

AMSTAR

AMSTAR can be used to assess the methodological quality of a systematic review and as a guide to performing a systematic review. Two agreements are required during quality assessment ensuring lower risk of bias. AMSTAR has guidelines explaining each outlined item.

https://amstar.ca/index.php

 

ROBIS

ROBIS is a new tool for assessing the risk of bias in systematic reviews (rather than in primary studies). Here you can find the tool itself, information to help you complete a ROBIS assessment, and resources to help you present the results of your ROBIS assessment.

http://www.bristol.ac.uk/population-health-sciences/projects/robis/

 

CASP: Systematic Review Checklist
Summary: Critical Appraisal Skills Program (CASP): This is a methodological checklist which provides key criteria relevant to systematic reviews.
Authors: Public Health Resource Unit, NHS, England

https://casp-uk.net/wp-content/uploads/2018/01/CASP-Systematic-Review-Checklist_2018.pdf

 

CEBM:  Systematic Reviews Critical Appraisal Worksheet 
Authors: Centre for Evidence Based Medicine (CEBM), University of Oxford

https://casp-uk.net/wp-content/uploads/2018/01/CASP-Systematic-Review-Checklist_2018.pdf


Public Health

 

MetaQat - Meta-Tool for appraising all types of public health evidence

http://www.publichealthontario.ca/en/ServicesAndTools/Pages/Critical-Appraisal-Tool.aspx


Quantitative Research (Public Health Research)

 

EPHPP- Quality Assessment tool for Quantitative Studies 

Also known as the "Hamilton" tool  

http://www.ephpp.ca/tools.html


Animal Research

 

Systematic Review Centre for Laboratory Animal Experimentation (SYRCLE) risk of bias assessment tool for assessing animal studies

http://www.ncbi.nlm.nih.gov/pubmed/24667063

 

(invivo studies) Review the ARRIVE guidelines  

https://www.nc3rs.org.uk/arrive-guidelines

 

Critical Appraisal of Studies using Animal Models

http://ilarjournal.oxfordjournals.org/content/55/3/405.full.pdf+html


Diagnostic Accuracy

 

QUADAS-2 

QUADAS-2 is the current version of QUADAS and the tool that we recommend for use in systematic reviews to evaluate the risk of bias and applicability of primary diagnostic accuracy studies.

http://www.bristol.ac.uk/media-library/sites/quadas/migrated/documents/quadas2.pdf 

 

STARD Checklist
Summary: STAndards for the Reporting of Diagnostic accuracy checklist is helps determine the accuracy and completeness of reporting of studies of diagnostic accuracy. The STARD Statement consists of a 25 item checklist and flow diagram.

http://www.stard-statement.org/​


CASP: Diagnostic Test Checklist

https://casp-uk.net/wp-content/uploads/2018/01/CASP-Diagnostic-Checklist-2018.pdf

 

CEBM: Diagnosis Critical Appraisal Worksheet
Authors: Centre for Evidence Based Medicine (CEBM), University of Oxford

https://www.cebm.net/wp-content/uploads/2018/11/Diagnostic-Accuracy-Studies.pdf

Learn more about Critical Appraisal

Grading the Strength of Evidence

Additional Grading Tables & Scales

 

Study Design Flow Chart

Calculators

  • Alan Schwartz: Diagnostic Test Calculator
    This calculator can determine diagnostic test characteristics (sensitivity, specificity, likelihood ratios) and/or determine the post-test probability of disease given the pre-test probability and test characteristics. Given sample sizes, confidence intervals are also computed.
  • Alan Schwartz: Risk Reduction Calculator
    Given information about the probability of an outcome under control and experimental treatments, this calculator produces measures of risk increase/decrease and number needed to treat or harm, including confidence intervals. If some patients were lost to follow-up, the calculator provides estimates for several different scenarios.
  • Alan Schwartz: Diagnostic Test Cutoffs
    A graphical demonstration of the effect of changing cutoff scores on sensitivity and specificity of a test.
  • Alan Schwartz: Statistical Testing Thresholds
    A graphical demonstration like the above, but written in terms of statistical test theory (type I and II error).