Systematic reviews

AMSTAR-2 (A MeaSurement Tool to Assess systematic Reviews)

Study designsSystematic reviews of randomized and non-randomized studies.
Number of items16
Ratingyes and no (other response options are available for some items such as partial yes and no meta-analysis conducted)
ValidityA face to face meeting was held with a group of experts to uptate the tool from the results of updated literature reviews on relevant critical appraisal tools, surveys of AMSTAR users, experience of participants in AMSTAR workshops at conferences, feedback from the AMSTAR website, and published critiques of the original tool (Shea et al 2017).

Studies on validity:
• Lorenz, R. C. (2020). AMSTAR 2 overall confidence rating: lacking discriminating capacity or requirement of high methodological quality? Journal of Clinical Epidemiology, 119, 142.
• Lorenz, R. C., Matthias, K., Pieper, D., Wegewitz, U., Morche, J., Nocon, M., et al. (2019). A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool. Journal of Clinical Epidemiology, 114, 133-140.
ReliabilityStudies on reliability:
• García-Alamino, J. M., López-Cano, M., Kroese, L., Helgstrand, F., & Muysoms, F. (2019). Quality Assessment and Risk of Bias of Systematic Reviews of Prophylactic Mesh for Parastomal Hernia Prevention Using AMSTAR and ROBIS Tools. World Journal of Surgery, 43(12), 3003-3012.
• Gates, A., Gates, M., Duarte, G., Cary, M., Becker, M., Prediger, B., et al. (2018). Evaluation of the reliability, usability, and applicability of AMSTAR, AMSTAR 2, and ROBIS: protocol for a descriptive analytic study. Systematic reviews, 7(1), 85.
• Inuganti, B., Inuganti, A., Vsn, M., Hyderboini, R., Chakrawarthy, M., Chidirala, S., et al (2018). A Comparison of Amstar and Robis Tools for Methodological Quality Assessment of Systematic Reviews of Alzheimer’s Disease. Value in Health, 21, S230.
• Leclercq, V., Beaudart, C., Tirelli, E., & Bruyère, O. (2020). Psychometric measurements of AMSTAR 2 in a sample of meta-analyses indexed in PsycINFO. Journal of Clinical Epidemiology, 119, 144-145.
• Lorenz, R. C., Matthias, K., Pieper, D., Wegewitz, U., Morche, J., Nocon, M., et al. (2019). A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool. Journal of Clinical Epidemiology, 114, 133-140.
• Pieper, D., Buechter, R. B., Li, L., Prediger, B., & Eikermann, M. (2015). Systematic review found AMSTAR, but not R (evised)-AMSTAR, to have good measurement properties. Journal of Clinical Epidemiology, 68(5), 574-583.
• Pieper, D., Jacobs, A., Weikert, B., Fishta, A., & Wegewitz, U. (2017). Inter-rater reliability of AMSTAR is dependent on the pair of reviewers. BMC Medical Research Methodology, 17(1), 98.
• Pieper, D., Puljak, L., González-Lorenzo, M., & Minozzi, S. (2019). Minor differences were found between AMSTAR 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies. Journal of Clinical Epidemiology, 108, 26-33.
Other informationPrevious version: AMSTAR developed in 2007
The second version of this tool was developed in 2017.
https://amstar.ca/Amstar-2.php
Main references• Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., et al. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. British Medical Journal, 358, j4008.
• Shea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., et al. (2009). AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 62(10), 1013-1020.

ROBIS (Risk of Bias in Systematic Reviews)

Study designsSystematic reviews
Number of items4 domains (5 to 6 items in each domain)
Ratingyes, probably yes, probably no, no, no information
ValidityA steering group of 11 experts defined the scope of the tool. Items were identified from reviewing 40 existing tools, a review of overviews and from the Methodological Expectations of Cochrane Intervention Review guidelines. A first draft of the tool was developed from a face to face meeting was held during a conference. The scope and content of the tool was refined during a Delphi study. The tool was piloted during conferences (Whiting et al 2016).

Studies on validity:
• Jaca, A., Ndze, V. N., Wiysonge, C. S. (2019). Assessing the methodological quality of systematic reviews of interventions aimed at improving vaccination coverage using AMSTAR and ROBIS checklists. Human Vaccines & Immunotherapeutics, 15(12), 2824-2835.
ReliabilityStudies on reliability:
• Banzi, R., Cinquini, M., Gonzalez-Lorenzo, M., Pecoraro, V., Capobussi, M., & Minozzi, S. (2018). Quality assessment versus risk of bias in systematic reviews: AMSTAR and ROBIS had similar reliability but differed in their construct and applicability. Journal of Clinical Epidemiology, 99, 24-32.
• Bühn, S., Mathes, T., Prengel, P., Wegewitz, U., Ostermann, T., Robens, S., & Pieper, D. (2017). The risk of bias in systematic reviews tool showed fair reliability and good construct validity. Journal of Clinical Epidemiology, 91, 121-128.
• García-Alamino, J. M., López-Cano, M., Kroese, L., Helgstrand, F., & Muysoms, F. (2019). Quality Assessment and Risk of Bias of Systematic Reviews of Prophylactic Mesh for Parastomal Hernia Prevention Using AMSTAR and ROBIS Tools. World Journal of Surgery, 43(12), 3003-3012.
• Gates, A., Gates, M., Duarte, G., Cary, M., Becker, M., Prediger, B., et al. (2018). Evaluation of the reliability, usability, and applicability of AMSTAR, AMSTAR 2, and ROBIS: protocol for a descriptive analytic study. Systematic Reviews, 7(1), 85.
• Inuganti, B., Inuganti, A., Vsn, M., Hyderboini, R., Chakrawarthy, M., Chidirala, S., et al. (2018). A Comparison of Amstar and Robis Tools for Methodological Quality Assessment of Systematic Reviews of Alzheimer’s Disease. Value in Health, 21, S230.
• Pieper, D., Puljak, L., González-Lorenzo, M., & Minozzi, S. (2019). Minor differences were found between AMSTAR 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies. Journal of Clinical Epidemiology, 108, 26-33.
Other informationhttps://www.bristol.ac.uk/population-health-sciences/projects/robis/
Main referencesWhiting, P., Savovic, J., Higgins, J. P., Caldwell, D. M., Reeves, B. C., Shea, B., et al. (2016). ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology, 69(1), 225-234.

Mixed Methods Systematic Reviews Appraisal Tool (MMSR)

Study designsMixed methods systematic reviews
Number of items27 items (on methods to identify and include studies, to analyse the findings in quantitatives and qualitatives analysis and to analyse the causal chain and reach conclusions).
Ratingyes, partially, no, not applicable
ValidityTool developed from frameworks for mixed methods research, and existing systematic review critical appraisal tools and checklists. The tool was reviewed by mixed methods experts and piloted by two independent reviewers on 7 papers (Jimenez et al 2018).
ReliabilityN/A
Other information
Main references• Jimenez, E., Waddington, H., Goel, N., Prost, A., & Pullin, A. (2018) Mixing and Matching: Using Qualitative Methods to Improve Quantitative Impact Evaluations (IEs) and Systematic Reviews (SRs) of Development Outcomes. London, UK: Centre of Excellence for Development Impact and Learning (CEDIL) Inception Paper 5.
• Jimenez, E., Waddington, H., Goel, N., Prost, A., Pullin, A., White, H., . . . Bhatia, R. (2018). Mixing and matching: using qualitative methods to improve quantitative impact evaluations (IEs) and systematic reviews (SRs) of development outcomes. Journal of Development Effectiveness, 10(4), 400-421.

Quality standards for realist synthesis

Study designRealist synthesis
Number of criteria9 criteria
RatingInadequate, adequate, good, excellent
ValidityTool developed from several sources such as email discussion list gathering more than 300 members, methodological experts, and methods training workshops (Wong et al 2014).
ReliabilityN/A
Other informationhttps://www.ramesesproject.org/Standards_and_Training_materials.php
Main references• Wong, G., Greenhalgh, T., Westhorp, G., & Pawson, R. (2014). Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: The RAMESES (Realist And Meta-narrative Evidence Syntheses - Evolving Standards) project. Health Services and Delivery Research, 2(30).

Quality standards for meta-narrative synthesis

Study designMeta-narrative synthesis
Number of criteria10 criteria
RatingInadequate, adequate, good, excellent
ValidityTool developed from several sources such as email discussion list gathering more than 300 members, methodological experts, and methods training workshops (Wong et al 2014).
ReliabilityN/A
Other informationhttps://www.ramesesproject.org/Standards_and_Training_materials.php
Main references• Wong, G., Greenhalgh, T., Westhorp, G., & Pawson, R. (2014). Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: The RAMESES (Realist And Meta-narrative Evidence Syntheses - Evolving Standards) project. Health Services and Delivery Research, 2(30).