Systematic reviews

AMSTAR-2 (A MeaSurement Tool to Assess systematic Reviews)

Study designsSystematic reviews of randomized and non-randomized studies.
Number of items16
Ratingyes and no (other response options are available for some items such as partial yes and no meta-analysis conducted)
ValidityA face to face meeting was held with a group of experts to uptate the tool from the results of updated literature reviews on relevant critical appraisal tools, surveys of AMSTAR users, experience of participants in AMSTAR workshops at conferences, feedback from the AMSTAR website, and published critiques of the original tool (Shea et al 2017).
ReliabilitySeveral studies on reliability:
• García-Alamino, J. M., López-Cano, M., Kroese, L., Helgstrand, F., & Muysoms, F. (2019). Quality Assessment and Risk of Bias of Systematic Reviews of Prophylactic Mesh for Parastomal Hernia Prevention Using AMSTAR and ROBIS Tools. World Journal of Surgery, 43(12), 3003-3012.
• Gates, A., Gates, M., Duarte, G., Cary, M., Becker, M., Prediger, B., et al. (2018). Evaluation of the reliability, usability, and applicability of AMSTAR, AMSTAR 2, and ROBIS: protocol for a descriptive analytic study. Systematic reviews, 7(1), 85.
• Inuganti, B., Inuganti, A., Vsn, M., Hyderboini, R., Chakrawarthy, M., Chidirala, S., et al (2018). A Comparison of Amstar and Robis Tools for Methodological Quality Assessment of Systematic Reviews of Alzheimer’s Disease. Value in Health, 21, S230.
• Leclercq, V., Beaudart, C., Tirelli, E., & Bruyère, O. (2020). Psychometric measurements of AMSTAR 2 in a sample of meta-analyses indexed in PsycINFO. Journal of Clinical Epidemiology, 119, 144-145.
• Lorenz, R. C., Matthias, K., Pieper, D., Wegewitz, U., Morche, J., Nocon, M., et al. (2019). A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool. Journal of Clinical Epidemiology, 114, 133-140.
• Pieper, D., Buechter, R. B., Li, L., Prediger, B., & Eikermann, M. (2015). Systematic review found AMSTAR, but not R (evised)-AMSTAR, to have good measurement properties. Journal of Clinical Epidemiology, 68(5), 574-583.
• Pieper, D., Jacobs, A., Weikert, B., Fishta, A., & Wegewitz, U. (2017). Inter-rater reliability of AMSTAR is dependent on the pair of reviewers. BMC Medical Research Methodology, 17(1), 98.
• Pieper, D., Puljak, L., González-Lorenzo, M., & Minozzi, S. (2019). Minor differences were found between AMSTAR 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies. Journal of Clinical Epidemiology, 108, 26-33.
Other informationPrevious version: AMSTAR developed in 2007
The second version of this tool was developed in 2017.
https://amstar.ca/Amstar-2.php
Main references• Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., et al. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. British Medical Journal, 358, j4008.
• Shea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., et al. (2009). AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 62(10), 1013-1020.

ROBIS (Risk of Bias in Systematic Reviews)

Study designsSystematic reviews
Number of items4 domains (5 to 6 items in each domain)
Ratingyes, probably yes, probably no, no, no information
ValidityA steering group of 11 experts defined the scope of the tool. Items were identified from reviewing 40 existing tools, a review of overviews and from the Methodological Expectations of Cochrane Intervention Review guidelines. A first draft of the tool was developed from a face to face meeting was held during a conference. The scope and content of the tool was refined during a Delphi study. The tool was piloted during conferences (Whiting et al 2016).
ReliabilitySeveral studies on reliability:
• Banzi, R., Cinquini, M., Gonzalez-Lorenzo, M., Pecoraro, V., Capobussi, M., & Minozzi, S. (2018). Quality assessment versus risk of bias in systematic reviews: AMSTAR and ROBIS had similar reliability but differed in their construct and applicability. Journal of Clinical Epidemiology, 99, 24-32.
• Bühn, S., Mathes, T., Prengel, P., Wegewitz, U., Ostermann, T., Robens, S., & Pieper, D. (2017). The risk of bias in systematic reviews tool showed fair reliability and good construct validity. Journal of Clinical Epidemiology, 91, 121-128.
• García-Alamino, J. M., López-Cano, M., Kroese, L., Helgstrand, F., & Muysoms, F. (2019). Quality Assessment and Risk of Bias of Systematic Reviews of Prophylactic Mesh for Parastomal Hernia Prevention Using AMSTAR and ROBIS Tools. World Journal of Surgery, 43(12), 3003-3012.
• Gates, A., Gates, M., Duarte, G., Cary, M., Becker, M., Prediger, B., et al. (2018). Evaluation of the reliability, usability, and applicability of AMSTAR, AMSTAR 2, and ROBIS: protocol for a descriptive analytic study. Systematic Reviews, 7(1), 85.
• Inuganti, B., Inuganti, A., Vsn, M., Hyderboini, R., Chakrawarthy, M., Chidirala, S., et al. (2018). A Comparison of Amstar and Robis Tools for Methodological Quality Assessment of Systematic Reviews of Alzheimer’s Disease. Value in Health, 21, S230.
• Pieper, D., Puljak, L., González-Lorenzo, M., & Minozzi, S. (2019). Minor differences were found between AMSTAR 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies. Journal of Clinical Epidemiology, 108, 26-33.
Other informationhttps://www.bristol.ac.uk/population-health-sciences/projects/robis/
Main referencesWhiting, P., Savovic, J., Higgins, J. P., Caldwell, D. M., Reeves, B. C., Shea, B., et al. (2016). ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology, 69(1), 225-234.

Mixed Methods Systematic Reviews Appraisal Tool (MMSR)

Study designsMixed methods systematic reviews
Number of items27 items (on methods to identify and include studies, to analyse the findings in quantitatives and qualitatives analysis and to analyse the causal chain and reach conclusions).
Ratingyes, partially, no, not applicable
ValidityTool developed from frameworks for mixed methods research, and existing systematic review critical appraisal tools and checklists. The tool was reviewed by mixed methods experts and piloted by two independent reviewers on 7 papers (Jimenez et al 2018).
ReliabilityN/A
Other information
Main references• Jimenez, E., Waddington, H., Goel, N., Prost, A., & Pullin, A. (2018) Mixing and Matching: Using Qualitative Methods to Improve Quantitative Impact Evaluations (IEs) and Systematic Reviews (SRs) of Development Outcomes. London, UK: Centre of Excellence for Development Impact and Learning (CEDIL) Inception Paper 5.
• Jimenez, E., Waddington, H., Goel, N., Prost, A., Pullin, A., White, H., . . . Bhatia, R. (2018). Mixing and matching: using qualitative methods to improve quantitative impact evaluations (IEs) and systematic reviews (SRs) of development outcomes. Journal of Development Effectiveness, 10(4), 400-421.