MMAT (Mixed Methods Appraisal Tool)
| Study designs | Tool developed for use in systematic mixed studies reviews. Includes items for 5 categories of studies: 1) qualitative studies, 2) randomized controlled trials, 3) non randmized studies, 4) descriptive studies, and 5) mixed methods studies. |
| Number of items | 25 (5 items for each category of studies) and 2 screening questions |
| Rating | yes, no, can’t tell |
| Validity | The MMAT was created from a literature review on mixed studies reviews. It was updated in 2018 from a literature on critical apprisal tool, usability study and e-Delphi study with experts. • Hong, Q. N., Gonzalez-Reyes, A., & Pluye, P. (2018). Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). Journal of Evaluation in Clinical Practice, 24(3), 459-467. doi:10.1111/jep.12884 • Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., et al. (2019). Improving the content validity of the Mixed Methods Appraisal Tool (MMAT): a modified e-Delphi study. Journal of Clinical Epidemiology, 111(7), 49-59.e41. |
| Reliability | The version 2011 has been submitted to two interrater reliability testing. The Kappa ranged from poor to good depending of the categories of the MMAT. • Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012). Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review. International Journal of Nursing Studies, 49(1), 47-53. • Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015). Systematic mixed studies reviews: Updating results on the reliability and efficiency of the mixed methods appraisal tool. International Journal of Nursing Studies, 52(1), 500-501. |
| Other information | Three versions: version 2009, 2011 and 2018. http://mixedmethodsappraisaltoolpublic.pbworks.com |
| Main references | • Version 2009: Pluye, P., Gagnon, M. P., Griffiths, F., & Johnson-Lafleur, J. (2009). A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. International Journal of Nursing Studies, 46(4), 529-546. • Version 2011: Pluye, P., Robert, E., Cargo, M., Bartlett, G., O’Cathain, A., Griffiths, F., et al. (2011). Proposal: A Mixed Methods Appraisal Tool for systematic mixed studies reviews. Retrieved November 15, 2013, from http://mixedmethodsappraisaltoolpublic.pbworks.com • Version 2018: Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., et al. (2018). Mixed Methods Appraisal Tool (MMAT), version 2018. Registration of Copyright (#1148552), Canadian Intellectual Property Office, Industry Canada. |
Quality assessment with diverse studies (QuADS)
| Study designs | Tool developed to be applied to a methodologically diverse set of research articles. Include criteria for qualitative studies and quantitatives studies. To appraise mixed methods studies, the authors suggest to use all the critera. |
| Number of items | 16 |
| Rating | not at all, very slightly, moderately, complete |
| Validity | The Quality Assessment Tool for Studies with Diverse Designs (QATSDD) was developed from existing tools and in consultation with senior academic researchers and an expert from the York Centre for Reviews and Dissemination. External feedback on the content validity, comprehension and ease of application was sought from nine researchers (Sirriyeh et al 2012). The face and content validity of Quality assessment with diverse studies (QuADS) was explored with 10 researchers who had expertise in reviewing studies with diverse designs within systematic reviews (Harrison et al 2021). |
| Reliability | Three raters appraised three papers. The interrater reliability of the QATSDD was substantial (k= 71.5%). The test–retest reliability (after 6-week interval) ranged from good to substantial agreement for each of the three reviewers on each of the three papers (Sirriyeh et al 2012). The inter-rater reliability of the QuADS tool was tested on 40 studies and a kappa of 0.65 was obtained (Harrison et al 2021). |
| Other information | The Quality assessment with diverse studies (QuADS) is an update of the Quality Assessment Tool for Studies with Diverse Designs (QATSDD). |
| Main references | • Harrison, R., Jones, B., Gardener, P., & Lawton, R. (2021). Quality assessment with diverse studies (QuADS): an appraisal tool for methodological and reporting quality in systematic reviews of mixed-or multi-method studies. BMC Health Services Research, 21(1), 1-20. • Sirriyeh, R., Lawton, R., Gardner, P., & Armitage, G. (2012). Reviewing studies with diverse designs: The development and evaluation of a new tool. Journal of Evaluation in Clinical Practice, 18(4), 746-752. |
Mixed Methods Impact Evaluations (MMIE)
| Study designs | Mixed methods impact evaluations |
| Number of items | 20 items (8 on quantitative aspect, 6 on qualitative aspect and 6 on integration) |
| Rating | yes and no (also "not application" for items on quantitative aspect) |
| Validity | Tool developed from evaluation criteria and appraisal frameworks found in the literature on qualitative, quantitative and mixed methods studies. The tool was reviewed by mixed methods experts and tested by independent reviewers (Jimenez et al 2018). |
| Reliability | N/A |
| Other information | N/A |
| Main references | • Jimenez, E., Waddington, H., Goel, N., Prost, A., & Pullin, A. (2018) Mixing and Matching: Using Qualitative Methods to Improve Quantitative Impact Evaluations (IEs) and Systematic Reviews (SRs) of Development Outcomes. London, UK: Centre of Excellence for Development Impact and Learning (CEDIL) Inception Paper 5. • Jimenez, E., Waddington, H., Goel, N., Prost, A., Pullin, A., White, H., et al. (2018). Mixing and matching: using qualitative methods to improve quantitative impact evaluations (IEs) and systematic reviews (SRs) of development outcomes. Journal of Development Effectiveness, 10(4), 400-421. |
ASSESS Tool
| Study designs | Tool to (1) describe studies focusing on implementation that use qualitative, quantitative, and/or mixed methodologies and (2) assess risk of bias of implementation outcomes. |
| Number of items | 24 items (with several subitems) on four domains |
| Rating | Item 23: 0 or 1 Item 24: higher (i.e., score of 1–2), lower (i.e., score of 3–5), or unclear (i.e., not able to be assessed). |
| Validity | The tool was developed using a Nominal Group techniques (NGT) (n=5) and a Delphi Group technique (n=32). The tool includes a list of new items and items from existing tools: TIDieR checklist, StaRI checklist, MMAT tool, implementation outcomes taxonomy, and FRAME framework]. Feedback of usability and utility documented with 9 participants. • Ryan, N., Vieira, D., Gyamfi, J., Ojo, T., Shelley, D., Ogedegbe, O., Iwelunmor, J., & Peprah, E. (2022). Development of the ASSESS tool: a comprehenSive tool to Support rEporting and critical appraiSal of qualitative, quantitative, and mixed methods implementation reSearch outcomes. Implementation Science Communications, 3(1), 1-13. |
| Reliability | |
| Other information | |
| Main references | • Ryan, N., Vieira, D., Gyamfi, J., Ojo, T., Shelley, D., Ogedegbe, O., Iwelunmor, J., & Peprah, E. (2022). Development of the ASSESS tool: a comprehenSive tool to Support rEporting and critical appraiSal of qualitative, quantitative, and mixed methods implementation reSearch outcomes. Implementation Science Communications, 3(1), 1-13. |
