Mixed methods studies

MMAT (Mixed Methods Appraisal Tool)

Study designsTool developed for use in systematic mixed studies reviews. Includes items for 5 categories of studies: 1) qualitative studies, 2) randomized controlled trials, 3) non randmized studies, 4) descriptive studies, and 5) mixed methods studies.
Number of items25 (5 items for each category of studies) and 2 screening questions
Ratingyes, no, can’t tell
ValidityThe MMAT was created from a literature review on mixed studies reviews. It was updated in 2018 from a literature on critical apprisal tool, usability study and e-Delphi study with experts.

• Hong, Q. N., Gonzalez-Reyes, A., & Pluye, P. (2018). Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). Journal of Evaluation in Clinical Practice, 24(3), 459-467. doi:10.1111/jep.12884
• Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., et al. (2019). Improving the content validity of the Mixed Methods Appraisal Tool (MMAT): a modified e-Delphi study. Journal of Clinical Epidemiology, 111(7), 49-59.e41.
ReliabilityThe version 2011 has been submitted to two interrater reliability testing. The Kappa ranged from poor to good depending of the categories of the MMAT.
• Pace, R., Pluye, P., Bartlett, G., Macaulay, A. C., Salsberg, J., Jagosh, J., & Seller, R. (2012). Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review. International Journal of Nursing Studies, 49(1), 47-53.
• Souto, R. Q., Khanassov, V., Hong, Q. N., Bush, P. L., Vedel, I., & Pluye, P. (2015). Systematic mixed studies reviews: Updating results on the reliability and efficiency of the mixed methods appraisal tool. International Journal of Nursing Studies, 52(1), 500-501.
Other informationThree versions: version 2009, 2011 and 2018.
http://mixedmethodsappraisaltoolpublic.pbworks.com
Main references• Version 2009: Pluye, P., Gagnon, M. P., Griffiths, F., & Johnson-Lafleur, J. (2009). A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. International Journal of Nursing Studies, 46(4), 529-546.
• Version 2011: Pluye, P., Robert, E., Cargo, M., Bartlett, G., O’Cathain, A., Griffiths, F., et al. (2011). Proposal: A Mixed Methods Appraisal Tool for systematic mixed studies reviews. Retrieved November 15, 2013, from http://mixedmethodsappraisaltoolpublic.pbworks.com
• Version 2018: Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., et al. (2018). Mixed Methods Appraisal Tool (MMAT), version 2018. Registration of Copyright (#1148552), Canadian Intellectual Property Office, Industry Canada.

Quality Assessment Tool for Studies with Diverse Designs (QATSDD)

Study designsTool developed to be applied to a methodologically diverse set of research articles. Include criteria for qualitative studies and quantitatives studies. To appraise mixed methods studies, the authors suggest to use all the critera.
Number of items16
Ratingnot at all, very slightly, moderately, complete
ValidityTool developed from existing tools and in consultation with senior academic researchers and an expert from the York Centre for Reviews and Dissemination. External feedback on the content validity, comprehension and ease of application was sought from nine researchers (Sirriyeh et al 2012).
ReliabilityThree raters appraised three papers. The interrater reliability was substantial (k= 71.5%). The test–retest reliability (after 6-week interval) ranged from good to substantial agreement for each of the three reviewers on each of the three papers (Sirriyeh et al 2012).
Other information
Main referencesSirriyeh, R., Lawton, R., Gardner, P., & Armitage, G. (2012). Reviewing studies with diverse designs: The development and evaluation of a new tool. Journal of Evaluation in Clinical Practice, 18(4), 746-752.

Mixed Methods Impact Evaluations (MMIE)

Study designsMixed methods impact evaluations
Number of items20 items (8 on quantitative aspect, 6 on qualitative aspect and 6 on integration)
Ratingyes and no (also "not application" for items on quantitative aspect)
ValidityTool developed from evaluation criteria and appraisal frameworks found in the literature on qualitative, quantitative and mixed methods studies. The tool was reviewed by mixed methods experts and tested by independent reviewers (Jimenez et al 2018).
ReliabilityN/A
Other information
Main references• Jimenez, E., Waddington, H., Goel, N., Prost, A., & Pullin, A. (2018) Mixing and Matching: Using Qualitative Methods to Improve Quantitative Impact Evaluations (IEs) and Systematic Reviews (SRs) of Development Outcomes. London, UK: Centre of Excellence for Development Impact and Learning (CEDIL) Inception Paper 5.
• Jimenez, E., Waddington, H., Goel, N., Prost, A., Pullin, A., White, H., et al. (2018). Mixing and matching: using qualitative methods to improve quantitative impact evaluations (IEs) and systematic reviews (SRs) of development outcomes. Journal of Development Effectiveness, 10(4), 400-421.