The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers

1. Introduction

This paper is the third of a three-part series on the topic of systematic reviews for information professionals (Hong & Pluye, 2018b; Pluye et al., 2018). It focuses on a critical appraisal tool that was developed for use in mixed studies reviews (MSRs): the Mixed Methods Appraisal Tool (MMAT). This tool can be useful for information professionals conducting or supporting MSRs and their trainers, as well as all those interacting with empirical studies in information sciences.

Despite the advantages of MSRs (Pluye et al., 2018), several challenges are encountered in this type of review because of the heterogeneity of included study designs. One of these challenges is related to the appraisal of the quality of the studies included in a review. Critical appraisal of included studies is a core step of systematic reviews. It consists of a systematic and careful examination of studies to ensure they are trustworthy, valid and reliable (Burls, 2009; Harden & Gough, 2012). Appraisal results can be used for different purposes such as to exclude low quality studies, to describe the quality of included studies, to perform sensitivity analysis and subgroup analyses, and to nuance the recommendations (Hong & Pluye, 2018a). Currently, there exist more than 500 critical appraisal tools (i.e., checklists including a list of criteria to judge the quality of a study) (Bai et al., 2012; Crowe & Sheppard, 2011; Heyvaert et al., 2013; Santiago-Delefosse et al., 2016). When conducting a MSR, reviewers have to choose different critical appraisal tools for each type of study design included in their review. The task of searching for and learning new tools can be time consuming. To address this challenge, a critical appraisal tool for assessing the quality of quantitative, qualitative, and mixed methods studies was developed: the Mixed Methods Appraisal Tool (MMAT) (Pluye et al., 2009). The aim of this paper is to present the latest enhancements made to the MMAT.

2. How was the MMAT developed?

The MMAT was developed back in 2006 from a literature review in which the quality appraisal procedures of 17 MSRs was analyzed thematically (Pluye et al., 2009). This led to develop a first version of the MMAT that is in line with a social constructionist worldview (Pluye et al., 2009). This version of the MMAT included 15 criteria for four categories of study designs (qualitative, quantitative experimental, quantitative observational, and mixed methods). It was pilot tested during workshops and in another study with four reviewers that used the MMAT to appraise the quality of six studies. This led to suggest a second version in 2011, in which changes were made to some existing criteria and new criteria for assessing nonrandomized studies were added (Pace et al., 2012). Then, two interrater reliability studies on the MMAT were conducted using respectively 32 and 261 papers (Pace et al., 2012; Souto et al., 2015). These studies showed the need to clarify some criteria in the MMAT, particularly those related to nonrandomized and qualitative studies. To further its development, usefulness testing was performed by interviewing 20 researchers who have used the MMAT in a systematic review or had contacted the developer of the tool for questions or permission to use the tool (Hong et al., 2018a). The results of this study were helpful to identify changes to be made in the MMAT. Also, a literature review on critical appraisal tools was conducted as well as a Delphi study with 73 methodological experts with expertise in qualitative, survey, and mixed methods research to identify the most relevant criteria to include in the MMAT (Hong et al., in revision). The results of these studies informed the revision of the MMAT, identified the core criteria to include for each category of study designs, and led to develop a third version of the tool (version 2018).

3. What is the MMAT version 2018?

Three main changes were made to the MMAT. First, on the basis of the latest studies on the MMAT, of the 19 criteria in the MMAT (version 2011), four were removed, seven were reformulated, five were replaced, and ten new were added (Hong et al., 2018b). The latest version of the MMAT (version 2018) includes a total of 25 criteria and 2 screening questions. As in the previous version, the MMAT can appraise five different categories of study designs: (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive and (e) mixed methods. These categories of designs were kept because they are the most common designs included in MSRs. For each category, there are now five core criteria (instead of four in the previous version), i.e., criteria that are the most relevant to appraise the methodological quality of studies. Each criterion is rated as ‘yes’, ‘no’ or ‘can’t tell’.

Second, the MMAT’s user guide was updated with the new criteria and provides explanations to help the reviewers judge the criteria in the MMAT. For each category of study design, a table is provided presenting a definition, common designs and approaches, and explanation of the criteria. In the latest version of the MMAT, an algorithm was added to help MMAT users choose the category (or categories) of criteria to use for their review. The algorithm was developed based on several existing algorithms of quantitative study designs (Hartling et al., 2010; Hartling et al., 2011; National Institute for Health Care Excellence, 2012; Scottish Intercollegiate Guidelines Network; Seo et al., 2016; West et al., 2002; Zaza et al., 2000). These algorithms were simplified for the purpose of the MMAT: only the main study designs are presented and study designs of qualitative and mixed methods studies were added. The MMAT (version 2018) checklist and user guide are available at this website and can be downloaded and used free of charge: http://mixedmethodsappraisaltoolpublic.pbworks.com/.

Third, changes were made on how to compute the overall score. In the previous version, an overall score could be calculated by dividing the number of criteria met by four (Pluye et al., 2011). In the literature, there has been much debate about the use of summative score in critical appraisal (Glenny, 2005). The use of a summative numerical score is a simple way of providing an overall idea of the quality of a study. However, a single number does not provide information on what aspects of studies are problematic and can even hide serious defects (Crowe & Sheppard, 2011). Also, it is unclear whether criteria should be weighted or not (Colle et al., 2002; Higgins & Green, 2008). Currently in the literature, it is discouraged to calculate an overall score from the ratings of each criterion (Herbison et al., 2006; Higgins & Green, 2008; Viswanathan et al., 2012). On this basis, it was decided to remove the summative numerical score from the MMAT. Instead, it is advised to provide a more detailed presentation of the ratings of each criterion to better inform the quality of the included studies, and sensitivity analysis can be performed.

4. How to use the MMAT?

Using the MMAT involves three main steps. First, the users can look at the two optional screening questions at the beginning of the tool. Responding ‘No’ or ‘Can’t tell’ to one or both questions might indicate that the paper is not an empirical study, and thus cannot be appraised using the MMAT, which is based on methodological criteria.

Second, the users need to choose the appropriate categories of study designs to appraise among the five categories in the MMAT: either (a) qualitative, (b) randomized controlled, (c) nonrandomized, (d) quantitative descriptive, or (e) mixed methods. To appraise the quality of a qualitative study, one category of criteria should be chosen (i.e., the qualitative category). For a quantitative study, users must decide which category of criteria is most appropriate (either randomized controlled trials, nonrandomized studies, or quantitative descriptive studies). When appraising a mixed methods study, three categories of criteria should be used, i.e., the qualitative category, one of the three quantitative categories, and the mixed methods category. In doing so, the MMAT acknowledges the methodological distinctive characteristics specific to each component used in mixed methods studies (i.e., qualitative, quantitative, and mixed methods) (O’Cathain, 2010).

The third step consists in rating the criteria of the chosen category (or categories). There are three response options: ‘Yes’ meaning the criterion is met, ‘No’ meaning the criterion is not met, and ‘Can’t tell’ when there is not enough information in the paper to judge if the criterion is met or not.

5. Conclusion

The MMAT can be a useful critical appraisal tool for MSRs since it provides, within a single tool, methodological quality criteria for different study designs. Also, the MMAT focuses on a limited number of core criteria, enabling a more efficient quality appraisal. Moreover, it includes specific criteria for mixed methods studies, which is not often found in other tools (Pluye, 2013). The criteria in the MMAT are more difficult to judge than in other appraisal tools because they focus on methodological quality and not on reporting quality. Methodological quality criteria are more difficult to interpret because the reviewers need to judge whether the reported results of a study are trustworthy (Carroll et al., 2012; Hong & Pluye, 2018a).

Up to now, the revision of the MMAT has focused on its content validity and usefulness. Further testing of its validity and reliability will be needed in the future. Also, more and more researchers are interested in developing new evidence on critical appraisal. As evidence develops, the MMAT may need modifications to keep it up to date with the latest methodological developments.