Doing systematic reviews in Psychology

Abstract

Systematic reviews by virtue of being a pre-determined, transparent, and comprehensive plan and search strategy are fast gaining popularity in psychology in South Africa. A systematic review allows one to obtain a thorough overview regarding the recent developments and debates on a given topic with the addition of metacommentary. In South Africa, we have noted stark differences in the reporting of systematic reviews. Often studies are identified as systematic reviews but methodologically have failed to meet the rigorous criteria that characterise this method. This article aims to provide a guide for the novice researcher on conducting systematic reviews. We draw on a practical case study by Hassem and Laher where the systematic review method was used to establish the efficacy of online depression screening tools in the South African context as a practical illustration of the systematic review method. In so doing, the affordances and limitations of the systematic review method are discussed.

Keywords

Critical Appraisal Skills Programme meta-analysis Preferred Reporting Items for Systematic Reviews and Meta-Analyses qualitative synthesis systematic review

Introduction

A systematic review (SR) can be classified as a research method that collates empirical evidence which serves to answer a specific research question (Higgins & Green, 2011; Oxman & Guyatt, 1993). An SR draws on a pre-determined, transparent, and comprehensive plan and search strategy. Bias is reduced through identifying, appraising, and synthesising studies included in the review (Higgins & Green, 2011; Oxman & Guyatt, 1993). Given the wealth of research information available globally and the limited time available, an SR allows one to obtain a quick yet thorough overview regarding the recent developments and debates on a given topic with the addition of metacommentary, that is, further explanation or commentary on a commentary (Graff & Birkenstein, 2014). For example, in SRs, metacommentary could involve identification of trends in evidence across the studies examined that allow for further discussion on future research in the field or a more critical examination of conceptual arguments.

SRs when rigorously conducted, therefore, have the potential to hold the topmost position in terms of hierarchies of evidence. By combining findings across studies in a field, SRs allow for (1) the production of robust and broad conclusions by producing an unbiased summary of what the cumulative evidence says on a particular topic; (2) a critical synthesis of literatures in the field thereby identifying relationships, contradictions, methodological flaws, gaps, and inconsistencies informing directions for future research or intervention; and (3) the development of a new theory or the evaluation of an existing theory and/or have clear implications for policy or practice (Siddaway et al., 2019, p. 751). In so doing, SRs offer what would typically only have been obtained previously with the replicability of studies. Further, SRs have the ability to use meta-analysis as a part of the process of analysing data. The flexibility of the method to incorporate both simple and advanced quantitative and qualitative analytic techniques adds to its appeal (Aromataris & Pearson, 2014).

SRs are not new to the literature or to the South African context. However, the method has not frequently been used in the field of psychology in South Africa. The SR method has much to offer psychology in South Africa where a number of fields in the discipline have grown much since 1994 with little integration of literature in the field. For example, the South African Journal of Psychology has published less than 1 SR per year since 1994. However, a cursory examination of research themes across manuscripts published in the journal shows that the field of psychological practice, community psychology, HIV/AIDS, violence, trauma, gender and sexuality, and racism among others has had much published in the area. SRs can be particularly useful to examine elements of this research to define future agendas for research and practice in the country. The systematic examination of evidence on a particular set of questions in a particular area offers much in terms of planning, policy, and intervention for the field.

In articles where reviews are used, there appears to be great variance in the implementation of the method. Grant and Booth (2009) point out the variety that exists among reviews as well as the number of terms used to denote similar methods. However, the review method, and the SR method in particular, has evolved internationally and there are accepted ways of conducting a review as detailed further on. The literature on the SR method has grown substantially, and navigating the literature to make informed methodological choices is time consuming and can be daunting. This is more so as there is no one accepted method for conducting SRs. There are slight differences depending on whose guidelines one follows. This article brings together this information for easier access on making appropriate choices using the SR method. This will also ensure that researchers in psychology have a consistent framework that is internationally applicable when conducting SR research.

Thus in this article, we provide a brief history of SRs. Following this, we provide some direction on the steps to follow in conducting an SR. Embedded in the process discussion, we draw on a practical case study where the SR method was used to establish the use and efficacy of online depression screening tools in the South African context (Hassem & Laher, 2019). In so doing, the affordances and limitations of the SR method are discussed.

Brief history of an SR

The history of SRs dates back to the 18th century, under the term ‘research synthesis’. Research synthesis was conducted at the time in the fields of mathematics, agriculture, psychology as well as medicine. It was only during the 20th century when the term SR was specifically utilised (Chalmers et al., 2002). Archie Cochrane is credited with the contemporary introduction and subsequent development of SRs in the field of research.

It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials. (Cochrane, 1979, p. 11)

Hence, the modern roots of SRs were strongly rooted in the tradition of evaluating findings of randomised control trials (RCTs, Grant & Booth, 2009). Meta-analyses were common in social science research pre-1990s when positivist research was often preferred. Paradigmatic shifts and increasing access to technology since then have created a space for richer and less statistically dependent forms of reviewing literature (Gough et al., 2017). As the SR method evolved, the meta-analysis has traditionally been understood as the quantitative form of the SR which considers quantitative studies while a meta-synthesis, meta-ethnography, qualitative evidence synthesis, and qualitative SR have been used interchangeably to refer to an SR which collates qualitative evidence (see Grant & Booth, 2009; Siddaway et al., 2019). Typically, SRs have chosen to focus on single methods using either quantitative studies only or qualitative studies only. However, there is the possibility of conducting a mixed methods SR where quantitative, qualitative, and theoretical studies may be included (see Lizarondo et al., 2017).

In 1992, the Cochrane Collaboration was established to assist individuals to make informed decisions regarding health care through the promotion of the use of SRs (see http://www.cochrane.org/). To date, the Cochrane SR method is viewed as the gold standard when assessing the effectiveness of RCTs. During 1998, the Cochrane database went online, and it is currently the largest SR collection. The versatility of using the Cochrane review method in the social sciences field is often limited. As a result, various institutions have been established such as the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre, see https://eppi.ioe.ac.uk/cms/) established in 1993 and the Campbell Collaboration (see https://campbellcollaboration.org/) in 2000. These institutions promote the use of SRs in the social sciences to implement and develop policy. The Joanna Briggs Institute (JBI; see https://joannabriggs.org/our-history) founded in 1996 and the International Prospective Register of Systematic Reviews (PROSPERO; see https://www.crd.york.ac.uk/PROSPERO/review_scope.asp) established in 2011, although more focused towards health sciences, also accommodate some research from the social sciences.

Despite the establishment of these institutions, it is evident that the SR in the social sciences is still developing as evident by the varying methods used for conducting and presenting SRs (Gough et al., 2017). The term SR is often used in the literature interchangeably with narrative or scoping reviews. SRs are different from literature or narrative reviews in that SRs involve a methodical, comprehensive, transparent, and replicable search of the literature. A review of the literature for a thesis or research study involves a selective search of literature that provides a flowing rationale for the study being undertaken. It is not necessarily comprehensive nor can it claim to be an unbiased account of the literature in the field (Siddaway et al., 2019).

SRs are also different from scoping reviews. A scoping review provides a quick overview of the literature prior to an in-depth investigation of the literature often to determine whether an SR would be in order, whether an SR has been done before, and whether the questions being asked require refinement to address the gaps in the literature within a particular field (Cherry & Dickson, 2017; Levac et al., 2010; Tricco et al., 2016). Hence, scoping reviews precede SRs and are not as rigorous as SRs. The Arksey and O’Malley framework for conducting scoping reviews is the method followed by most (see Arksey & O’Malley, 2005; Levac et al., 2010). Recently, the JBI published guidelines for conducting scoping reviews (see Peters et al., 2015).

Various resources have been developed to assist social science researchers with a framework to ensure methodological rigour in conducting SRs such as the EPPI (see https://eppi.ioe.ac.uk/cms/), Gough et al. (2017) as well as Building Capacity to Use Research Evidence (BCURE) programme developed by Africa Policy Network and University of Johannesburg (Laher et al., 2019). Hence, the guidelines in this article are by no means sacrosanct. Instead, they represent an attempt at assisting novice researchers navigate the SR method and make informed decisions.

Method of conducting an SR

The method of conducting an SR is standard across all fields of research; however, dependent on the specific research question utilised, various steps in the procedure could be adapted to suit that particular field of study. There are many references to guidelines on the steps to follow for undertaking SRs. Organisations like the JBI, for example, specify the steps for conducting an SR. However, across the various SR frameworks, four elements are common: namely, defining the question, selecting the literature, conducting a quality appraisal, and analysing the studies. For the online depression screening study, the eight-step process as advocated by Uman (2011, see Figure 1) was used. Hence, this article will discuss the SR method using Uman’s (2011) eight-step process.

Figure 1.

Eight-step process as indicated by Uman (2011).

However, before beginning an SR, it is useful to determine whether the SR protocol should be registered. SRs are most commonly registered with the Cochrane Reviews, PROSPERO, or the JBI. The Cochrane Reviews registration consist of 52 Cochrane Review Groups (CRGs), which focus on the following areas of health: abdomen and endocrine, acute and emergency care, children and families, circulation and breathing, mental health and neurosciences, musculoskeletal, oral, skin and sensory, and public health systems. To register an SR with the CRG, one is required to search the Cochrane library (see https://www.cochranelibrary.com) as all SR titles and published and unpublished protocols are registered in this database and thus avoids any SR duplications. The SR title and protocol must be submitted to the CRG for peer review. Once a title is accepted, the CRGs provide authors with methodological and editorial support to prepare the review and navigate the editorial process. SRs registered with Cochrane are periodically updated (see https://www.cochranelibrary.com/cdsr/about-cdsr).

SRs with a health-related outcome within the fields, such as welfare, public health, health and social care, education, crime, justice, and international development, can be registered with PROSPERO. PROSPERO is a database of prospectively registered SRs and is located within the Centre for Research and Dissemination at the University of York. To register an SR with PROSPERO, the data extraction stage should not have commenced. Registration requires information regarding the design and conduct of the SR. Assessment of SR protocols is not peer reviewed but rather assessed based on the required information provided and scope of the SR. All SRs registered with PROSPERO are published in an open access electronic database. PROSPERO is linked to the Cochrane database but runs its own review process (https://www.crd.york.ac.uk/prospero/).

The JBI also offers an opportunity to register titles of SRs with them. However, this is only open to JBI affiliates. The process is much simpler than Cochrane or PROSPERO. The JBI recommends that eligible review protocols be registered with PROSPERO as well. Any review registered with JBI and PROSPER should include the registration number provided by PROSPERO as well as the reference to the published protocol at the beginning of the ‘Methods’ section of the review report (see Aromataris & Munn, 2020).

Once one has established whether to register the SR, one is ready to begin with the SR using the steps provided.

Step 1: define the research question

The research question is approached just as you would approach a research question for any other research project. Background research on the topic of interest needs to be conducted prior to the research question being developed. Sometimes, a scoping review is conducted before an SR. Research questions for an SR can be descriptive, causal, exploratory, relational, or normative in nature (Cherry & Dickson, 2017). In addition to defining the research question, Uman (2011) advocates that a review title that includes the phrase ‘systematic review’ should be developed as this ensures easier identification of the study in online databases.

The main research question from Hassem and Laher was as follows:

Research Question: What online depression screening tools are available for use by the general public of South Africa?

Often review boards, like the ethics committees at hospitals in South Africa or prospective journals, require SRs to be registered. SRs can be registered with the following institutions: Cochrane institute, PROSPERO, Campbell Collaboration as well as the JBI. One needs to check the criteria for registration with each of the institutions and match this with the nature of the research questions in the SR. The online depression screening tool case study example did not meet the SR criteria for registration with any of these institutions.

Step 2: determining the inclusion and exclusion criteria

Defining inclusion and exclusion criteria is important to formulate at this stage as it will serve as a guide or reference point when developing the search terms. Inclusion or eligibility criteria refer to specific characteristics that a study needs to have met to be included in the review, whereas exclusion criteria stipulate the characteristics of a study that will immediately exclude a study from the review (Cherry & Dickson, 2017; Higgins & Green, 2011).

Commonly used methods for defining the inclusion for quantitative studies are the PICO and PICOS frameworks, while SPIDER, ECLIPSE, and SPICE are commonly used for qualitative studies and mixed method studies (Booth, 2006; Centre for Reviews and Dissemination [CRD], 2009; Cook et al., 2012; Higgins & Green, 2011; Wildridge & Bell, 2002). These are outlined in Table 1. However, a number of other similar tools exist. Booth et al. (2019) report in excess of 30 frameworks for extracting data. These extraction criteria will not always be applicable to all SR studies as was evident from the online depression screening tool case study.

Table 1.

Search frameworks used for quantitative and qualitative studies.

Quantitative search tools		Qualitative search tools
P: population	P: population	S: sample	E: expectation	S: setting
I: intervention	I: intervention	PI: phenomenon of interest	C: client	P: perspective
C: comparison	C: comparison	D: design	L: location	I: intervention
O: outcome	O: outcome	E: evaluation	I: impact	C: comparison
	S: study design	R: research type	P: professionals	E: evaluation
			S: service
Higgins & Green (2011)	CRD, 2009	Cooke et al. (2012)	Wildridge & Bell (2002)	Booth (2006)

CRD: Centre for Reviews and Dissemination.

In addition to utilising a framework from those specified above, the researcher needs to consider the following when stipulating the inclusion and exclusion criteria: publication dates, language in which the publication is written in, and type of publication (published literature and whether grey literature [legislation, government reports, thesis, or dissertations] will be included or not) (Siddaway et al., 2019; Uman, 2011).

Grey literature or information refers to documents which are informally published and not controlled by commercial publishing institutions (Adams et al., 2016). It is recommended to use grey literature when one is assessing guidelines, developing guidelines or policy and/or evaluating the effectiveness of interventions. The inclusion of grey literature is often debated with the point of contention being that published literature generally only represents significant findings and those which were non-significant would not appear in the literature introducing a bias in the SR (Siddaway et al., 2019). Hence, including grey literature aids in reducing the impact of publication bias (Adams et al., 2016). The process of defining and refining inclusion and exclusion criteria is cyclical and will evolve as the study evolves.

The review by Hassem and Laher (2019) did not use any of the frameworks to develop inclusion criteria as this was a review on available online screening tools for depression. In order for articles to be included, articles needed to be published between 1970 and 2019. For including articles, 1970 was chosen as the start date as the first uses of online testing occurred in the 1970s. Further, articles needed to be written in English, the samples used must have been 18 years and older as this would be the eligible sample for the online screening tool, and an article must have included a description of a depression screening tool that had been specifically adapted or designed for online use. Articles were excluded if the article had a patient sample (medical or psychiatric) and if a depression screening tool was combined with another screening instrument and the psychometric properties were not clearly reported for each subscale. Grey literature was excluded from the search as the study specifically required instruments which were psychometrically sound from studies which were methodologically rigorous.

Step 3: formulating a search strategy

Following the identification of inclusion and exclusion criteria, a search strategy needs to be formulated. This step can be divided into two steps, namely: identifying appropriate search terms and identifying databases to search.

Search terms are developed by segmenting the research question into individual concepts (Siddaway et al., 2019). The search terms need to be comprehensive as this will ensure that all relevant records will be identified. In addition, one needs to look at alternative concepts or terms which describe a concept and utilise synonyms as well as the plural or singular words for a given term (Siddaway et al., 2019; Uman, 2011). For example, if one needs to identify web-based psychological tools, the following search terms could be included: internet, web-based, online, computer, assessments, screening tools, and diagnostic tools. There is no specific number of search terms a study would need to have, as this is dependent on the research question/s.

In addition to developing search terms, the databases selected for searching allow one to use Boolean functions to add with the search. Boolean functions are words or special characters such as ‘AND’, ‘OR’, or ‘NOT’ which can be used to exclude or combine search terms. There are various ‘wildcard’ characters such as ‘*’, ‘$’, and ‘#’ which can also be used. The truncation symbol ($) is particularly useful as it allows one to search for words which start with a particular combination of letters, for example, searching ‘Ethic$’ will results in the following variations of the word: Ethical and Ethics also being searched. The truncation symbol (#) allows one to obtain different spellings of a particular word and using the truncation symbol (*) between words will return records which have the exact phrase of the words searched. Wildcards are dependent on the databases used. Hence, it is recommended that one checks the wildcards in each database being searched beforehand. Using Boolean functions generates results that are more focused and productive (Cherry & Dickson, 2017; Siddaway et al., 2019). Table 2 provides an explanation of some of the common Boolean functions which may be used.

Table 2.

Boolean operators utilised in Hassem and Laher (2019).

Boolean functions	Purpose	Case example
AND	Requires all terms or phrases used to be included in each result. The search is narrowed as terms are combined.	depression AND online psychological screening
OR	Requires either term or phrase to be included in each result. This search will broaden the results.	Online OR Web-based OR Internet
NOT	The second search term used will be excluded from in each result.	Online NOT paper-based
#	Allows for different spellings of a word	Behaviour# (behaviour and behaviour)

Selecting a database to conduct the search is critical to the study. Databases are field specific or multidisciplinary. It is recommended that at least two or more databases be searched for an SR (see Siddaway et al., 2019). The selection of a database is dependent on the field of study as well as to whether published and unpublished works are to be included. It is recommended to look the list of electronic databases available for searching as well as the field/s of study that each database taps. In addition to database searching, chain citing (searching through the reference lists of key articles) should be done (Uman, 2011). The following databases can be utilised to search for grey literature: WorldCat, OpenDOAR, OpenGrey as well as Google Scholar (Siddaway et al., 2019). In South Africa, Sabinet is a useful database. It is possible to stop searching on databases once saturation is reached, that is, once the same articles continue coming up in searches or when the article titles are no longer of relevance to the research question/s. Finally, it is important that the specific search terms used for the specific databases searched be documented. Table 3 summarises the databases used in the online depression screening study and the reason these databases were selected (Hassem & Laher, 2019).

Table 3.

Example of database selection and reason for selection from Hassem and Laher (2019).

Search engine	Reason for selection
Sabinet	Provides access to literature from African electronic journals as well as African Digital Repository (includes theses and dissertations). This was necessary to search African studies typically not covered by databases located in the global North.
PsycINFO	Provides access to social science and behavioural research globally.
PubMed	Provides access to literature in the fields of general and mental health globally.
EBSCOHost: Psychology & Behavioural Sciences Collection	Provides access to multidisciplinary literature globally. This was needed as online screening of mental health cuts across psychology, psychiatry, and other possible cognate disciplines.

Step 4: screening the literature

Screening the literature is the most time-consuming part of an SR. During this phase, the selected databases need to be searched using the specific keywords as decided. Following this, article titles and abstracts need to be screened by applying the inclusion and exclusion criteria (Higgins & Green, 2011; Uman, 2011). For this step, it is recommended that referencing software such as Zotero or EndNote be utilised. Such software will allow all results retrieved from the database searches to be saved. A folder for each of the databases searched can be created and articles can be systematically saved. Although we have not used it, the JBI provides access to the System for the Unified Management of the Assessment and Review of Information (JBI SUMARI) which can be used when conducting SRs (see https://www.jbisumari.org). It has different suites which can be used for different steps in the review. Rayyan is another open access software which may be used to screen titles and abstracts (see https://rayyan.qcri.org/welcome). So too is Abstrackr (see http://abstrackr.cebm.brown.edu/). In addition to a referencing software, it is recommended that a reporting system such as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) be utilised (Moher, Liberati, Tetzlaff, Altman, & The PRISMA Group, 2015).

A document containing the PRISMA diagram must be prepared so that a detailed account of the number of records identified or removed can be accurately documented. Figure 2 is the PRISMA diagram from the online depression screening study (Hassem & Laher, 2019).

Figure 2.

PRISMA diagram for Hassem and Laher (2019).

It is recommended that a Microsoft Excel document be opened and references be imported if specific SR software is not being used. A new spreadsheet for each screening process can be opened, and articles are recorded under each section. Once all databases have been searched, bibliography from referencing software of all records retrieved needs to be created. This will alphabetise the references and will be easier for one to remove any duplicate records obtained. Alternatively, if referencing software has an option to remove duplicate records options, this could be used. The duplications must be noted as assists in identifying overlaps in the databases. Once duplicate records have been removed, article titles must be screened.

Creating an electronic bibliography of all titles will help filter out irrelevant titles by searching for key terms in titles that are not applicable. For example, in the online depression screening case study (Hassem & Laher, 2019), the term ‘HIV’ was used and all article titles that had the word HIV were removed. Once irrelevant titles have been excluded, abstracts can be screened for each included title using the inclusion and exclusion criteria. Once all possible inclusion articles have been identified through the title and abstract screening, all full-text copies of each article are retrieved.

The next step entails reading each article diligently and checking if the inclusion and exclusion criteria have been met. During this stage, all reasons for excluding any article must be recorded. It is possible that very few articles meeting your inclusion criteria will be found.

Article selection also requires inter-rater reliability, that is, two independent reviewers screening the literature to reduce bias. However, this is often not practical and as long as a detailed record of the search process was kept, this should suffice (Siddaway et al., 2019). However, in the online depression screening study, an independent reviewer went through all studies retrieved to reach the final set of articles included in the study. Having more than one reviewer is also useful with borderline records. Borderline records are those which fulfil almost all the inclusion or exclusion criteria. Decisions on whether to include or exclude such a record should be discussed among reviewers. If many records are borderline, the inclusion and exclusion criteria may need to be revised (Siddaway et al., 2019).

SRs have been criticised for the appeal the method holds for research funders, the state and groups of and/or individual researchers as a policy defining mechanism. SRs have the power to shift political agendas in any particular direction (Gough et al., 2017). These arguments are however true of most research (see Kramer et al., 2019). However, given these dynamics of power and positionality, it becomes extremely important that issues of reflexivity in research are addressed. Being more explicit about the personal and political in research and increasing the potential for the increased involvement of different sections of society nationally and internationally is an important goal for all research (Gough et al., 2017). Reflexivity can be accomplished through keeping a reflexive journal as well as approaching a librarian who is specialised in conducting systematic searches to conduct the literature search (Rees et al., 2017). Further, having two reviewers involved in the process reduces bias.

Step 5: conducting a quality assessment

Each full-text article identified for inclusion needs to be appraised for its quality to reduce bias in the studies selected. Assessing the quality of the study includes assessing the relevance, study design, methodology (sample, ethics, and procedure), data analysis as well as the reporting of results (Uman, 2011). This will allow the researcher to determine if the study is methodologically rigorous enough to be included in the SR. There are in excess of 80 appraisal tools that have been developed (Siddaway et al., 2019). The most commonly used tools are the risk of bias tool for randomised controlled trials (see Higgins & Green, 2011), the JBI critical appraisal tools (tools specific to various study designs; see https://joannabriggs.org/ebp/critical_appraisal_tools) as well as the Critical Appraisal Skills Programme tool (CASP, Critical Appraisal Skills Programme, 2017).

The CASP Qualitative Checklist Tool is commonly accepted as the instrument of choice in the social sciences (Critical Appraisal Skills Programme, 2017). The CASP tool consists of 10 questions which assess the quality of qualitative articles in terms of the research aims, methodology used, sample selection, data analysis, reflexivity, presentation of findings as well as the values of the research. CASP does not have an appraisal for quantitative or theoretical studies. In examining tools for appraising quantitative studies, it was found that most were aligned to the health sciences and RCTs in particular or contained criteria specific for meta-analytic studies. There were also no criteria for theoretical articles. Hence, Hassem and Laher adapted the CASP tool for quantitative and theoretical articles as indicated in Appendix 1. The quantitative tool consists of 11 items, while the theoretical appraisal tool consists of 6 items. The adapted quantitative CASP tool was utilised along with the CASP qualitative tool in the online depression screening tool study (Hassem & Laher, 2019). The last column in Table 4 provides the quality appraisal scores for the articles included in the depression screening tool study. A total appraisal score for a qualitative study was 11, scores between 8 and 11 were considered strong enough for the article to be included in the SR, scores of 7–4 were considered moderate, and scores of 3–0 were considered weak. The total appraisal score for quantitative studies was 11, and the cut-off score for inclusion was 4.

Table 4.

Example of descriptive data extracted from Hassem and Laher (2019).

Authors	Type of study				Sample demographic		Sample age		Sample gender		Critical appraisal score
Authors	Intervention	Comparison of paper vs. internet test	Internet test validation	Other	Size	Country	Range	Mean	Female	Male	Critical appraisal score
Lin et al. Lin et al.			X		579	Taiwan	18–52	26.5	72.7%	27.3%	11
Donker et al.			X		502	The Netherlands	18–80	43	57%	43%	11
Harvey et al.				X Association study	3401 of which 1161 completed depression screening tool	UK telecommunications company	Not provided	Not provided	Not provided	Not provided	5
Krog et al.				X qualitative study	9 General practitioners	Denmark	44–67	Not provided	3	6	9 (total of 10)

Step 6: data extraction

Data extraction is guided by the research question/s for an SR. There are two types of data that can be extracted, namely, descriptive (study characteristics) as well as analytical data (outcome data; Cherry & Dickson, 2017). Descriptive data can include authors’ publication year, study design, sample size and age range, place or region of the sample obtained, instruments utilised, and theoretical framework utilised. Analytical data refers to the data you would use as data for your study which would be dependent on your research question.

Tables 4 and 5 present the data extracted for the online depression screening study (Hassem & Laher, 2019). Table 4 highlights the descriptive data extracted from each study, whereas Table 5 provides analytical data for the online depression screening tool study example (tool used and reliability and validity data). The data extracted for Table 5 were guided by the research question for the SR.

Table 5.

Example of analytical data extracted from Hassem and Laher (2019).

Authors	MDD tool used	Depression diagnosis criteria	Reliability information	Validity information
Lin et al. Lin et al.	ISP-D	DSM IV	Test retest (184 participants):□ 2 weeks: 0.830□ 2–4 weeks: 0.449□ Longer than 4 weeks: 0.499	Sensitivity: 76.4%, specificity: 81.8% (55 participants)
Donker et al.	➢ SID (Dutch version)➢ CES-D (Dutch version)➢ K10 (Dutch version)	CES-D: previously validated depression scales (BDI)	□ CES-D: α = .92□ K10: α = .90.□ SID consists of only one itemCronbach’s α could not be calculated	□ Correlations among the three measures ranged from .68 of the SID with the CES-D and with the K10 (both p < .001) to .84 of the CES-D with the K10 (p < .001)□ SID, cut-off score: 5; sensitivity: 0.87; specificity: 0.51□ CES-D, cut-off score: 22; sensitivity:0.94; specificity: 0.62□ K10, cut-off score: 29, 31, 32; sensitivity: 0.69–0.81; specificity: 0.67–0.79
Krog et al.	eMDI (Dutch version)	ICD 10	Previous experience with the MDI paper made the eMDI easier to interpret and use. eMDI made process of documenting patient scores easier and efficient. However, the login takes a long time.
Spek et al.	EDS		The internet-administered EDS has good internal consistency comparable to that of the paper-and-pencil EDS α = .87	The internet-administered EDS correlated significantly with the internet-administered BDI (r = .75; p < .001). The positive predictive values were comparable to those found in previous paper-and-pencil studies

ICD 10: International Classification of Diseases, Tenth Revision; BDI: Beck Depression Inventory; DSM: Diagnostic and Statistical Manual of Mental Disorders; MDD: Major Depressive Disorder; ISP-D: Internet-Based Self-Assessment Program for Depression; SID: Single Item Depression; CES-D: Center for Epidemiology Studies Depression; eMDI: electronic Major Depressive Inventory; MDI: Major Depressive Inventory; EDS: Edinburgh Depression Scale.

Step 7: data analysis

Data analysis is dependent on the research question/s and can be quantitative or qualitative. A quantitative data analysis may involve a meta-analysis or some form of content analysis as was done in the online depression study (see Table 5). Qualitative data analysis allows for multiple methods of analysis such as various interpretive and integrative approaches.

Meta-analysis synthesises the quantitative data of multiple studies to determine if the intervention is effective (Sutton et al., 2000). Meta-analysis is represented through heterogeneity, reporting bias, effect size, and fixed or random effect model (Sutton et al., 2000). When a meta-analysis is to be included as part of the SR, it is advised that one consults with a statistician. A statistician would be able to help assess whether this would be possible based on the data presented in each article and how to proceed with the typical effect size analyses that are required.

Hsieh and Shannon (2005, p. 1277) describe content analysis as a family of analytic approaches ranging from impressionistic, intuitive, interpretive analyses to systematic, strict textual analyses depending on the theoretical and substantive interests of the researcher and the problem being studied. For SRs, content analysis is used primarily as a quantitative research method to extract specific categories of information from articles selected for inclusion in the study. Data are coded into explicit categories and then described using descriptive statistics.

Qualitative data analysis would typically allow one to determine the ‘why’ and ‘how’ an intervention works (Hannes & Macaitis, 2012). An interpretive qualitative analysis allows for the researcher to understand the phenomenon and aids in theory development (Cherry et al., 2017). To select an appropriate qualitative data analysis method, one needs to consider the following factors: the research question, the availability and type of data, available resources, time as well as the end result of the synthesis and how the results would be utilised. The following qualitative methods of data analysis can be utilised: thematic synthesis (see Thomas & Harden, 2008), grounded theory for synthesis (see Eaves, 2001; Kearney, 1998), framework synthesis ( see Oliver et al., 2008), meta-ethnography (see Britten et al., 2002; Noblit & Hare, 1988), meta-aggregation (see Lockwood et al., 2015), textual narrative synthesis (see Lucas et al., 2007), qualitative meta-narrative (see Greenhalgh et al., 2005), meta study (see Paterson et al., 2001) as well as a qualitative meta-summary (see Sandelowski et al., 2007).

There is some software available to assist with data analysis in SRs. EPPI-Reviewer is available to Cochrane authors for free but others need to pay (see https://eppi.ioe.ac.uk/cms/er4/). This system includes reference management, screening, data extraction, and risk-of-bias assessment as well as quantitative and qualitative analysis functions. It allows for the coding of text and the generation of keywords. GRADEpro GDT software is also freely available and allows for generating evidence profiles and summary-of-findings tables for SRs (see https://gradepro.org). Cochrane supports Review Manager, but this software is only commercially available and is tailored to the Cochrane SR system (see https://training.cochrane.org/online-learning/core-software-cochrane-reviews/revman/revman-5-download). Review Manager includes built-in meta-analysis features. OpenMeta(Analyst) is software for performing meta-analysis of continuous, binary, or diagnostic test accuracy data (King et al., 2017).

Due to the nature of the research question for the online depression screening tool study, a content analysis was the chosen method for data analysis. Content analysis was conducted using a three-step approach. Data with regard to instrument description, sample selection as well as instrument validation were extracted. This was followed by free coding, category creation, and abstraction. Finally, data were reported using the codes created. Results were discussed in terms of available tools, psychometric properties as well as applicability to the South African context (see Hassem & Laher, 2019).

Step 8: disseminate the findings

The final step in an SR is the dissemination of findings which can be done through various publications, ranging from journal articles to conference presentations and policy documents. The most commonly cited way of reporting SRs is based on the PRISMA 17 item checklist (see Moher et al., 2015). The format of presenting an SR will depend on which type of publication it is presented as and to whom. The best advice is to consult the relevant institutions or publication guidelines for this. Hence, university or American Psychological Association (APA) guidelines could be followed for a thesis, journal guidelines for a journal article, and so on. A recommendation would be to also look at recent similar publications in the same journal or institution to get a sense of the formats required. Siddaway et al. (2019) present useful guidelines that can be used for an SR article.

Conclusion

It is evident that SRs as a method have much to offer psychology and the social sciences. If properly conducted, the method allows for a high-quality synthesis of literature which highlights gaps between what is known and what is yet to be known. The best review articles comment on, evaluate, extend, or develop theory by linking theory to evidence and evidence to theory (Baumeister, 2013). SRs bring together the literature in a specific field on a specific set of questions and can therefore aid researchers and practitioners by collecting and synthesising the evidence ultimately influencing theory, policy, and practice. SRs on the efficacy of clinical interventions, research, or assessment techniques in South Africa can go a long way towards furthering progress in these fields within a shorter period of time as both researchers and practitioners would have a sound basis for decision-making based on a rigorous method. This is particularly pertinent in the South African and African context where access to knowledge is limited due to difficulties with technological access or issues of affordability and paywalls (see Kramer et al., 2019). The value of the SR method is evident. However, many graduate programmes in South Africa do not include SRs as part of the methods curriculum. It is recommended that graduate training programmes consider the addition of SRs as this will promote the use of a much needed method for the South African context.

Footnotes

Appendix 1 Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This work is based on the research supported in part by the National Research Foundation of South Africa (Grant Number:112948).

ORCID iD

Sumaya Laher

References

Adams

Hillier-Brown

F. C.

Moore

H. J.

Lake

A. A.

Araujo-Soares

White

Summerbell

(2016). Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: Critical reflections on three case studies. Systematic Reviews, 5(1), Article 164.

Arksey

O’Malley

(2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32.

Aromataris

Munn

(2020). JBI systematic reviews. In Aromataris

Munn

(Eds.), JBI manual for evidence synthesis. https://doi.org/10.46658/JBIMES-20-02

Aromataris

E. C.

Pearson

(2014). The systematic review: An overview. The American Journal of Nursing, 114(3), 53–58.

Baumeister

R. F.

(2013). Writing a literature review. In Prinstein

M. J.

Patterson

M. D.

(Eds.), The portable mentor (pp. 119–132). Springer.

Booth

(2006). Clear and present questions: Formulating questions for evidence based practice. Library Hi Tech, 24, 355–368.

Booth

Noyes

Flemming

Moore

Tunçalp

Ö.

Shakibazadeh

(2019). Formulating questions to explore complex interventions within qualitative evidence synthesis. BMJ Global Health, 4, Article e001107.

Britten

Campbell

Pope

Donovan

Morgan

Pill

(2002). Using meta ethnography to synthesise qualitative research: A worked example. Journal of Health Services Research & Policy, 7(4), 209–215.

Campbell Collaboration. (2000). https://campbellcollaboration.org/

10.

Centre for Reviews and Dissemination. (2009). Systematic reviews: CRD’s guidance for undertaking reviews in health care. University of York.

11.

Chalmers

Hedges

L. V.

Cooper

(2002). A brief history of research synthesis. Evaluation & the Health Professions, 25(1), 12–37.

12.

Cherry

M. G.

Dickson

(2017). Defining my review question and identity inclusion and exclusion criteria. In Boland

Cherry

M. G.

Dickson

(Eds.), Doing a systematic review: A student’s guide (2nd ed., pp. 43–60). SAGE.

13.

Cherry

M. G.

Smith

Perkins

Boland

(2017). Reviewing qualitative evidence. In Boland

Cherry

M. G.

Dickson

(Eds), Doing a systematic review: A student’s guide (2nd ed., pp. 193–222). SAGE.

14.

Cochrane

A. L.

(1979). 1931–1971: A critical review, with particular reference to the medical profession. In Teeling-Smith

(Ed.), Medicines for the year 2000 (pp. 1–11). Office of Health Economics.

15.

The Cochrane Collaboration. (1992). www.Cochrane.org/

16.

Cooke

Smith

Booth

(2012). Beyond PICO: The SPIDER tool for qualitative evidence synthesis. Qualitative Health Research, 22(10), 1435–1443.

17.

Critical Appraisal Skills Programme. (2017). CASP Qualitative Checklist. http://www.casp-uk.net/checklists

18.

Donker

van Straten

Marks

Cuijpers

(2010). Brief self-rated screening for depression on the Internet. Journal of Affective Disorders, 122(3), 253–259.

19.

Eaves

Y. D.

(2001). A synthesis technique for grounded theory data analysis. Journal of Advanced Nursing, 35(5), 654–663.

20.

Evidence for Policy and Practice Information and Co-ordinating Centre. (1993). https://eppi.ioe.ac.uk/cms/

21.

Gough

Oliver

Thomas

(2017). Systematic reviews and research. SAGE.

22.

Graff

Birkenstein

(2014). ‘They say / I say’: The moves that matter in academic writing (3rd ed.). W.W. Norton.

23.

Grant

M. J.

Booth

(2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91–108.

24.

Greenhalgh

Robert

Macfarlane

Bate

Kyriakidou

Peacock

(2005). Storylines of research in diffusion of innovation: A meta-narrative approach to systematic review. Social Science & Medicine, 61(2), 417–430.

25.

Hannes

Macaitis

(2012). A move to more systematic and transparent approaches in qualitative evidence synthesis: Update on a review of published papers. Qualitative Research, 12(4), 402–442.

26.

Harvey

S. B.

Glozier

Henderson

Allaway

Litchfield

Holland-Elliott

Hotopf

(2011). Depression and work performance: An ecological study using web-based screening. Occupational Medicine, 61(3), 209–211.

27.

Hassem

Laher

(2019). A systematic review of online depression screening tools for use in the South African context. South African Journal of Psychiatry, 25(1), 1–8.

28.

Higgins

J. P.

Green

(2011). Cochrane handbook for systematic reviews of interventions (Version 5.1.0). The Cochrane Collaboration. http://handbook.cochrane.org

29.

Hsieh

H. F.

Shannon

S. E.

(2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288.

30.

International Prospective Register (PROSPERO). (2011). https://www.crd.york.ac.uk/PROSPERO/review_scope.asp

31.

Kearney

M. H.

(1998). Ready-to-wear: Discovering grounded formal theory. Research in Nursing & Health, 21(2), 179–186.

32.

King

Garrity

Stevens

Nussbaumer-Streit

Hartling

Harrod

C. S.

Guise

J. M.

Kamel

(2017). Performing rapid reviews. Tricco

A. C.

Langlois

E. V.

Straus

S. E.

(Eds.), Rapid reviews to strengthen health policy and systems: A practical guide. World Health Organization. CC BY-NC-SA 3.0 IGO.

33.

Kramer

Fynn

Laher

(2019). Research as practice: Contextualising applied research in the South African context. In Laher

Fynn

Kramer

(Ed.), Transforming research in the social sciences: Case studies from South Africa. Wits University Press.

34.

Krog

M. D.

Nielsen

M. G.

J. V.

Bro

Christensen

K. S.

Mygind

(2018). Barriers and facilitators to using a web-based tool for diagnosis and monitoring of patients with depression: A qualitative study among Danish general practitioners. BMC Health Services Research, 18(1), 503.

35.

Laher

Fynn

Kramer

(2019). Trends in social science research in Africa: Rigour, relevance and responsibility. In Laher

Fynn

Kramer

(Eds.), Transforming research methods in the social sciences: Case studies from South Africa (pp. 393–412). Wits Press.

36.

Levac

Colquhoun

O’Brien

K. K.

(2010). Scoping studies: Advancing the methodology. Implementation Science, 5(1), Article 69.

37.

Lin

C. C.

Bai

Y. M.

Liu

C. Y.

Hsiao

M. C.

Chen

J. Y.

Tsai

S. J.

Ouyang

W. C.

C. H.

Y. C.

(2007). Web-based tools can be used reliably to detect patients with major depressive disorder and subsyndromal depressive symptoms. BMC Psychiatry, 7(1), 12.

38.

Lin

C. C.

Y. C.

Bai

Y. M.

Tsai

S. J.

Hsiao

M. C.

C. H.

Liu

C. Y.

Chen

J. Y.

(2003). The validity of an Internet-based self-assessment program for depression. In AMIA Annual Symposium Proceedings (Vol. 2003, p. 911). American Medical Informatics Association.

39.

Lizarondo

Stern

Carrier

Godfrey

Rieger

Salmond

Apostolo

Kirkpatrick

Loveday

(2017). Mixed methods systematic reviews. In Aromataris

Munn

(Eds.), Joanna Briggs Institute Reviewer’s Manual. The Joanna Briggs Institute. https://reviewersmanual.joannabriggs.org/

40.

Lockwood

Munn

Porritt

(2015). Qualitative research synthesis: Methodological guidance for systematic reviewers utilizing meta-aggregation. International Journal of Evidence-Based Healthcare, 13(3), 179–187.

41.

Lucas

P. J.

Baird

Arai

Law

Roberts

H. M.

(2007). Worked examples of alternative methods for the synthesis of qualitative and quantitative research in systematic reviews. BMC Medical Research Methodology, 7(1), Article 4.

42.

Moher

Liberati

Tetzlaff

Altman

D. G.

(2009). PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. https://doi.org/10.1371/journal.pmed.100009

43.

Moher

Shamseer

Clarke

Ghersi

Liberati

Petticrew

Stewart

L. A.

(2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4, Article 1. https://doi.org/10.1186/2046-4053-4-1

44.

Noblit

G. W.

Hare

R. D.

(1988). Meta-ethnography: Synthesizing qualitative studies (Vol. 11). SAGE.

45.

Oliver

S. R.

Rees

R. W.

Clarke-Jones

Milne

Oakley

A. R.

Gabbay

. . . Gyte

(2008). A multidimensional conceptual framework for analysing public involvement in health services research. Health Expectations, 11(1), 72–84.

46.

Oxman

A. D.

Guyatt

G. H.

(1993). The science of reviewing research. In Mosteller

Warren

K. S.

(Eds.), Doing more good than harm: The evaluation of health care interventions (pp. 156–163). New York Academy of Sciences.

47.

Paterson

B. L.

Thorne

S. E.

Canam

Jillings

(2001). Meta-study of qualitative health research: A practical guide to meta-analysis and meta-synthesis (Vol. 3). SAGE.

48.

Peters

M. D. J.

Godfrey

C. M.

McInerney

Soares

C. B.

Khalil

Parker

(2015). Methodology for JBI scoping reviews. The Joanna Briggs Institute Reviewers’ Manual 2015. The Joanna Briggs Institute.

49.

Rees

Sutcliffe

Dickson

Thomas

(2017). The role of reviewer reflexivity: Reflections from a mixed-method consultative systematic review. Global Evidence Summit, Cape Town, South Africa. Cochrane Database of Systematic Reviews, 9(Suppl 1). https://doi.org/10.1002/14651858.CD201702.

50.

Sandelowski

Barroso

Voils

C. I.

(2007). Using qualitative metasummary to synthesize qualitative and quantitative descriptive findings. Research in Nursing & Health, 30(1), 99–111.

51.

Siddaway

A. P.

Wood

A. M.

Hedges

L. V.

(2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annual Review of Psychology, 70, 747–770.

52.

Sutton

A. J.

Abrams

K. R.

Jones

D. R.

Sheldon

T. A.

Song

(2000). Methods for meta-analysis in medical research. John Wiley & Sons Ltd.

53.

Thomas

Harden

(2008). Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology, 8, Article 45. https://doi.org/10.1186/1471-2288-8-45

54.

Tricco

A. C.

Lillie

Zarin

O’Brien

Colquhoun

Kastner

. . . Straus

S. E.

(2016). A scoping review on the conduct and reporting of scoping reviews. BMC Medical Research Methodology, 16, Article 15. https://doi.org/10.1186/s12874-016-0116-4

55.

Uman

L. S.

(2011). Systematic reviews and meta-analyses. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 20(1), 57–59.

56.

Wildridge

Bell

(2002). How CLIP became ECLIPSE: A mnemonic to assist in searching for health policy/management information. Health Information & Libraries Journal, 19(2), 113–115.