Social Work Literature Searching

Abstract

Objectives:

To compare the performance of a range of search facilities; and to illustrate the execution of a comprehensive literature search for qualitative evidence in social work.

Context:

Developments in literature search methods and comparisons of search facilities help facilitate access to the best available evidence for social workers.

Method:

The performance of 14 databases and web search engines was appraised, by applying a search formula for articles relating to perpetrators of intimate partner violence and the process of change.

Results:

Seventy-two out of seventy-eight relevant articles were found on just six of the search facilities used. Social Services Abstracts performed the best. Web search engines did not contribute any unique hits.

Conclusion:

The need to use a range of databases was confirmed. Databases have performed inconsistently across case studies to date. New approaches to pilot-testing facilities and search terms proved useful. Accessing qualitative evidence to inform practice must become more straightforward.

Keywords

databases bibliographic evidence-based practice information storage and retrieval review literature as topic systematic literature searching intimate partner violence domestic violence

In the context of increasing pressure for the social work profession to engage in evidence-based practice (B. Taylor, 2014), advances in the efficiency of the profession’s literature searching methods are to be welcomed. Literature searching strategies should comprise of a number of tasks (Grayson & Gomersall, 2003; Ogilvie, Hamilton, Egan, & Petticrew, 2005; Papaioannou, Sutton, Carroll, Booth, & Wong, 2010), for example, hand-searching, searching of trials, registers, or consultations with experts. The anchor of the process, and the focus of the current case study, is the electronic bibliographic database search (Grant, 2004; Reed & Baxter, 2009), a task which has been identified as particularly challenging in social work (Bronson & Davis, 2011; Clapton, 2010; Grayson & Gomersall, 2003; Ogilvie et al., 2005; Papaioannou et al., 2010).

Differing levels of search comprehensiveness are appropriate to differing review purposes (De Brún & Pearce-Smith, 2013). The case study reported here is an example of highly comprehensive database searching, which prioritizes search sensitivity (the extent to which all relevant articles are located) over search precision (the extent to which irrelevant search hits are excluded). This is the type of literature searching advocated in guidance for Cochrane reviewers (J. Higgins & Green, 2009, p. 95). The current study will be of particular interest to those tasked with synthesizing and imparting social work evidence (those in training or expert roles and researchers completing systematic reviews); it provides empirical evidence of database performance and illustrates key methodological considerations in completing highly sensitive database searches in social work.

While evidence sought by health care professionals is largely accessible via one or two databases (Grayson & Gomersall, 2003; Nourbakhsh, Nugent, Wang, Cevik, & Nugent, 2012; B. Taylor, Wylie, Dempster, & Donnelly, 2007), social work, by comparison, draws upon a variety of academic disciplines and a large range of search facilities can be relevant. In addition, the field of search facilities available to social work is changing, partly because of advances in online access and partly because of changes in the bibliographic database market. In response to these issues, a small body of literature has emerged which provides insight into the relative utility of the different search facilities used by social work professionals.

B. Taylor, Wylie, Dempster, and Donnelly (2007); and Taylor, Dempster, and Donnelly (2003) found that Medline and the Social Science Citation Index (SSCI) performed best in response to questions centered on social work decision making with older people.

McFadden, Taylor, Campbell, and McQuilkin (2012) found that the SSCI, Google Scholar, and the Cumulative Index to Nursing and Applied Health (CINAHL) performed well in a question regarding social worker resilience.

Clapton (2010) found that Social Care Online, PsycINFO, and ChildData performed well in a test relating to literature on “looked after children.”

Ogilvie, Hamilton, Egan, and Petticrew (2005) found that established social work databases performed poorly in relation to a question on the effectiveness of the promotion of walking and cycling, in comparison to a transport-related database.

Stansfield, Kavanagh, Rees, Gomersall, and Thomas (2012) used a research question relating to children’s views on body size to show that a wide range of databases provided unique and relevant hits (Pubmed performed best) and that the geographical focus of databases are a potential source of bias for literature reviews.

In addition, three comparison studies (Flatley, Lilla, & Widner, 2007; Holden, Barker, Covert-Vail, Rosenberg, & Cohen, 2008; Shek, 2008) have focused on the performance of Social Work Abstracts in particular. They illustrate shortfalls in this facility and relate these to geographical bias and variable database management through time, resulting in missing journal volumes.

We would suggest, following our review of previous studies, that database performance appears to be variable across social work subtopics and that a significant number of relevant articles can often be found uniquely on one database. There is therefore a high possibility of missing key evidence if a search is performed on one or only a small number of databases (Grayson & Gomersall, 2003; Holden et al., 2008; B. Taylor et al., 2007).

The choice of databases used in a literature search has obvious knock-on effects for the accuracy of systematic reviews and, in turn, for evidence-based practice in social work. However, our review of previous studies of database performance highlights large gaps in the evidence to guide our choice of database. In many areas of social work, there are no published comparisons of database utility available. The question remains as to which databases should be employed for which subtopics, and which, if any, should be used in all searches.

Purpose

This case study stands as an addition to the evidence described above. It takes a fresh view of the potential sources of evidence available (including web search engines not previously considered, Bing and Microsoft Academic Search) and online research community resources (CiteULike and Mendeley). It casts the net widely to consider new additions to the field of search facilities marketed to social work professionals (Scopus). It also illustrates a more exhaustive approach to literature searching than that demonstrated in previous case studies.

The study uses a social work subtopic area, working with perpetrators of intimate partner violence (IPV), for which there are no previously reported investigations into which databases work best. This is an area in which the evidence base for intervention is under considerable debate (Debbonaire, 2012; Dutton & Corvo, 2007; Smedslund, Dalsbø, Steiro, Winsvold, & Clench-Aas, 2012) and one in which the calls for a focus on the evidence base have been particularly explicit (Eckhardt, Holtzworth-Munroe, Norlander, Sibley, & Cahill, 2008; Polaschek & Collie, 2004; Vlais, 2011). The IPV topic is also characteristic of social work searching more generally in that it is cross-disciplinary (spanning health care, criminology, sociology, and psychology), reflecting the inherent complexity of many social work topics.

Also in reference to the generalizability of the current study, the literature search question should be considered, “what are the perspectives of program participants, and their partners, on the process of change experienced while involved in an IPV program?” Qualitative evidence, addressing questions of perspectives on social problems and care processes are of particular importance in social work research (Holosko, 2010). The process of finding this evidence is recognized to be challenging (Evans, 2002; Shaw et al., 2004; Stansfield et al., 2012). The current study question illustrates the difficulties: it was not easily refined to avoid picking up studies on practitioner perspectives or the large body of research on IPV survivor experiences unrelated to perpetrator programs. The case study is therefore relevant to literature searching in social work more generally through choice of topic and research question.

While direction on the mechanics of completing an effective and efficient literature search exists, it is offered in summary terms as part of a commentary on the literature review process, for example, Chandler, Churchill, Higgins, Lasserson, and Tovey, 2013; Cooper, 2010; Littell and Corcoran, 2010; Petticrew and Roberts, 2006. The significance of database choice is a common theme in these summaries; there is consensus that a range of databases must be chosen to obtain a comprehensive search result in the social sciences. Readers are also introduced to Boolean algebra (AND, OR, and NOT commands) and some “search syntax” (e.g., commands which find words in close approximation). Exemplars of search strategy operationalization are also available, Fisher, Qureshi, Hardyman, and Homewood, 2006; Littell and Corcoran, 2010; Rutter, Francis, Coren, and Fisher, 2010.

This case study is reported in some detail in order to build on this direction and emulate, to some degree, the practical direction available to our counterparts in health care such as De Brún and Pearce-Smith’s (2013) end-to-end illustration of literature searching. De Brún and Pearce-Smith’s guide is a useful reference, but the health care sector has the advantages of long-established databases employing inbuilt thesauruses and benefits from extensive indexing of a more limited range of relevant terms. Health care literature searching guidance focuses on finding quantitative studies; it covers the hierarchy of outcome evidence, search filters for randomized control trials (J. Higgins & Green, 2009), and the formulation of medical research questions, in the problem–intervention–comparison–outcome format (De Brún & Pearce-Smith, 2013), which are of little value to the researcher tasked with finding qualitative evidence. Conversely, the emphasis of this study is on working with the large contingent of possible search facilities and search terms, which challenge the efficiency and effectiveness of searching for qualitative evidence in social work.

In summary, the current case study is reported to show which search facilities a social work researcher should use to find material relating to an IPV-related research question on intervention processes. A secondary objective of the study is to contribute to the development of a method which is appropriate to comprehensive literature searching for qualitative material, one which responds to the particular challenges of finding the evidence in social work.

Method

De Brún and Pearce-Smith (2013) and Reed and Baxter (2009) present the process of literature searching, in health care and in social sciences, respectively, as a series of steps. The search method employed in this case study was a development of these models but also incorporates elements of McFadden et al.’s (2012) recent database comparison method.

Establishing Criteria to Direct the Appraisal of Article Relevance

“Whether or not to include a paper in the synthesis is a judgement that combines several different aspects of relevance and technical quality,” Fisher et al., 2006, p. 17. This article stops short of considering how the quality of an article or study can be assessed to concentrate on how criteria for measuring relevance can be applied. To meet the minimum threshold of relevance, articles had to satisfy the following criteria.

Articles had to describe a study which captured data on the views of some aspect of a program of intervention for IPV perpetrators. Studies of case work with IPV perpetrators or their families were excluded. As many social care interventions for this service user grouping are integrated with judicial directives, studies which considered perspectives on court-directed intervention, or police involvement, were accepted as relevant.

Study participants had to be, or had to have been, IPV perpetrators or the partners of IPV perpetrators. A number of identified studies investigated the perspective of practitioners who work on IPV perpetrator programs. These studies had the potential to shed light on how IPV perpetrators and their partners view and experience perpetrator interventions, but as they were not about the service user perspective they were not accepted as relevant hits; these studies were instead filed in what could be called a “fringe grouping” for future reference during a systematic review of literature.

Articles had to report empirical research. Policy documents and theoretical material were excluded.

“Gray literature,” defined by Schöpfel and Farace (2010) as informally published reports and commentary by organizations whose sole purpose is not publishing, can be more recent. Details of study findings may be available in a thesis many months before they are published in a journal, for example. It has also been suggested that there could be less likelihood of publication bias outside mainstream academic publishing (Burdett, Stewart, & Tierney, 2003; McFadden, Taylor, Campbell, & McQuilkin, 2012) and that gray literature harbors important and often original work within the field of social sciences in particular (Rothstein, Sutton, & Borenstein, 2006). For these reasons, gray literature was accepted as valid.

The date that the database searches were completed, April 18, 2013, is the upper limit of the date range for articles considered. Articles dating as far back as 1983 were included; this was the year that Pence and Paymar published an introduction to the Duluth model of practice for work with IPV perpetrators. This development in the field predicated a significant uplift in academic interest, and it was conceivable that the views of IPV perpetrators or their partners may have been presented in those early days of program development.

Previously published examples of literature searches in social care have been limited to English-language articles only. One reason cited for this search limit is the practical problems in having an article translated (Taylor, Dempster, & Donnelly, 2003). However, the abstracts of non-English articles are now usually presented in English. Given that abstracts can present key information, and that the practical difficulties of having text translated have been greatly reduced, with the range of online (often free) software available, non–English-language articles were included as valid search hits.

Compilation of a List of Journal Databases and Web Search Engines

Researchers should make informed decisions on which databases to use; however, in the social care field, the evidence on which to base these decisions is limited (Clapton, 2010). A list of academic journal databases and online search engines that were thought to have the potential to provide relevant material was compiled. Amid rapid changes, in how academic material is disseminated and accessed, this list was compiled on the premise that new databases and emerging channels (such as online search tools) are continually evolving and must be periodically reviewed as potentially effective options.

The list (see Table 1) was compiled using the following sources:

those search facilities commonly used by the authors of systematic reviews related to social work published by the Campbell and Cochrane collaborations;

sources which have been critiqued through the course of previous investigations into literature search methods in social work; and

sources which were reviewed favorably by relevant blogs and online social networks in the social care field.

Table 1.

Database Selection and Appraisal.

	Precision of first 40 hits (%)	Adequate tools^a for search refinement?
Australian Family and Society Abstracts	85	Yes
Applied Social Science Index and Abstracts (ASSIA) via Proquest	100	Yes
Bing	35	No
Campbell Library	100	No
CiteULike	85	Yes
Cochrane Library	50	Yes
Cumulative Index to Nursing and Applied Health (CINAHL)	100	Yes
EMBASE via OVID	100	Yes
Google Scholar (Advanced)	100	No
IBSS via Proquest	100	Yes
MEDLINE via OVID	100	Yes
Mendeley	100	No
Microsoft Academic Search	100	No
National Criminal Justice Research Service (NCJRS) via Proquest	100	Yes
PsycINFO via OVID	100	Yes
SCOPUS	90	Yes
Social Care Online, previously CareData	100	Yes
Sociological Abstracts via Proquest	100	Yes
Social Services Abstracts (SSA) via Proquest	100	Yes
Proquest Health and Medical Complete via Proquest	100	Yes
Proquest Thesis (UK and Ireland) via Proquest	100	Yes
Web Of Science via Web of Knowledge	100	Yes
ZETOC	90	No

^aSearch facilities were deemed to have adequate tools for search refinement if, as a minimum, they accommodated common search syntax (such as truncation and proximity searching) and the use of Boolean operators (AND, OR, and NOT).

Performing a Test-Search on Potential Databases an d Web Search Engines

Using the rationale which underpins pilot work in social research more generally, a “test-search” was designed and conducted on the search facilities in Table 1. Among the purposes of research pilot lies the generation of preliminary data that can help guide the focus of the sampling frame (van Teijlingen & Hundley, 2001). In this context, the sampling frame is analogous to the range of databases investigated.

Preliminary data gathered through the test-search was used to choose between the initial list of 23 facilities. The data were used to make an assessment of each database’s threshold of precision, that is, its ability to exclude irrelevant results. In addition, since the field of interest is a niche within the wide-ranging field of IPV, the tools offered by databases and search engines to filter search outputs according to a prescribed search formula were also assessed.

The test-search was conducted by entering the terms “intimate partner violence” and “journal” into each database/search engine. The term “journal” was required in order to direct web search engines toward academic content. The titles of the first 40 items of each search output were then reviewed by the first author. The precision of each search in referring to academically relevant literature on the topic of IPV is presented in Table 1, alongside an assessment of the tools available to refine searches.

Using the information presented in Table 1, nine of the databases were deselected. Databases and search engines were deselected if they had less than adequate tools for search refinement. Search facilities were also deselected if they returned less than 100% precision in the test-search: if a search facility returns a number of irrelevant results to the unambiguous terms used in the test-search described, then there is little value in tasking that facility with more complex search formulae such as those used in this case study and in systematic database searching more generally. Two exceptions were made to this rule: Google Scholar was not deselected as its use as an academic search facility had recently attracted considerable commentary; Scopus was not deselected because it is a recently launched facility, marketed as holding 50 million records, which had not been investigated in previous literature search studies.

Establishing Concept Groups for a General Search Formula

Entering your research question into a database is unlikely to initiate a comprehensive search of its content. Search formulae, as described by B. J. Taylor et al. (2003), refers to the combinations of terms that are entered in a database or web search engine to find articles pertaining to a particular research question. Search formulae are made up of:

groupings of terms pertaining to a particular concept referred to by McFadden et al. (2012, p. 2) as “concept groups,”

Boolean algebra (AND, OR, and NOT) used to link concepts, and

database-specific search features (such as the proximity filter “adjn” in OVID SP).

A general search structure was created which was then adjusted to suit each database. The first step in creating a general search structure is deciding on the concepts which are most likely to be present in the titles or abstracts of relevant articles. The question which directed this literature search was, “what are the perspectives of program participants and their partners on the process of change experienced while involved in an IPV perpetrator program?” The question can be split into six concepts.

Perspectives,

Program participants and their partners,

Process of change,

Experiences,

IPV, and

Intervention.

The use of up to five concept groups is not unknown in systematic literature searching (see Clapton, 2010). However, it is prudent to consider the added value of each concept, Grayson and Gomersall (2003) advise limiting the number of concepts and maximizing the number of terms employed to represent each concept. The above list of concepts was reduced from six to three as follows.

The concepts “perspectives” and “experiences” were so similar; there was no value in attempting to dissect them from each other even though they each represented different meanings in the research question; these concepts were therefore combined.

It was thought that the additional precision afforded by limiting the search to perspectives of “program participants and their partners” was likely to be small. Arguably the only other populations who are likely to have been studied in this field are practitioners, such as IPV program facilitators.

“Process of change” is quite a narrow concept, and any qualitative feedback on IPV perpetrator programs from program participants and their partners was likely to shed light on the process of change, or lack thereof. Process of change was therefore dropped as it was thought to be too limiting and somewhat redundant.

In this way, three concept groups were decided upon.

Concept Group 1 IPV

Concept Group 2 Interventions

Concept Group 3 Perceptions

Selecting Search Terms for Each Concept Group

The sensitivity of a search (the extent to which all relevant articles are included) is largely dictated by the range of terms used to represent each concept. The more exhaustive the process of finding synonyms, and international variations of key terms, the more likely that all available articles will be located. Possible concept group terms for this search were collected from a number of sources.

Bibliographies from pivotal journal articles and books in the field were reviewed.

OVID’s MeSH mapping tool which indexes key terms in health and social care was used.

Systematic reviews published by the Campbell Collaboration on IPV contained lists of search terms employed which were directly relevant.

Identified terms were further explored for synonyms with a thesaurus.

This produced a result akin to a brainstorm output, as follows:

Abuse in marriage	IPV	Partner assaultive
Batterer	Maltreatment of partners	Partner maltreatment
Domestic abuse	Maltreatment of wives	Partner-violent
Domestic abusers	Marital assault	Relationship violence
Domestic violence (DV)	Marital conflict	Who batter
DV	Maritally violent	Wife abuse
Family violence	Marriage conflict
Intimate partner homicide	Men who abuse their partners

It can be seen that the concept group heading IPV is used as a guide rather than a necessary synonym of all potential concept group terms. For example, the term “batterer” is related to IPV but is not strictly speaking another way of saying IPV. This flexibility was important as the review of relevant bibliographies showed that some articles framed their title and abstract around the service user grouping (batterers, “men who batter,” or “men who are violent”) rather than the social care issue in question—IPV. Consider the following article titles:

“En el grupo tomas conciencia (In group you become aware)”: latino immigrants’ satisfaction with a culturally informed intervention for men who batter (Parra-Cardona et al., 2013).

Self-deception among men who are mandated to attend a batterer intervention program (Smith, 2007).

African American male batterers’ perceptions of treatment program effectiveness (Simmons, 2006).

Casting a wide net for potentially useful concept group terms ensures a high level of search sensitivity, but it can also reduce the precision of a search if the net contribution of each individual term is not considered. Each term must be checked for relevance (Reed & Baxter, 2009). The IPV terms shown above were tested on PsycINFO (via the OVID search platform) for precision. Each term was searched for singularly, and the first 40 hits it returned were reviewed to check that it did not attract articles from another topic area. For example, only half of the hits for “DV” were found to be relevant to the concept group; upon review of the article titles, it could be seen that all these records would be picked up by a search for “domestic violence” or IPV. There was no clear advantage to including the abbreviation “domestic violence” but significant disadvantage as it proffered a lot of irrelevant hits. In this way, some terms were discarded. Other terms were adjusted, for example, when the term “batter*” was entered, it attracted some articles about batteries of psychological tests (the * used here signifies truncation, batter* will also find extensions of “batter,” such as batterer). The treatment of this term in the search formula was therefore adjusted; it was entered as “who batter” and batterer.

Running the Searches

Table 2 illustrates how terms for the concept group IPV were brought together using the PsycINFO database.

Table 2.

Building a Search Formula for the IPV Concept Group.

	Search terms	No. of hits	Notes
1	IPV or Partner Abuse or Intimate Partner Violence or Domestic Violence	13,301	IPV was entered into OVID’s inbuilt thesaurus (APA Thesaurus of Psychological Index Terms) which suggested 3 related terms
3	“family violence”	2,531	—
4	“intimate partner homicide”	70	—
5	((domestic* or partner or marital* or marriage or relationship or wife or wives) adj3 (maltreatment or conflict or violen* or abus* or assault*))	24,296	A number of phrases identified during the scoping-for-terms exercise had a large number of derivations. Individually, these phrases were not widely used but collectively they played a significant role in the literature. These were terms such as “abuse in marriage,” “maltreatment of partners,” and “marital conflict.” To capture multiple derivations of these terms, they were separated into “population” and “concept” and then combined by the proximity locating tool “adj” (adj3 meaning “within three words of”).
6	((domestic* or partner or marital* or marriage or relationship or wife or wives) adj3 (maltreatment or conflict or violen* or abus* or assault*)) not (working conditions or family–work relationship, or work–family, or youth))	23,204	When Search 5 was displayed according to order of “Key Concepts” (an option available under the “Sort By” tool featured in Ovid), it was evident that this search was picking up a significant quantity of articles focused on the work place and youth work. This search eliminates these irrelevant articles using the Boolean operator NOT.
7	Partner-violent.mp	61	The hyphenation in the term “partner-violent” made it incompatible with the approach used in Search 6. Partner-violent is therefore searched for separately here.
8	Batterer	530	—
9	“who batter”	214	—
10	1 or 3 or 4 or 6 or 7 or 8 or 9	39,911	Searches are combined using Boolean operator OR to provide a total output for the IPV concept.

Note. IPV = Intimate Partner Violence. Items in boldface were provided by the database’s inbuilt thesaurus.

Terms for the other two concept groups, “interventions” and “perceptions,” were also tested on the PsycINFO database and the final construction of the general search formula is presented in Table 3.

Table 3.

General Search Formula Applied to all Databases.

Search No.	Search Terms
1	IPV or “partner abuse” or “intimate partner violence” or “domestic violence”
2	family violence
3	intimate partner homicide
4	((domestic* or partner or marital* or marriage or relationship or wife or wives) adj3 (maltreatment or conflict or violen* or abus* or assault*)) not (working conditions or family–work relationship or work–family or youth)
5	Partner-violent
6	Batterer
7	who batter
8	1 or 2 or 3 or 4 or 5 or 6 or 7
10	Intervention
11	Judici* not judicious*
12	Treatment*
13	Program* not (program? ing or program? ed)
14	Therap*
15	Diversion*
16	Sanction*
17	Duluth
18	Prosecut*
19	10 or 11 or 12 or 13 or 14 or 15 or 16 or 17 or 18
20	“attitude change” or “response to intervention”
21	Perception*
22	View*
23	Thoughts
24	Perspective*
25	Reaction*
26	Opinion*
27	Belief*
28	Qualitative
29	20 or 21 or 22 or 23 or 24 or 25 or 26 or 27 or 28
30	8 and 19 and 29

Note. Items in boldface were provided by the database’s inbuilt thesaurus.

Screening Total Hits for Relevant Articles

With minor variations the search formula outlined in Table 3 was applied across 14 databases and search facilities. Each search output was then screened to identify relevant hits. This was done in two phases. Phase one consisted of a review of article titles. The first author reviewed the hits from each database on screen, as a list of article titles. Titles which appeared to be relevant or at least warranted further scrutiny were exported, with their abstract, and filed under database name in the reference management application EndNote. This process deselected around 97% of the initial search outputs. The second author replicated this process on a sample of the initial search output (the International Bibliography of the Social Sciences [IBSS] database output was chosen as an appropriately sized sample). This served as a reliability check on the process of weeding out explicitly irrelevant articles.

Phase two consisted of a more in-depth review of the remaining articles. The first author reviewed the abstracts and, as necessary, parts of article text to assess each article's relevance against exclusion and inclusion criteria. The second author reviewed abstracts of the resulting set of articles. A final set of relevant articles was agreed upon by the first and second authors.

Identifying Unique Hits

Unique hits are journal articles which were found on one database only. Determining the number of unique hits found on each database is important; databases which retrieve relevant articles, not found on the other databases, are of particular value to the overall search. Unique hits were identified by:

tagging all selected articles with the database they were located on,

pooling all selected articles together in a single list,

sorting alphabetically by first author, and

reviewing this list and noting which articles were found just once and upon which database.

Calculating Sensitivity

The extent to which each database retrieved all relevant journal articles in existence, that is, the sensitivity of the database (McFadden et al., 2012) was calculated using the total number of relevant items retrieved across all databases (not counting duplicates) as a denominator:

d a t a b a s e s e n s i t i v i t y = \frac{r e l e v a n t d a t a b a s e h i t s}{r e l e v a n t h i t s f r o m a l l d a t a b a s e s}

Calculating Precision

The ability of each database to avoid retrieving irrelevant items, the precision of the database (McFadden et al., 2012) was also calculated.

d a t a b a s e p r e c i s i o n = \frac{r e l e v a n t d a t a b a s e h i t s}{t o t a l d a t a b a s e h i t s}

Results

The main purpose of this study was to build on the work of previous researchers who have provided critical appraisal of search facilities used on a range of social care topics. The total number of relevant articles retrieved in this search across fourteen databases was 78. These 78 articles comprised of qualitative studies (approximately 70%), surveys (approximately 10%), and mixed method studies (predominantly interviews and surveys, around 20%).

These 78 articles comprised of articles which were found uniquely on one database (53 articles, see Table 4) and those which were found on more than one search facility (25 articles). In some cases, articles were found on up to eight search facilities; this explains how the total number of relevant hits across the whole search was much higher (184, see Figure 1) than the actual number of articles available for synthesis.

Figure 1.

Search overview. Note: This overview is based on guidance from the Preferred Reporting Items For Systematic Reviews And Meta-Analyses: the PRISMA statement (Moher, Liberati, Tetzlaff, & Altman, 2009). It should be noted that the current search format departs from that recommended by Moher et al. in which duplicate articles were removed at the end of the process instead of at the beginning. This facilitated a comparison of each search facility’s ability to provide eligible articles, both uniquely, and which other facilities had also identified.

Table 4.

Database Performance Indicators.

Database	Total hits retrieved	Relevant hits	Unique hits		Sensitivity		Precision
Database	Total hits retrieved	Relevant hits	No.	Rank	%	Rank	%	Rank	Average ranking
SSA	373	30	9	2	38	2	8	1	1.7
PsycINFO	912	31	11	1	40	1	3	4	2.0
ASSIA	335	21	4	4	27	3	6	2	3.0
Sociological Abstracts	481	17	9	2	22	5	4	3	3.3
CINAHL	749	18	7	3	23	4	2	5	4.0
NCJRS	299	12	7	3	15	7	4	3	4.3
SCOPUS	299	13	1	6	17	6	4	3	5.0
EMBASE	918	11	2	5	14	8	1	6	6.3
IBSS	297	5	1	6	6	10	2	5	7.0
Proquest Health and Medical Complete	383	7	0	7	9	9	2	5	7.0
Social Care Online (previously CareData)	385	5	1	6	6	10	1	6	7.3
Web Of Science (incorporating SSCI)	1336	7	0	7	9	9	1	6	7.3
Google Scholar (Advanced function)	100	2	0	7	3	11	2	5	7.7
MEDLINE	1438	5	1	6	6	10	0	7	7.7
Total	8305	184^a	53	—	—	—	—	—	—

Note. Abbreviations are defined in Table 1.

^aThis column total refers to “hits” not articles. Some of the articles were found up to eight times, that is to say that the same article was found on eight different search facilities. While the number of relevant hits is quite high, the number of unique relevant articles found was 78;53 of which (see column 4 above) were found only once and 25 of which were found on two or more facilities.

Database and Search Engine Performance Indicators

The study employed three measures to investigate the relative usefulness of each database and web search engine for this research question: (1) the number of relevant articles uniquely identified, (2) sensitivity, and (3) precision. Table 4 ranks each of the facilities against each of these three performance indicators and places them in order based on an average of these rankings.

Top Performing Databases

The top ranking six databases provided 92% of the 78 relevant articles found (47 of the 53 articles found uniquely on one database, and all of the 25 articles which found on multiple databases were also available on these top-performing databases). Figure 2 presents the relative strengths of these databases in terms of precision, sensitivity, and total number of relevant hits.

Figure 2.

The top six performing databases.

Discussion and Applications to Social Work

This case study has shown that a number of databases performed well in this topic area. It has also illustrated the challenge of completing a highly sensitive search for qualitative evidence. These findings are discussed here in relation to previous database comparisons and in relation to their implications for the development of a literature search method suited to social work.

The best performing database overall was Social Services Abstracts (SSA). A review of the journal titles covered by SSA reveals a predominance of social work, health, family, and gender-related titles. These journal titles relate closely to the case study research question, and while other factors such as the quality of indexing, consistency of database management, and the availability of search refinement tools have been cited as key factors in database performance, it is suggested here that the relevance of database coverage is also a key factor. Previous authors of database appraisals have also suggested that database performance is topic dependent (Hood & Wilson, 2001; B. J. Taylor et al., 2003). One explanation for this could be that certain databases are suited to certain kinds of questions (McFadden et al., 2012). McFadden et al. make the distinction between organizational-type questions and more clinical-type questions. On the basis of the current study’s addition to the literature, it could be suggested that PsycINFO and SSA are particularly geared toward intervention process questions, perhaps in keeping with their focus on the fields of psychology and social care.

This literature search took a wide view of the facilities available. Given the current pace of advancement in all things Information Communications Technology (ICT) related, it was thought prudent to pause to take stock of what was currently available. Performing a test search on 23 search facilities proved to be a relatively painless exercise, the greatest difficulty with it being the necessity of becoming familiar with the various search tools available on each database or search engine. The test search helped locate two databases which have been ignored in previous literature search case studies for social work: the National Criminal Justice Research Service and Sociological Abstracts. These two databases contributed sixteen unique hits, which would have been missed had a less objective assessment of the search facilities available, been employed. Note also, eleven out of fourteen facilities in this study provided unique hits (Table 4).

Casting the net widely, over a range of search facilities has paid dividends in terms of the sensitivity of the search, the authors can be particularly confident that all relevant research has been identified. This confidence comes at a price however—a time-consuming screening of a large number of hits. The questions emerge then, as to the practicality and necessity of such a thorough search, featuring a wide-scale screening of search facilities, search terms, and finally of articles. In historical terms, the current study would appear to depart from what is traditionally accepted to be a comprehensive literature search. The overall search precision in this search was low in comparison to previous literature search case studies (see Table 5). A very large number of search hits, 8,305, were screened to find just 78 matches to the specified selection criteria. Adding duplicates to these 78 articles (see Figure 1) results in an overall precision percentage of 2%. However, this low precision rate does not stand in isolation: Akoensi, Koehler, Lösel, and Humphreys (2012) screened 10,446 hits to find just twelve studies; and Sugavanam, Mead, Bulley, Donaghy, and van Wijck (2013) screened 53,998 studies to find seventeen studies. The low rate of precision reported in the current study aligns with a trajectory toward the screening of larger and larger numbers of hits in systematic reviews of qualitative evidence (see Table 5).

Table 5.

Quantity of Search Hits in Previous Studies.

Study	Total search hits	Relevant hits	Precision (%)
Taylor, Dempster, and Donnelly (2003)	597	332	56
Taylor, Wylie, Dempster, and Donnelly (2007)	2,088	607	29
McFadden, Taylor, Campbell, and McQuilkin (2012)	3,455	136	4
Current study (2014)	8,305	184	2

Note. Figures for both total hits and relevant hits include duplicates.

There is an unwelcome implication in this for those tasked with reviewing and synthesizing evidence for social work: considerable resources and a particular set of skills are required. McFadden et al. (2012) suggest the need for assistance from librarians or information scientists in literature searching; this is recognized more generally in social sciences (Grant, 2004) and is an established premise within health care (Lasserre, 2012). In light of the disparate nature of evidence reporting in social work, and the challenges of finding qualitative evidence in particular, properly resourced, properly reported, highly sensitive searches are necessary. Systematic reviews of intervention efficacy studies are commonplace. For example, in relation to the efficacy of interventions with IPV perpetrators, a series of systematic reviews have been published—Babcock, Green, and Robie (2004); Feder, Wilson, and Austin (2008); Smedslund, Dalsbø, Steiro, Winsvold, and Clench-Aas (2012). Systematic reviews of qualitative studies are much less common despite their value in discerning what works for whom and why “ … delving into questions of meaning, identifying barriers and facilitators to change” (Starks & Trinidad, 2007, p. 153). The current study has uncovered 78 articles which can help us understand service user perspectives on IPV perpetrator programs but we could find only one attempt at literature synthesis in this area, Sheehan, Thakor, and Stewart (2012) who searched two databases and identified six relevant studies. The synthesis of qualitative research at a standard comparable to that employed in Campbell and Cochrane systematic reviews is needed on a much larger scale to provide access to an understanding of intervention processes as well as outcomes. This is an argument which has been tabled by various commentators on the development of methods for synthesizing qualitative studies (Dixon-Woods & Fitzpatrick, 2001; Rutter, Francis, Coren, & Fisher, 2010; Saini & Shlonsky, 2012). The practice implications of this gap in research activity are significant: social workers are less likely to get an objective overview of qualitative insights into the moderators and facilitators of intervention success and more likely to rely on narrative reviews or even individual studies to gain an understanding of the process variables, associated with their interventions.

In the short term, given that a step change in the volume of qualitative research synthesis will not happen overnight, researchers interested in IPV perpetrator programs can circumvent some of the work involved in this study by focusing on the most useful databases identified. Referring to Table 4. It can be seen that the overlaps were significant and in some cases this rendered a database redundant or almost redundant. As discussed, the majority of relevant hits were located on just six databases, three of which can be searched at the same time via the Proquest platform; Cooper (2010) encourages the searching of multiple databases simultaneously where possible, to avoid duplication of effort. In effect, 73 out of the 78 articles identified could have been found through just four searches.

There are opportunities therefore to reduce the burden of searching, but only if what we learn, during the painstaking process of completing a comprehensive systematic literature search, is shared. In addition to the comparison of databases provided in Table 4, the findings of the current case study can be compared with previous database searching case studies in social work (Table 6). The 2003 and 2007 studies were on the topic of professional decision making on the admission of older people to institutional care; the 2012 study related to a search on social worker resilience.

Table 6.

Sensitivity and Precision across Four Studies.

	Taylor et al. (2003)	Taylor et al. (2007)	McFadden et al. (2012)	Current study	Taylor et al. (2003)	Taylor et al. (2007)	McFadden et al. (2012)	Current study
	Sensitivity %				Precision %
CINAHL	19	30	20	23	84	32	10	2
Medline	41	52	12	0	48	29	2	0
PsycINFO	n/a	12	13	40	0	51	10	3
SC Online	1	4	2	6	94	30	7	1
SSA	n/a	9	42	39	n/a	41	5	8
SSCI	55	46	45	4	54	20	4	11

Note. CINAHL = Cumulative Index to Nursing and Applied Health; SSA = Social Services Abstracts; SSCI = Social Science Citation Index. Databases which have been investigated at least three times have been included in this analysis.

At this juncture, the early stages of building an evidence base, we could suggest that CINAHL has performed at a consistently moderate level of sensitivity across topics, and Social Care Online performed consistently poorly (see Table 6). We could also surmise, from the differing search sensitivities in Table 6, that there is an unpredictability around database performance across topics. This provides an explanation of researchers’ tendency, more recently, to consult greater numbers of databases and to screen larger numbers of hits. The small-scale comparison in Table 6 hints at the benefits of building an evidence base around database performances and shows how the literature reviewed in the introduction can come together to help social work researchers make more informed choices about literature searching.

Regarding further study in this area, the authors advocate the publication of further database comparison case studies on subtopics not previously investigated. Also, considering that authors of systematic reviews now include significant detail on the search strategies they have used, there is learning to be gleaned from a review of recent systematic reviews. For example, identifying effective search filters (strings of terms that conceptualize each of the different parts of a research question) responds to the challenges of variable terminology in social work. The publication of tried and tested search formulae, such as that detailed in the current study, and investigations of the utility of specific filters such as a methodological search filter for qualitative studies (e.g., McKibbon, Wilczynski, and Haynes, 2006) are also to be welcomed.

A number of limitations of the current study must be acknowledged. These databases and search engines were designed for a number of different types of users. Social Care Online was designed specifically for the social care profession, Sociological Abstracts for the wider range of social science professions while Google Scholar, Web of Science, and Scopus target everyone interested in academic content. This study is therefore not a critique of the quality of search facilities, but a comparison of their usefulness with respect to one particular task within the field of social work research.

Social Work Abstracts is a database which has been appraised in some detail previously with mixed results (see Holden et al., 2008; Shek, 2008; B. J. Taylor, 2009), but it was not accessible via the resources available to the authors and has not been included in this case study. However, it is recognized to be an important resource for social workers and should be included where possible in subsequent literature search case studies.

The pilot exercise used here to select databases and search engines was a quick but arguably a crude method. One criticism of a simple test-search such as this is that not all databases and search engines present their most relevant hits first and a review of the first 40 hits is therefore not an appropriate measure of potential. The counterargument to this is that the test-search employed here reflects how these facilities are used, if the first few pages of a search result do not prove fruitful then the researcher is likely to move on, there are a variety of alternative search facilities to choose from.

While gray literature was accepted when it satisfied search selection criteria, there was no effort in this case study to search for gray literature per se. In hindsight, it is noted that several valuable items of gray literature were found. Subsequent case studies may benefit from a deliberate attempt to find gray literature, through such organizations as the Social Care Institute for Excellence, Research at Home Office or American Institutes for Research, all of which provide free online access to a range of research reports, conference reports, and practice guides (Littell and Corcoran, 2010) provide a further list of gray literature sources).

Conclusion

In the context of previous appraisals of the usefulness of various search facilities, this study underlines the unpredictability of database performance and confirms the importance of using a number of resources when searching for social work-related material. This study has demonstrated some of the mechanics of a thorough literature search, suggesting ways in which search facilities, search terms, concept groups, and search formulae can be chosen and tested. The complexity of systematic literature searching alongside the use of disparate search facilities, and the necessity of reviewing large quantities of article titles and abstracts, may present a barrier to performing highly sensitive searches. It is suggested here that we can combat these issues by linking with librarians or information specialists where available, by sharing the outcomes of thorough searches on social work subtopics (such as the learning from this case study that the majority of relevant hits on this topic could have been found with just four searches) and, at a policy level, by acknowledging the vitality of systematic reviews including reviews of qualitative studies.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Department of Education and Learning, Northern Ireland. PhD Bursary Award.

References

Akoensi

T. D.

Koehler

J. A.

Lösel

Humphreys

D. K.

(2012). Domestic violence perpetrator programs in Europe, Part II: A systematic review of the state of evidence. International Journal of Offender Therapy and Comparative Criminology. 57, 1206–1225.

Babcock

J. C.

Green

C. E.

Robie

(2004). Does batterers’ treatment work? A meta-analytic review of domestic violence treatment. Clinical Psychology Review, 23, 1023–1053.

Bronson

D. E.

Davis

T. S.

(2011). Finding and evaluating evidence: Systematic reviews and evidence-based practice. Oxford, England: University Press.

Burdett

Stewart

L. A.

Tierney

J. F.

(2003). Publication bias and meta-analyses. International Journal of Technology Assessment in Health Care, 19, 129–134.

Chandler

Churchill

Higgins

Lasserson

Tovey

(2013). Methodological standards for the conduct of new Cochrane Intervention Reviews (Version 2.3). Retrieved from http://www.editorial-unit.cochrane.org/sites/editorial-unit.cochrane.org/files/uploads/MECIR_conduct_standards%202.3%2002122013.pdf

Clapton

(2010). Bibliographic databases for social care searching. Social Care Institute for Excellence. Retrieved from http://www.scie.org.uk/publications/reports/report34.asp

Cooper

(2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed., Vol. 2). London, England: Sage.

De Brún

Pearce-Smith

(2013). Searching skills toolkit: Finding the evidence. Oxford, England: John Wiley & Sons.

Debbonaire

(2012). Respect response to criticisms about our approach to accreditation of domestic violence perpetrator programmes. Retrieved from http://www.respect.uk.net/data/files/Briefingpapers/final_respect_briefing_paper_responding_to_our_critics_18th_january_2012___for_newsletter_and_for_website.pdf

10.

Dixon-Woods

Fitzpatrick

(2001). Qualitative research in systematic reviews: Has established a place for itself. BMJ: British Medical Journal, 323, 765–766. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1121325/

11.

Dutton

D. G.

Corvo

(2007). The Duluth model: A data-impervious paradigm and a failed strategy. Aggression and Violent Behavior, 12, 658–667. doi:10.1016/j.avb.2007.03.002

12.

Eckhardt

Holtzworth-Munroe

Norlander

Sibley

Cahill

(2008). Readiness to change, partner violence subtypes, and treatment outcomes among men in treatment for partner assault. Violence and Victims, 23, 446–475.

13.

Evans

(2002). Database searches for qualitative research. Journal of the Medical Library Association, 90, 290.

14.

Feder

Wilson

D. B.

Austin

(2008). Court-mandated interventions for individuals convicted of domestic violence. Campbell Systematic Reviews, 12, 1–49.

15.

Fisher

Qureshi

Hardyman

Homewood

(2006). Using qualitative research in systematic reviews: Older people’s views of hospital discharge. London, England: Social Care Institute for Excellence. Retrieved from http://www.scie.org.uk/publications/reports/report09d.pdf

16.

Flatley

R. K.

Lilla

Widner

(2007). Choosing a database for social work: A comparison of social work abstracts and social service abstracts. The Journal of academic librarianship, 33, 47–55.

17.

Grant

M. J.

(2004). How does your searching grow? A survey of search preferences and the use of optimal search strategies in the identification of qualitative research. Health Information & Libraries Journal, 21, 21–32.

18.

Grayson

Gomersall

(2003). A difficult business: Finding the evidence for social science reviews. London, England: ESRC UK Centre for Evidence Based Policy and Practice, Queen Mary University of London. Retrieved from https://www.kcl.ac.uk/sspp/departments/politicaleconomy/research/cep/pubs/papers/assets/wp19.pdf

19.

Higgins

Green

(2009). Cochrane handbook for systematic reviews of interventions version 5.0.2. West Sussex, England: The Cochrane Collaboration. Retrieved from www.cochrane-handbook.org.

20.

Holden

Barker

Covert-Vail

Rosenberg

Cohen

S. A.

(2008). Does social work abstracts work? Research on Social Work Practice, 18, 487–499.

21.

Holosko

M. J.

(2010). An overview of qualitative research methods. In Thyer

(Ed.), The handbook of social work research methods (2nd ed., pp. 263–272). London, England: Sage.

22.

Hood

W. W.

Wilson

C. S.

(2001). The scatter of documents over databases in different subject domains: How many databases are needed? Journal of the American Society for Information Science and Technology, 52, 1242–1254.

23.

Lasserre

(2012). Expert searching in health librarianship: A literature review to identify international issues and Australian concerns. Health Information & Libraries Journal, 29, 3–15. doi:10.1111/j.1471-1842.2011.00974.x

24.

Littell

J. H.

Corcoran

(2010). Systematic reviews. In Thyer

(Ed.), The handbook of social work research methods (p. 313). London, England: Sage.

25.

McFadden

Taylor

B. J.

Campbell

McQuilkin

(2012). Systematically identifying relevant research: Case study on child protection social workers’ resilience. Research on Social Work Practice, 22, 626–636.

26.

McKibbon

K. A.

Wilczynski

N. L.

Haynes

R. B.

(2006). Developing optimal search strategies for retrieving qualitative studies in PsycINFO. Evaluation and the Health Professions, 29, 440–454.

27.

Moher

Liberati

Tetzlaff

Altman

D. G.

(2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151, 264–269.

28.

Nourbakhsh

Nugent

Wang

Cevik

Nugent

(2012). Medical literature searches: A comparison of Pub Med and Google Scholar. Health Information & Libraries Journal, 29, 214–222. doi:10.1111/j.1471-1842.2012.00992.x

29.

Ogilvie

Hamilton

Egan

Petticrew

(2005). Systematic reviews of health effects of social interventions: 1. Finding the evidence: How far should you go? Journal of Epidemiology and Community Health, 59, 804–808.

30.

Papaioannou

Sutton

Carroll

Booth

Wong

(2010). Literature searching for social science systematic reviews: Consideration of a range of search techniques. Health Information & Libraries Journal, 27, 114–122. doi:10.1111/j.1471-1842.2009.00863.x

31.

Parra-Cardona

J. R.

Escobar-Chew

A. R.

Holtrop

Carpenter

Guzmán

Hernêndez

Zamudio

González Ramírez

2013. “En el grupo tomas conciencia (In group you become aware)”: Latino immigrants’ satisfaction with a culturally informed intervention for men who batter. Violence against women 19, 107–132.

32.

Pence

Paymar

(1983). Education groups for men who batter: The Duluth model. New York, NY: Springer.

33.

Petticrew

Roberts

(2006). Systematic reviews in the social sciences: A practical guide. London, England: Blackwell.

34.

Polaschek

D. L. L.

Collie

R. M.

(2004). Rehabilitating serious violent adult offenders: An empirical and theoretical stocktake. Psychology, Crime & Law, 10, 321–334.

35.

Reed

J. G.

Baxter

P. M.

(2009). Using reference databases. In Cooper

Hedges

L. V.

Valentine

J. C.

(Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 73–102). New York, NY: Russell Sage.

36.

Rothstein

H. R.

Sutton

A. J.

Borenstein

(2006). Publication bias in meta-analysis: Prevention, assessment and adjustments. London, England: Wiley.

37.

Rutter

Francis

Coren

Fisher

(2010). SCIE systematic research reviews: Guidelines (2nd edition) SCIE systematic research reviews. London, England: Social Care Institute for Excellence. Retrieved from http://www.scie.org.uk/publications/researchresources/rr01.asp

38.

Saini

Shlonsky

(2012). Systematic synthesis of qualitative research. Oxford, England: Oxford University Press.

39.

Schöpfel

Farace

D. J.

(2010). Grey literature. In Bates

M. J.

Maack

M. N.

(Eds.), Encyclopedia of library and information sciences (3rd ed., pp. 2029–2039). London, England: CRC Press.

40.

Shaw

R. L.

Booth

Sutton

A. J.

Miller

Smith

J. A.

Young

Dixon-Woods

(2004). Finding qualitative research: An evaluation of search strategies. BMC Medical Research Methodology, 4, 5.

41.

Sheehan

K. A.

Thakor

Stewart

D. E.

(2012). Turning points for perpetrators of intimate partner violence. Trauma, Violence & Abuse, 13, 30–40. doi:https://dx-doi-org.web.bisu.edu.cn/10.1177/1524838011426016

42.

Shek

D. T.

(2008). Comprehensiveness of social work abstracts as a database for research ers and practitioners. Research on Social Work Practice, 18, 500–506.

43.

Simmons

C. A.

Lehmann

Collier-Tenison

(2008). Men’s Use of controlling behaviors: A comparison of reports by women in a domestic violence shelter and women in a domestic violence offender program. Journal of Family Violence, 23, 387–394.

44.

Smedslund

Dalsbø

T. K.

Steiro

A. K.

Winsvold

Clench-Aas

(2012). Cognitive behavioural therapy for men who physically abuse their female partner. Cochrane Systematic Reviews, CD006048. Retrieved from http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD006048.pub2/pdf/standard

45.

Smith

M. E.

(2012). Self-deception among men who are mandated to attend a batterer intervention program. Perspectives in Psychiatric Care, 43, 193–203.

46.

Stansfield

Kavanagh

Rees

Gomersall

Thomas

(2012). The selection of search sources influences the findings of a systematic review of people’s views: A case study in public health. BMC Medical Research Methodology, 12, 55.

47.

Starks

Trinidad

S. B.

(2007). Choose your method: A comparison of phenomenology, discourse analysis, and grounded theory. Qualitative Health Research, 17, 1372–1380.

48.

Sugavanam

Mead

Bulley

Donaghy

van Wijck

(2013). The effects and experiences of goal setting in stroke rehabilitation: A systematic review. Disability and Rehabilitation, 35, 177–190. doi:https://dx-doi-org.web.bisu.edu.cn/10.3109/09638288.2012.690501

49.

Taylor

(2014). Book review: Practitioner’s guide to using research for evidence-based practice. Research on Social Work Practice, 24, 377–378. doi:10.1177/1049731513512155

50.

Taylor

Wylie

Dempster

Donnelly

(2007). Systematically retrieving research: A case study evaluating seven databases. Research on Social Work Practice, 17, 697–706.

51.

Taylor

B. J.

(2009). Invited commentary on papers by Holden et al. and Shek on the quality of social work abstracts. Research on Social Work Practice, 19, 366–369. doi:10.1177/1049731508321560

52.

Taylor

B. J.

Dempster

Donnelly

(2003). Hidden gems: Systematically searching electronic databases for research publications for social work and social care. British Journal of Social Work, 33, 423–439.

53.

van Teijlingen

Hundley

(2001). The importance of pilot studies. Social Research Update, 35, 1–4.

54.

Vlais

(2011). Typologies of men who present to men’s behaviour change programmes. NTV Notes: Research Corner. Retrieved from http://ntv.org.au/wp-content/uploads/docs/resources/2011-ntv-research-corner-typologies-jun11.pdf