A Replication of Failure,Not a Failure to Replicate

Abstract

Purpose:

The increasing role of systematic reviews in knowledge production demands greater rigor in the literature search process. The performance of the Social Work Abstracts (SWA) database has been examined multiple times over the past three decades. The current study is a replication within this line of research.

Method:

Issue-level coverage was examined for the same 33 SWA core journals and the same time period as our 2009 study.

Results:

The mean percentage of issues missing in the current study was 20%. The mean percentage of issues missing in the current study was significantly greater than the mean percentage of issues missing in the 2009 study.

Discussion:

The research of other groups, and that of our own, has failed to prompt the National Association of Social Workers Press (NASW-P) to act. SWA was failing, it is failing, and NASW-P has failed to correct those failures.

Keywords

bibliometric database empirically based practice evidence-based practice evidence-supported interventions impact factor score literature review meta-analysis NASW NASW-P research synthesis scholarship social work abstracts systematic review

The research program discussed in this article began when one of us (G.H.) noticed articles he was searching for were missing from Social Work Abstracts (SWA). After some additional searching and discussion, our research group decided that scholarly database performance in social work might be an issue worth further investigation.

Keeping up with social work research is not easy. Problems with the processing and dissemination of scholarship in social work have often been discussed (e.g., Barker & Thyer, 2006; Epstein, 1990, 2004; Fraser, 1994; Gambrill, 1994; Holden et al., 2008; Howard, 2009; Jenson, 2005; Kirk & Franke, 1997; Lindsey & Kirk, 1992; Pardeck & Meinert, 1999; Reid, 1977; Schilling et al., 2005). As Simpson (1978) noted: “[a] more insidious difficulty is the need to keep abreast of a scattered, unintegrated literature if one is to do social work research, teach it, or use it intelligently” (p. 147). Across varying fields of scholarly inquiry, abstracting and indexing services (A&Is, i.e., scholarly databases) are employed to assist users in uncovering the scholarly information that they need. Inger and Gardner (2008) note:

The dominant subject A&Is—e.g. Biosis, PubMed, SciFinder, focus on structured access to the highest quality information within a discipline. They typically cover all the key literature but not necessarily all the literature in a discipline. Their utility flows from the one-stop-shop nature of the service that they offer and the perceived certainty and reassurance that they offer to users in providing the authoritative source of search results within a discipline. (p. 12, emphasis added)

There have been increases in both the availability and the sophistication of these scholarly databases. Yet, the particular ones chosen to conduct a review may create problems. This issue has been termed database selection bias (Gomersall, 2010; Stansfield, Kavanaugh, Rees, Gomersall, & Thomas, 2012).

Attempts to facilitate information search in social work date back to at least the early 1950s (Graham, Al-Krenawi, & Bradshaw, 2000; Kirk, 1994; Oxford Journals, 2013). Between 1951 and 1955, the Social Work Research Group Abstracts was published by the Social Work Research Group. Abstracts for Social Workers began in 1965 and was then incorporated into Social Work Research & Abstracts during the 1977–1993 period. This effort then evolved in 1994 into the current form: SWA. The National Association of Social Workers Press (NASW-P, 2014) describes SWA as “the primary source of articles on social work and social welfare, as well as on related fields. For over 30 years, it has been the starting point for literature searches in the field” (n.p.). No evidence for this assertion of primacy is provided. EBSCOhost, a vendor selling SWA expands this claim:

The world’s most valuable and comprehensive scholarly, social work studies full-text database:… Social Work Abstracts offers extensive coverage of more than 850 social work and human services journals dating back to 1965. Produced by the National Association of Social Workers (NASW), the database provides indexing and abstracts dealing with all aspects of the social work field, including theory and practice, areas of service and social issues and problems. Researchers seeking scholarly and professional perspectives on subjects such as therapy, education, human services, addictions, child and family welfare, mental health, civil and legal rights, and more will find Social Work Abstracts to be an indispensable resource. (2013, n.p., emphases added; cf. EBSCO Publishing, 2013)

Again, no evidence is provided to support these assertions.

The social science knowledge base is expanding as is both the need for and number of systematic reviews. A search (30/8/13) on PsychInfo OvidSP for the phrase “systematic review” in the title, for the following 4 years, produced the number of hits noted:

2000: 28

2004: 151

2008: 407

2012: 910

Clearly, there is an increasing need for high performing scholarly databases, so that knowledge aggregation processes may proceed effectively and efficiently. A number of comparisons of databases have been published within social work (e.g., Flatley, Lilla, & Widner, 2007; Kmetz, 2005; McFadden, Taylor, Campbell, & McQuilkin, 2012; Ruppel, 2008; Taylor, Dempster, & Donnelly, 2003; Taylor, Wylie, Dempster, & Donnelly, 2007) and beyond (e.g., Norris & Oppenheim, 2007). Our studies followed in the footsteps of or in concert with other examinations of SWA specifically (Jacoby, Murray, Alterman, & Welbourne, 2002; Kemp & Brustman, 1997; Mendelsohn, 1984, 1986; Shek, 2008; Stover, 1993; Tomaiuolo, 1993).

In our original study of 23 social work journals, we found that 38% of the 1,040 issues from a 9-year period were missing from SWA (Holden, Barker, Covert-Vail, Rosenberg, & Cohen, 2008). In fact, the American Psychological Association database PsycINFO (PI) did a statistically significantly better job of covering this group of social work journals (as defined by Journal Citation Reports) than SWA did (29% of issues missing for PI vs. 38% missing for SWA). Note that because a particular journal issue was coded as nonmissing if any article from that issue was present in the data base, there may have been articles missing from the issues that were categorized as present. In other words, as disquieting as this 38% missing result was, the actual situation may have been worse. In addition, it was found that SWA covered NASW-P journals significantly better than non-NASW-P journals (2% vs. 43% of issues missing, respectively).

In the second study, we focused on SWA core journals for a different time period without comparison to PsychINFO (Holden, Barker, Covert-Vail, Rosenberg, & Cohen, 2009). Strothmann (2010) provided comprehensive overview of the derivation of the “core” journal idea beginning with its origin in Bradford’s Law of Scattering (i.e., that approximately one third of the articles in a field will be concentrated in a small set of journals; Black, 2004). Especially in the current world of limited educational funding, the idea of a set of core journals can facilitate journal title acquisition decisions for librarians. Subscriptions to noncore journals are more likely to be discontinued, rendering articles published in those journals inconvenient for researchers to obtain. As we noted in our 2009 article, the 1989 SWA editorial policy stated:

Because the purpose of SWA is to preserve and expand the social work knowledge base … NASW has divided journals into core and noncore. Core social work journals are comprehensively abstracted, and noncore journals, which are reviewed for articles related directly to social work, are either abstracted or cited selectively.… Highest priority is given to the 34 core journals in social work, each of which is abstracted comprehensively. (Beebe & Payne, 1989, p. 2, italics added)

Although the coverage in SWA was better in our 2009 study than in the 2008 study, 15.6% of the 1,028 core issues were missing. The statistically significant bias in coverage, favoring NASW-P journals was still present (2% vs. 16% missing for NASW-P and non-NASW-P journals, respectively; cf. Kmetz, 2005). A database missing 15.6% of journal issues it claims to have covered completely slows the advancement of knowledge.

As these were well-disseminated findings, which have attracted discussion in the literature (e.g., Barretti, 2011), we had thought that perhaps NASW-P would have used the opportunity during the subsequent years to improve the coverage by SWA that our research had found wanting. NASW-P has never contacted our team to discuss our two prior articles, nor have we seen a public announcement by NASW-P detailing what changes, if any, they might have made in SWA in response to the findings of our team and of others. The lack of any response from the NASW-P is troubling, given the multiplicative effects over time as literature develops which was premised on faulty views of the coverage of a database.

Given the importance of this issue and our agreement with many scholars regarding the importance of replication (e.g., Brandt et al., 2014; Ezell, 2011; Howard, 2011; Roediger, 2012; Rosenthal, 1990), the current study intended to explore the degree to which the problems in SWA that were detailed in our 2009 study were corrected. Six null hypotheses subsumed under two general research questions were tested for the 1989–1996 period.

Research question 1: How well does SWA currently cover the journals that it designated as core journals?

The proportion of issues missing in SWA in the current study will not be significantly different from the proportion that one would expect to be missing (i.e., 0).

The proportion of issues missing will not significantly differ across years of publication.

The proportion of issues missing in SWA will not be significantly different for NASW-P and non-NASW-P journals.

Research question 2: Does the coverage of the SWA core journals for the 1989–1996 period differ between the current study and our 2009 study?

The mean proportion of issues missing in the current study will not be significantly different from the mean proportion missing in our 2009 study.

The effect of publication year on the proportion of missing issues will not be significantly different for the current study and our 2009 study.

The effect of journal type (i.e., NASW-P vs. non-NASW-P journals) on the proportion of missing issues will not be significantly different for the current and 2009 study.

Method

As this was a replication, this study followed the same basic procedures as Holden et al. (2009) with exceptions noted.

Sample

The population for these studies was conceptualized as the journals in SWA across its history. One difficulty regarding sampling (especially if we had chosen a random sampling approach) is that coverage dates are not clearly described:

1977 – Present (Ovid - http://tinyurl.com/m3sxdrf).

1968 to present (NASW-Press - http://tinyurl.com/29lykxy).

Fortunately, this discrepancy did not impact our purposive sampling approach.

Inclusion criteria

In both the 2009 study and this direct replication, we selected 33 of the 34 core journals designated by NASW-P in 1989 (cf. Brandt et al., 2014, for an emerging discussion of replacing the modifier direct with close to describe this type of replication). Although Beebe and Payne (1989) mention 34 journals, only 33 were examined in our studies, because one journal ceased publication in 1988 (G. Xu, personal communication, October 11, 2007). There were a total of 1,028 issues across these 33 journals. The choice of this sample time period is meaningful. First, coverage should have been at its best in SWA, as the core journals set had just been designated. Second, these results would both replicate our 2008 study conceptually (Are there coverage issues in SWA?) and extend the results to a new time period (1989–1996).

Exclusion criteria

All SWA content outside of the 1,028 journal issues specified earlier.

Procedure

The journal issue list and data were simply abstracted from our 2009 article. In the previous studies, a CD version of the SWA database (Silverplatter CD, 1977–2006/09 version) was purchased to ensure a permanent record. This option was no longer available so searches were conducted (all on 1/2/13) through the NYU library (Ovid/SP_UI03.08.00.103, Social Work Abstracts Plus: 1968-12/2012), using the Advanced Search database interface to search by publication for target journal titles. The results for each journal were then downloaded for analysis. The use of a different interface (Ovid/SP vs. Silverplatter) is a deviation in terms of direct replication.

To ensure the quality of the data extraction, the two lead authors independently searched the downloaded file for each of the 33 journals, recording the presence or absence of the 1,028 issues. While in the end, interrater reliability is not relevant because disagreements were resolved via mutual review and discussion, the results of this process are presented in Table 1 for the sake of transparency. The simple percentage of agreement was 99.0%. As a number of authors have pointed out, the typical measure of agreement corrected for chance is subject to a phenomenon (κ paradoxes) that may lead to misunderstanding of the actual level of agreement (e.g., Brennan & Prediger, 1981; Feinstein & Chichetti, 1990; Gwet, 2002, 2008; Kuppens, Holden, Barker, & Rosenberg, 2011). So given the unbalanced marginal totals, in addition to kappa (κ = .97), we computed the AC ₁ statistic (Gwet, 2008) which equaled .99. Both indicate excellent interrater agreement.

Table 1.

Interrater Agreement Re: Journal Issue Presence in SWA.

	Rater A		Total
Rater B	Present	Absent
Present	819	6	825
Absent	4	199	203
Total	823	205	1,028

Data Analysis Strategy

The proportions of missing issues (i.e., N missing issues/N published issues) were modeled using a generalized linear model based on a logit link function. Data were analyzed with the program SAS, version 9.3 using PROC GENMOD. The continuous explanatory variable publication year was centered on initial publication year (i.e., 1989), so that the intercept reflected the expected value at the beginning of this period. For interpretational reasons, the estimated logit estimates in a logistic regression model can easily be transformed to percentages using the formula e^logit/(1 + e^logit). The Holm–Bonferroni method (Holm, 1979) was applied to control for multiple hypothesis testing.

A sequential modeling strategy was adopted, reflecting an increased complexity with each successive model. To address the first research question, a baseline model (Model 1) without any explanatory variables was first fitted to estimate the mean proportion of missing issues in the current study. Afterward, publication year and journal type (non-NASW-P vs. NASW-P) were entered as explanatory variables to examine their main effects on the proportion of missing issues (Model 2). To address the second research question, the proportion of missing issues of the 2009 study was added to the database. To test for differences in the mean proportion of missing issues between our current study and our 2009 study, a generalized linear model with one explanatory variable, namely study (i.e., 2009 vs. 2013), was first fitted to the combined data set (Model 3). Afterward, two additional explanatory variables, publication year and journal type (i.e., non-NASW-P vs. NASW-P), were entered as well as two interaction effects to examine whether the effect of publication year and the effect of journal type significantly differed for our current study and our 2009 study (Model 4).

Results

The descriptive statistics at the individual journal (N = 33) level for both the original and the current study are detailed in Table 2. In the original study, five journals had at least one article present for every issue of interest:

Table 2.

Proportion of Issues (N = 1,028) Covered in SWA Across Journals (N = 33) During the 1989–1996 Period.

Journal	Issues Published (1989–1996)^a	Issues Missing (2009): n (Proportion)	Issues Missing (2013): n (Proportion)
1. Administration in Social Work	29	1 (.03)	3 (.10)
2. Affilia	32	1 (.03)	2 (.06)
3. American Journal of Orthopsychiatry	32	8 (.25)	9 (.28)
4. Arete	15	1 (.07)^b	1 (.07)
5. Australian Social Work	32	3 (.09)	3 (.09)
6. (The) British Journal of Social Work	48	18 (.38)	20 (.42)
7. Child & Adolescent Social Work Journal	46	3 (.06)	10 (.22)
8. Child Welfare	48	1 (.02)	3 (.06)
9. Clinical Social Work Journal	32	10 (.31)	10 (.31)
10. The Gerontologist	55	44 (.80)	44 (.80)
11. Health and Social Work	32	1 (.03)	1 (.03)
12. International Social Work	32	3 (.09)	4 (.13)
13. Jewish Social Work Forum (off 2002 Core list)	8	6 (.75)	6 (.75)
14. Journal of Continuing Social Work Education	8	0 (0)	0 (0)
15. Journal of Gerontological Social Work	29	3 (.11)	4 (.14)
16. Journal of Independent Social Work (discontinued 1991) —continued as Journal of Analytic Social Work (1993–1998)	21	2 (.08)	3 (.14)
17. Journal of Marital and Family Therapy	32	8 (.25)	10 (.31)
18. Journal of Social Service Research	25	1 (.04)	2 (.08)
19. Journal of Social Work & Human Sexuality (ends in 1993) Journal of Family Social Work (1995–)	6	0 (0)	1 (.17)
20. Journal of Social Work Education	24	1 (.04)	1 (.04)
21. Journal of Sociology and Social Welfare	32	6 (.19)	6 (.19)
22. Journal of Teaching in Social Work	16	0 (0)	1 (.06)
23. Public Welfare	32	12 (.38)	15 (.47)
24. School Social Work Journal (off 2002 Core list)	16	2 (.13)	3 (.19)
25. Smith College Studies in Social Work	24	0 (0)	2 (.08)
26. Social Casework (then Families in Society)	80	5 (.06)	8 (.10)
27. Social Service Review	32	2 (.06)	3 (.09)
28. Social Work	48	1 (.02)	5 (.10)
29. Social Work in Education (now Children and Schools)	32	0 (0)	3 (.09)
30. Social Work in Health Care	39	1 (.02)	3 (.08)
31. Social Work Research & Abstracts (then Social Work Research)	32	1 (.03)	5 (.16)
32. Social Work with Groups	27	1 (.04)	3 (.11)
33. The Social Worker/Le Travailleur Social	32	14 (.44)	15 (.47)

Note. Table 2 is an expanded version of Table 1 in Holden et al. (2009).

^aFor Haworth Press journals, the copyright date was used for the publication year. Some publishers produce double or triple issues in one physical issue. Since one cannot distinguish which issue an article belongs to in such instances, the double or triple issue was counted as a single issue in this study. In a few instances, the proportion of issues missing appears incorrect, given the number of issues missing and the number of issues published. The proportion of issues missing in the table is the average of the proportion of issues missing for each year for the journal and is therefore susceptible to slight differences due to rounding.

^bThere was a rounding error in the original table, so this cell has been changed from .06 to .07.

Journal of Continuing Social Work Education

Journal of Social Work & Human Sexuality

Journal of Teaching in Social Work

Smith College Studies in Social Work

Social Work in Education

In the replication study, the Journal of Continuing Social Work Education was the only one in that category. Figure 1 shows the percentage of issues missing across the years for the 2009 and the current study.

Figure 1.

Proportion of issues missing for the 1989–1996 time period in the 2009 and current study.

Research Question 1

How well does SWA currently cover the journals that it designated as core journals for the 1989–1996 period? As presented in Table 3, the logit estimate for the constant in the baseline model (Model 1) is −1.366. When converted to percentages by using the previously mentioned formula (i.e., e^−1.366/[1 + e^−1.366] = .20), the constant indicated that on average 20% of issues were missing in the current study. This mean percentage of missing issues was significantly different from zero, χ²(1) = 310.58, p < .001. In total, 40% of the missing issues were attributed to the following five journals:

Table 3.

Logit Estimates for the Generalized Linear Models.

	Model 1^a	Model 2^a	Model 3^b	Model 4^b
Main effects
Intercept	−1.366***	−1.917***	−1.691***	−2.618***
Publication year		0.172***		0.272***
Journal type		−0.977***		−2.363***
Study			0.325**	0.702**
Interaction effects
Publication Year × Study				−0.099
Journal Type × Study				1.385

Note. Holm–Bonferroni method was applied to control for multiple testing.

^aDependent variable is the proportions of missing issues of the 2013 study (n = 264). ^bDependent variable is the proportions of missing issues of the 2009 and 2013 study (n = 528).

**p < .01., ***p < .001.

The Gerontologist (80% of its issues missing)

Jewish Social Work Forum (75% of its issues missing)

Public Welfare (47% of its issues missing)

The Social Worker/Le Travailleur Social (47% of its issues missing)

British Journal of Social Work (42% of its issues missing)

Table 3 also shows the logit estimates for model with the explanatory variables (Model 2). Significant main effects were obtained for both predictors. When converted to percentages, the constant (β₀ = −1.917) indicated that the percentage of missing issues for the base category—a non-NASW-P journal in 1989—was 13%. As publication year increased, the percentage of missing issues significantly increased by 2 percentage points, χ²(1) = 23.64, p < .001, while for NASW-P journals the percentage of missing issues significantly decreased by 8 percentage points, χ²(1) = 10.98, p < .001. Therefore, all three null hypotheses under research question 1 were rejected.

Research Question 2

Does the coverage of the SWA core journals for the 1989–1996 period differ between the current and our 2009 study? To test specific hypotheses, two logistic regression models were fitted to the proportion of missing issues obtained in both studies. The results are presented in Table 3.

Given that NASW was alerted to coverage problems at least as far back as 2008, we examined whether the lack of comprehensive coverage was corrected in the time since publication? Somewhat perplexing, the percentage of issues missing in the current study significantly increased by nearly 5 percentage points compared to the 2009 study, χ²(1) = 7.89, p = .005 (Model 3). The pattern of these increases in missing issues during specific years can be seen in Figure 1.

After applying the Holm–Bonferroni correction for multiple testing, neither the interaction effect between journal type and study, χ²(1) = 4.38, p = .036, nor the interaction effect between publication year and study, χ²(1) = 3.28, p = .070, reached the adjusted statistical significance threshold of p < .025 and p < .05, respectively. Therefore, only one of the three null hypotheses underresearch Question 2 was rejected.

Discussion

A number of researchers have found that SWA has not optimally performed its role in the scholarship system. The current study sought to understand whether corrections to the database had been made subsequent to our prior empirical work which exposed the apparent failure of SWA to completely cover the core literature. Unfortunately, the findings were worse than our 2009 study. Null hypotheses 1a–c, and 2a were rejected and null hypotheses 2b and 2c were not. On average, 20% of the issues examined were missing from SWA. In addition, a positive slope over the publication years and a significant bias in favor of NASW-P journals were observed. The mean proportion of missing issues in the current study was also significantly greater than in the 2009 study that it was replicating.

While our second study in this series of three was not a direct replication of the first and the current study is not a direct replication of the second (because of the changes in the SWA search interface), we have become increasingly convinced of the inadequacy of SWA. Why? Thus far, we have observed:

38% of 1,040 issues missing (Journal Citation Reports journal list, 1997–2005)

15.6% of 1,028 issues missing (from the NASW journals core list, 1989–1996)

20% of 1,028 issues missing (from the NASW journals core list, 1989–1996)

Obviously, given our nonrandom sampling approach in these three studies, we are not generalizing the results beyond the years studied. That said, we think 17 years of performance like this clearly calls the adequacy of the product into question. Perhaps systematic reviewers should be more circumspect when confronted with corporate claims regarding database coverage.

Many tests of databases focus on assessment of the specificity, sensitivity, or precision of searching of databases for particular search efforts. The current study examines a different and larger issue. Here, rather than examining a few specific searches—we evaluated coverage of the database in a more comprehensive manner. Why are these repeated findings so disconcerting? Even with a database that has complete coverage of a set of journals, there may be problems for a variety of reasons: How a manuscript is described by its authors; how it is indexed by the database; inconsistencies in indexing; and/or with the performance of the database software (e.g., Hay, Adams, & Lefebvre, 1996). These types of problems have led to recommendations for systematic reviewers to go beyond electronic database searching and employ additional approaches such as handsearching; contacting the traditional and electronic invisible colleges; searching grey literature; reference harvesting; and citation chaining, and so on (Cooper, 2010; Higgins & Green, 2011; Littell, Corcoran, & Pillai, 2008). Hay, Adams, and Lefebvre succinctly describe why these issues are of vital importance to a practice profession. “Failure to identify trials, for whatever reason, can result in effective treatments not being used or ineffective treatments being advocated” (1996, p. 92). Hay, Adams, and Lefebvre capture the important point here. Clearly, the proportion of issues missing from SWA was statistically significantly different from zero in all three studies. Are these results practically or clinically significant? If you are a systematic reviewer or a practitioner looking for intervention studies on a life-threatening problem—one study, out of one of the missing issues, could have contained extremely pertinent information.

In general, this study provides new findings that build on the previous evidence of NASW’s mishandling of scholarship beyond the recent SWA-related studies noted previously (e.g., Leighninger, 2006; Midgley, 2006; and the related letters to the editor in the December 2006 and March 2007 issues of the Journal of Sociology and Social Welfare). One wonders why there has not been greater open discussion about these problems related to scholarship at NASW. More specifically, the findings of the current study lead one to ask: Why do libraries continue to purchase Social Work Abstracts? While there are likely a variety of reasons, one based on an error may be present. As Flatley, Lilla, and Widner (2007) note: “[t]here has been the persistent but incorrect rumor that accrediting bodies such as CSWE require SWA. In truth, CSWE does not endorse or recommend any particular databases for accreditation of a university’s social work program” (p. 55).

All of the questions regarding implications for the field noted in our original study remain, ripening as unanswered (or unanswerable), with passing time:

How much scholarship in social work is incomplete due to its reliance on SWA?

What mistakes have resulted from literature reviews that were incorrect because of SWA’s shortcomings? What can be done about those mistakes now?

How complete is the coverage of intraissue content?

How do inadequacies in journal issue coverage affect citation analyses? For instance, have the IFSs (impact factor scores) of non-NASW journals been suppressed (relative to NASW journals) as a result of SWA’s better coverage of NASW journals? If yes, how much and for how long?

Was less adequate coverage of non-NASW journals the result of communications errors (publishers with NASW), internal NASW errors, or part of an effort to suppress the IFSs of non-NASW journals (Holden et al., 2008, p. 498)?

During the review process, the editor-in-chief of Research on Social Work Practice and the reviewers observed issues we had not addressed. These are worthy of note and perhaps future investigation. First, what is the place of Google Scholar (GS) in the search process? There are general questions one should ask of any system used for searching. Are those materials obtainable every time you search (e.g., Holden & Barker, 1990)? What body of material is supposed to be accessible via use of a system and what proportion of those materials are actually accessible? User interface/system usability issues may also play a role. Some of the studies that we have read seem to indicate that GS has promise, but that problems remain (e.g., Gehanno, Rollin, & Darmoni, 2013; Ruppel, 2009; Walters, 2006; cf. McFadden et al., 2012, for a social work perspective). A number of recently published articles, some in direct response to Gehanno, Rollin, and Darmoni (Boeker, Vach, & Motschall, 2013; Giustini & Kamel Boulos, 2013), concluded that there were a series of critical issues (e.g., low precision, inadequate results processing, search interface restrictions). Similar criticisms were observed by Bramer, Giustini, Kramer, and Anderson (2013):

Unfortunately, many of the original shortcomings identified between GS and traditional bibliographic databases such as MEDLINE and Embase are still in evidence: GS lacks a controlled vocabulary, search histories and sets cannot be built and manipulated and wildcards and limits (for instance study types) cannot be used precisely. Only the first 1,000 citations of any search in GS are viewable and search strings must be kept under 256 characters. (p. 2)

Moreover, in the discussion of the results of their study of GS, they note another serious problem—the ongoing alterations to search functions within GS.

The second suggestion from the editor-in-chief and reviewers was that we discuss whether or not it makes sense for SWA to continue, given its deficiencies and the interdisciplinary nature of much social work scholarship. The answers to this question appear, at least in part, dependent on the respondent. Given any profits from SWA and the promulgated value of possessing the “definitive social work database” (http://tinyurl.com/m3sxdrf), its associated institutional personnel (NASW/NASW-P) would likely vote against abandoning SWA.

While social workers might feel some loss of professional pride or status if SWA was discontinued, a more relevant question is, would its discontinuance affect their scholarship? Would practitioners be unable to locate scholarship relevant to their practice? We think most could adjust to the loss of SWA quite readily and libraries would save money. The University of Maryland Health Sciences and Human Services Library recommended discontinuance some time ago, but the School of Social Work administration requested that the SWA subscription be continued (Jacoby et al., 2002).

Caveats

Although we strove for accuracy in the search processes, data abstraction and coding, and statistical analysis, the possibility of errors is always present. These findings are restricted to this set of journals and this time period. Both this and the 2009 study analyzed issue level coverage only—not item level coverage. It is possible that issues coded as present in these studies have individual items that are missing from SWA. Finally, one has to wonder about the more negative findings in the current study. These might have been due to coding errors in the original or the current study (two coders were used in the current study but only one in the original study). Then again the change in data base interfaces between the two studies might have been the causal or a contributing factor.

Just as the findings of SWA inadequacy were replicated in the current study, we will repeat the recommendations from the article based on the study being replicated here:

Given that NASW Press and their vendors have likely profited for years from sales of a product that NASW knew was considered defective (from direct communications and from published work, see Green, Baskind, & Bellin, 2002; Kmetz, 2005; Social Work Librarians Group, 2000; Tomaiuolo, 1993) prior to the findings in our original study and this replication, it seems reasonable to ask whether purchasers of SWA are owed a refund from NASW Press. Moving forward, it seems reasonable to suggest that NASW takes responsibility and alert all customers to the problems with SWA. Moreover, if these problems cannot be rectified in a short time, then NASW should seriously consider discontinuing publishing SWA. (Holden et al., 2009, p. 6)

Long ago, Simpson stated: “[t]he toughest question is this: How can we make research utilization an indispensable part of practice, so a social worker will turn naturally to it when faced with difficult problems?” (1978, p. 155). While the amount of change in social workers’ utilization of research since 1978 is debatable, we conclude that there is no question that research utilization would improve if NASW-P provided a better database.

In conclusion, the failures of SWA were replicated. SWA still does not live up to the claims that are used to sell it, in a corporatized fashion, to the profession and to innumerable libraries supporting social work researchers. SWA was failing, it is failing, and NASW-P has failed to correct those failures. Moreover, the research of other groups, and that of our own, has failed to prompt NASW-P to act. Even so, it is still possible for NASW-P to make SWA the model scholarly database that social workers deserve. Let us hope our call for reform is heeded so as to avoid yet another replication of these results.

Footnotes

Authors’ Note

Use of data appearing previously: As is clearly explained, some secondary data that appeared in our 2009 article [doi:10.1177/1049731508329392] is also analyzed in this article.

Declaration of Conflicting Interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The first author is the editor and the second and third authors are founding editors of Information for Practice, a web service that shares some journal article dissemination functions with SWA.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Barker

K. L.

Thyer

B. A.

(2006). An empirical evaluation of the editorial practices of social work journals. Journal of Social Service Research, 32, 17–31. doi:10.1300/J079v32n01_02

Barretti

M. A.

(2011). Women, feminism, and social work journals 10 years later: 1998-2007. Affilia, 26, 264–277. doi:10.1177/0886109911417688

Beebe

Payne

A. H.

(1989). Users guide to social work abstracts. Silver Spring, MD: NASW.

Boeker

Vach

Motschall

(2013). Google Scholar as replacement for systematic literature searches: Good relative recall and precision are not enough. BMC Medical Research Methodology, 13, 1, 131. doi:10.1186/1471-2288-13-131

Black

P. E.

(2004). Bradford's law. In Pieterse

Black

P. E.

(Eds.), Dictionary of Algorithms and Data Structures. Retrieved December 16, 2013 from http://www.nist.gov/dads/HTML/bradfordsLaw.html

Bramer

W. M.

Giustini

Kramer

B. M. R.

Anderson

P. F.

(2013). The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: A review of searches used in systematic reviews. Systematic Review, 2, 115. doi:10.1186/2046-4053-2-115

Brandt

M. J.

Ijzerman

Dijksterhuis

Farach

F. J.

Giner-Sorolla

Grange

J. A.

… van't Veer

(2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224. doi:10.1016/j.jesp.2013.10.005

Brennan

R. L.

Prediger

D. J.

(1981), Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687–699. doi:10.1177/001316448104100307

Cooper

(2010). Research synthesis and meta-analysis: A step by step approach (4th ed.). Los Angeles, CA: Sage.

10.

EBSCO Host. (2013). Social Work Abstracts. Retrieved 4/4/14 from: http://web.archive.org/web/20130129061351/http://www.ebscohost.com/academic/social-work-abstracts

11.

EBSCO Publishing. (2013). Social work abstracts. Retrieved from http://www.ebscohost.com/academic/social-work-abstracts

12.

Epstein

(1990). Confirmational response bias among social work journals. Science, Technology, & Human Values, 15, 9–38.

13.

Epstein

(2004). Confirmational response bias and the quality of the editorial processes among American social work journals. Research on Social Work Practice, 14, 450–458. doi:10.1177/1049731504265838

14.

Ezell

(2011). Are social workers ignoring the “Cornerstone of Science” by failing to replicate their research? Social Work Research, 33, 131–134.

15.

Feinstein

A. R.

Cicchetti

D. V.

(1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43, 543–549.

16.

Flatley

R. K.

Lilla

Widner

(2007). Choosing a database for social work: A comparison of social work abstracts and social service abstracts. Journal of Academic Librarianship, 33, 47–55. doi:10.1016/j.acalib.2006.08.007

17.

Fraser

M. W.

(1994). Scholarship and research in social work: Emerging challenges. Journal of Social Work Education, 30, 252–266.

18.

Gambrill

(1994). Social work research: Priorities and obstacles. Research on Social Work Practice, 4, 359–388. doi:10.1177/104973159400400307

19.

Gehanno

J. F.

Rollin

Darmoni

(2013). Is the coverage of Google Scholar enough to be used alone for systematic reviews. BMC Medical Informatics and Decision Making, 13, 7. doi:10.1186/1472-6947-13-7

20.

Giustini

Kamel Boulos

M. N.

(2013). Google Scholar is not enough to be used alone for systematic reviews. Online Journal of Public Health Informatics, 5, 2. doi:10.5210/ojphi.v5i2.4623

21.

Gomersall

(2010). Systematic reviews in the social sciences: Are we being misled? Retrieved August 24, 2013, from http://www.rin.ac.uk/blogs/guest/alan-gomersall/systematic-reviews-social-sciences-are-we-being-mislead-0

22.

Graham

J. R.

Al-Krenawi

Bradshaw

(2000). The social work research group/NASW research section/Council on social work research, 1949-1965: An emerging research identity in the American profession. Research on Social Work Practice, 10, 5, 622–643.

23.

Green

Baskind

Bellin

(2002). Robert Green, Frank Baskind, and Melissa Bellin reply. Journal of Social Work Education, 38, 492–495.

24.

Gwet

(2002). Computing inter-rater reliability with the SAS system. Statistical Methods for Inter-Rater Reliability Assessment Series, 3, 1–16. doi:10.1348/000711006X126600

25.

Gwet

(2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61, 29–48. doi:10.1348/000711006X126600

26.

Hay

P. J.

Adams

C. E.

Lefebvre

(1996).The efficiency of searches for randomized controlled trials in the International Journal of Eating Disorders: A comparison of handsearching, EMBASE and PsycLIT. Health Libraries Review, 13, 91–96.

27.

Higgins

J. P. T.

Green

(2011). Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0). Retrieved from http://handbook.cochrane.org/

28.

Holden

Barker

K. M.

(1990). Potential for technological dependency: An example. Social Work Research and Abstracts, 26, 35–36.

29.

Holden

Barker

Covert-Vail

Rosenberg

Cohen

S. A.

(2008). Does social work abstracts work? Research on Social Work Practice, 18, 487–499. doi:10.1177/1049731507308986

30.

Holden

Barker

Covert-Vail

Rosenberg

Cohen

S. A.

(2009). Social work abstracts fails again. Research on Social Work Practice, 19, 715–721. doi:10.1177/1049731508329392

31.

Holden

Thyer

Baer

J. C.

Delva

Dulmus

C. N.

Shanks

T. W.

(2008). Suggestions to improve social work journal editorial and peer review processes: The San Antonio response to the Miami statement. Research on Social Work Practice, 18, 66–71. doi:10.1177/1049731507303496

32.

Holm

(1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70.

33.

Howard

M. O.

(2009). Publication practices in social work. Social Work Research, 33, 67–69.

34.

Howard

M. O.

(2011). Are social workers ignoring the “Cornerstone of Science” by failing to replicate their research? A rejoinder. Social Work Research, 33, 67–69.

35.

Inger

Gardner

(2008). How readers navigate to scholarly content. Retrieved from http://www.sic.ox14.com/howreadersnavigatetoscholarlycontent.pdf

36.

Jacoby

B. E.

Murray

Alterman

Welbourne

(2002). Resource selection for an interdisciplinary field: A methodology. Journal of the Medical Library Association, 90, 393–399.

37.

Jenson

(2005). Structural factors and the quality of publication in social work journals. Social Work Research, 29, 67–71.

38.

Kemp

B. E.

Brustman

M. J.

(1997). Social policy research: Comparison and analysis of CD-ROM resources. Social Work Research, 21, 111–120.

39.

Kirk

(1994). A new beginning. Social Work Research, 18, 1, 3–4.

40.

Kirk

S. A.

Franke

T. M.

(1997). Agreeing to disagree: A study of the reliability of manuscript reviews. Social Work Research, 21, 121–126.

41.

Kmetz

(2005). Social work abstracts (OVID) and social services abstracts (CSA). The Charleston Advisor, 7, 2, 14.

42.

Kuppens

Holden

Barker

Rosenberg

(2011). A Kappa related decision: K, Y, G or AC ₁ . Social Work Research, 35, 185–89.

43.

Leighninger

R. D.

(2006). The perils of self-censorship. Journal of Sociology & Social Welfare, 33, 9–10.

44.

Littell

J. H.

Corcoran

Pillai

(2008). Systematic reviews and meta-analysis. New York, NY: Oxford University Press.

45.

Lindsey

Kirk

S. A.

(1992). The role of social work journals in the development of the knowledge base for the profession. Social Services Review, 66, 295–310.

46.

McFadden

Taylor

B. J.

Campbell

McQuilkin

(2012). Systematically identifying relevant research: Case study on child protection social workers’ resilience. Research on Social Work Practice, 22, 626–636. doi:10.1177/1049731512453209

47.

Mendelsohn

H. N.

(1984). Social work online. Database, 7, 3, 36–49.

48.

Mendelsohn

H. N.

(1986). Searching social work abstracts: A review. Database, 9, 22–27.

49.

Midgley

(2006). International social work, globalization and the challenge of a unipolar world. Journal of Sociology & Social Welfare, 33, 4, 11–18.

50.

NASW-P. (2014). Media Kit 2014. Retrieved from http://www.naswpress.org/advertisers/mediaKit_140227.pdf.

51.

Norris

Oppenheim

(2007). Comparing alternatives to the Web of Science for coverage of the social sciences’ literature. Journal of Informetrics, 1, 161–169. doi:10.1016/j.joi.2006.12.001

52.

Oxford Journals. (2013). Social work research instructions to authors. Retrieved from http://www.oxfordjournals.org/our_journals/swr/for_authors/msprep_submission.html

53.

Pardeck

J. T.

Meinert

R. G.

(1999). Improving the scholarly quality of social work's editorial board and consulting editors: A professional obligation. Research on Social Work Practice, 9, 121–127. doi:10.1177/104973159900900116

54.

Reid

W. J.

(1977). A journal for research: An editorial. Social Work Research and Abstracts, 13, 2–3.

55.

Roediger

H. L.

(2012). Psychology’s woes and a partial cure: The value of replication. The Observer, 25. Retrieved September 2, 2013, from http://www.psychologicalscience.org/index.php/publications/observer/2012/february-12/psychologys-woes-and-a-partial-cure-the-value-of-replication.html

56.

Rosenthal

(1990). Replication in behavioral research. In Neulip

J. W.

(Ed.), Handbook of replication research in the behavioral and social sciences (pp. 1–30). Corte Madera, CA: Select Press.

57.

Ruppel

(2009). Google Scholar, social work abstracts (EBSCO), and PsycINFO (EBSCO). The Charleston Advisor, 10, 5–11.

58.

Schilling

Baer

J. C.

Barth

Fraser

Herman

Holden

… Tucker

D. J.

(2005). Peer review and publication standards in social work journals: The Miami statement. Social Work Research, 29, 119–121.

59.

Shek

D. T. L.

(2008). Comprehensiveness of social work abstracts as a database for researchers and practitioners. Research on Social Work Practice, 18, 500–506. doi:10.1177/1049731507314027

60.

Simpson

R. L.

(1978). Is research utilization for social workers? Journal of Social Service Research, 2, 143–157.

61.

Social Work Librarians Group. (2000, February 26). Librarians group business meeting minutes. New York, NY: Author. Retrieved from http://www.cswe.org/

62.

Stansfield

Kavanaugh

Rees

Gomersall

Thomas

(2012). The selection of search sources influences the findings of a systematic review of people’s views: A case study in public health. BMC Medical Research Methodology, 12, 55. Retrieved from http://www.biomedcentral.com/1471-2288/12/55. doi:10.1186/1471-2288-12-55

63.

Stover

(1993, September). The best family studies databases on CD-ROM—A survey of nine products. CR-ROM Professional, 6, 48–52.

64.

Strothmann

(2010). Use of non-social work journals in social work research: Results of a citation analysis. Behavioral & Social Sciences Librarian, 29, 244–266. doi:10.1080/01639269.2010.521011

65.

Taylor

B. J.

Dempster

Donnelly

(2003). Hidden gems: Systematically searching electronic databases for research publications for social work and social care. British Journal of Social Work, 33, 423–439. doi:10.1093/bjsw/33.4.423

66.

Taylor

B. J.

Wylie

Dempster

Donnelly

(2007). Systematically retrieving research: A case study evaluating seven databases. Research on Social Work Practice, 17, 697–706. doi:10.1177/1049731507304402

67.

Tomaiuolo

N. G.

(1993, August). Social work abstracts plus on compact disc. CD-ROM World, 3, 79–81.

68.

Walters

W. H.

(2006). Google Scholar coverage of a multidisciplinary field. Information Processing and Management, 43, 1121–1132. doi:10.1016/j.ipm.2006.08.006