The Positive Contributions of Quantitative Methodology to Social Work Research

Gifts

I will glance at two kinds of position taking. First, those that are sometimes associated with researchers working in the experimental design tradition, and second, caveats and cautions expressed by quantitative scholars regarding the nature of the scientific enterprise.

There was an ambiguity about the value of experimental designs even in the rather overdrawn halcyon days of the 1960s and 1970s. Perhaps, the most enduring of these was the work of Cook and Campbell (1979), for example, in their discussion of “obstacles to conducting randomized experiments in field settings” in Chapter 8 of the first edition. One of the aforementioned U.K. Home Office studies was a controlled trial of the effectiveness of a therapeutic community approach at the Kingswood Training School in Bristol (Clarke & Cornish, 1972; Cornish & Clark, 1975). They concluded:

The research design used in the Kingswood study was signally unsatisfactory on a number of counts. Not only did ethical difficulties inherent in the design contribute to the premature closure of the project, but even if it had been completed as planned, the results would have been of doubtful scientific value since they would have provided a poor basis for any generalisations about effective treatment methods. In addition the research disrupted many aspects of the administration and life in the school and, because it took such a long time (and would have taken even longer if it had gone according to plan), it is doubtful that decisions about treatment policy could have been suspended until its completion. (Clarke & Cornish, 1972, p. 19)

The problems were tellingly expressed by Reid (1988). While he was convinced the helping professions can demonstrate that they are effective, he warned that the practical significance of identified effects is often slight and their durability ambiguous. He sometimes appeared doubtful about the methodological criteria of such studies. In his chastening and memorable metaphor, “It is like trying to decide which horse won a race viewed at a bad angle from the grandstand during a cloudburst” (Reid, 1988, p. 48).

Reid’s work also illustrated the wider reservations by some quantitative scholars about the hegemony of particular assumptions about the nature of scientific knowledge. He was keen to remind me and a coauthor in a prepublication review of our then pending book that in our discussion of common sense, we “might consider some reference to the long-held notion that science is, indeed, as Dewey once put it, ‘common sense, writ large’” (unpublished review). Despite his misgivings about positivist bashing and epistemological grandstanding, he was ready to acknowledge that theoretical assumptions are implicit in scientific practice (cf. Reid, 2002). He developed this in one of his invaluable reviews of the empirical practice movement (Reid & Zettergren, 1999) and returned to it in his and Kirk’s argument that “it may make sense to construe scientific practice as a ‘perspective’ on intervention,” and his willingness “to accredit client ideas about measurement, data collection and the like that might not fit conventional research notions.” They went as far as to say that this understanding of a scientific practice perspective “could be used with advocacy research, even though the practitioner researcher might need to forgo his or her ‘neutrality’” (Kirk & Reid, 2002, p. 89).

Reid was making two kinds of concession that may seem surprising from a superficial reading of his extensive body of work. First, he is nodding acknowledgment to advocacy researchers, and second, he is readily conceding that conventional scientific logic may not be the only player in the game. He may have agreed with Byrne (2011) when he writes that “measurement is a process of interpretation, no less than the processes of interpretation which underpin qualitative research practice” (Byrne, 2011, p. 32). Acknowledging the place of interpretive and perspectival stances in quantitative research opens the way for greater hesitancy in evidence claims.

Cronbach et al. (1980) at Cornell laid out more general hesitations in their profound assessment of program evaluation that repays repeated reading and every time poses new insights (Cronbach et al., 1980). They suggested that summative evaluations have a deeply conservative tendency through powerful biases to showing no effects. They cite a writer to the effect that the great power of these is “when the main purpose is to knock down some new proposal” (Cronbach et al., 1980, p. 158). The question, “Does it work?” is a skeptical question and “functions as an exclusionary gatekeeper” (Bogdan & Taylor, 1994, p. 296). Even assuming the design problems had been solved the question still would not be helpful to practitioners. “Conscientious practitioners do not approach their work as sceptics; they believe in what they do” (p. 297). Bogdan and Taylor (1994) have worked to develop what they call “optimistic research.” ⁸ “We have evolved an approach to research that has helped us bridge the gap between the activists, on the one hand, and empirically grounded sceptical researchers, on the other” (p. 295).

Cronbach et al. (1980) also offered a critique of the injunctions to evaluate against clear and measurable goals, with assignment, and reliable and objective measures that sustain internal validity. While not going as far as what they call the humanist school of evaluation, they say “We do not consider it reasonable to separate the effects of the program from the rest of the client’s experience” (Cronbach et al., 1980, p. 217). Using school evaluation as illustration, they say that a “program that appears superior to a rival program in isolation may be inferior when each program is embedded in the regular sequence of school experience.” In addition, “After the experimenter with his artificial constraint leaves the scene, the operating program is sure to be adapted to local conditions” (p. 217). They comment to similar effect with, as they say, a “stroke or two of caricature” (p. 58) on the notion of evaluation as a summative single stand-alone study in which “the program is to ‘play statue’ while the evaluator’s slow film records its picture” (p. 56).

Cronbach (1986) argued in a later article that to reduce inferential risk we should collect data about the local situation. Independent variables are in fact a local “combination of treatment, units, setting, and so on,” and arguments about causes are oversimplified. He pleads for the importance of the “search” in research, and for not discouraging quantitative researchers “from wide-eyed unstructured observation” (p. 101). He complains of evaluation research being geared to justification rather than discovery. “Investigators … should sniff around the phenomenon and probe unsystematically for a long while before they mount a wrap-up study intended to ‘establish’ what they have perceived” (Cronbach, 1986, p. 102). He also casts doubt on the value of the notion of replication. “A program evaluation is so dependent on its context that replication of it is only a figure of speech” (Cronbach et al., 1980, p. 222).

This “tough-minded master of conceptual distinctions” (Scriven, 1986, p. 15) detects consequences for how we draw inferences. In his important critique of Campbell’s position on internal validity, he argues that “External validity”—validity of inferences that go beyond the data—is the crux of social action, not ‘internal validity’” (1980, p. 231). The statistical inference only allows us to say that something associated with this made a difference. “This is a report on a local historical event, not a conclusion about a recurrent relation” (pp. 314–315). Thus, it is external and construct validity that matter. To this end, evaluation can provide the credible, plausible, and probable—but not the necessary—evidence.

By way of advice he insists “progress requires that we respect poorly formed and even ‘untestable’ ideas.”. In a nice phrase he says, “We should be stern only where it would cost us much to be wrong” (Cronbach, 1986, p. 86). He espouses epistemological modesty in a challenging sense when he says

In my opinion, social science is cumulative, not in possessing ever-more-refined answers about fixed questions, but in possessing an ever-richer repertoire of questions. (p. 91)

A brilliant tour de force, unusually rewarding if closely read, trenchant in analysis of the status quo, and creating truly unique alternatives sensitive to the scholarly needs for general knowledge and the practitioner need for local application. (Shadish, Cook, & Leviton, 1990, p. 375)

Borrowings

The quantitative scholar, Ian Sinclair, goes as far as to say that qualitative methods are in many ways “more adapted to the complexity of the practitioner’s world than the blockbuster randomized control trial (RCT).”

Qualitative research draws attention to features of a situation that others may have missed but which once seen have major implications for practice. It counteracts a tendency to treat the powerless as creatures with something less than normal human feelings. It contributes to an ethically defensible selection of outcome measures. And, in combination with simple statistical description, it can lead to an informed and incisive evaluation of programs in social services. (Sinclair, 2000)

Quantitative social work research does face peculiarly acute difficulties arising from the intangible nature of its variables, the fluid, probabilistic way in which these variables are connected, and the degree to which outcome criteria are subject to dispute. (pp. 9–10)

Patton (2002) gives an example of such logic borrowing in showing how the creation of qualitative matrices is especially useful for exploring linkages between process and outcome. The analytic sequence entails the development of categorizations of types and levels of outcomes and of program processes. The categories are developed through orthodox qualitative analysis. The relationships between processes and outcomes may come either from participants or through subsequent analysis.

In either case, the process/outcomes matrix becomes a way of organizing, thinking about and presenting the qualitative connections between program implementation dimensions and program impacts. (Patton, 2002, p. 472)

Suppose we have been evaluating a juvenile justice program that places delinquent youth in foster homes …. A regularly recurring process theme concerns the importance of “letting kids learn to make their own decisions.” A regularly recurring outcome theme involves “keeping the kids straight” … By crossing the program process (“kids making their own decisions”) with the program outcome (“keeping the kids straight”), we create a data analysis question: What actual decisions do juveniles make that are supposed to lead to reduced recidivism? We then carefully review our field notes and interview quotations looking for data that help us understand how people in the program have answered this question based on their actual behaviors and practices. By describing what decisions juveniles actually make in the program, the decision makers to whom our findings are reported can make their own judgements about the strength or weakness of the linkage ….” (pp. 472–473)

A more general influence of quantitative approaches in qualitative research can be seen in the development of structured methods within ethnography. Systematic self-observation is one such method. Rodriguez and Ryave (2002) explore its use as a research strategy. As a qualitative research tool, they see systematic self-observation as training informants “to observe and record a selected feature of their everyday experience,” so that participants “go about their lives while alertly observing” the matter of interest (p. 2). The focus is on understanding the ordinary, in particular the covert, the elusive and the personal. In an effort to overcome the “numbness to the details of everyday life” (p. 4), respondents are asked to observe “a single, focused phenomenon that is natural to the culture, is readily noticeable, is intermittent … is bounded … and is of short duration” (p. 5) and also to focus on the subjective.

The recording involves writing a narrative about the situation, the participants, what occurred, the words spoken and thoughts/feelings experienced at the time (i.e., not retrospectively), and doing it as soon as possible after the event. In observing, they are instructed in no way to act differently than usual, to never produce instances nor to judge the propriety of the action—“do not judge it, do not slow down, do not speed up, do not change it, do not question it—just observe it” (p. 17). They refer to a key skill as gaining a “new mindfulness” about everyday life. In their own studies, they have used the method to research telling lies, telling secrets in everyday life, withholding compliments and the role of envy in making social comparisons in everyday life—mundane everyday matters that are not far from the concerns of social workers.

Compared with single system designs for evaluating intervention, systematic self-observation may well be a preferred research method in two respects:

It will allow a more contextualized and richer understanding of the nature of a problem in a service user’s life, as part of an assessment and planning process.