Two (Very) Different Worlds

Policymakers’ Thinking and the Dow Jones Averages

My first inkling that policymakers think differently than I normally do came when I was a PhD. student at Stanford University. As it turned out, one of my professors was, at the time, a key player in the educational policymaking process in California. He functioned both as the governor’s informal advisor on education matters and, as a result of a gubernatorial appointment, as the President of the California State Board of Education.

What an opportunity, I thought. It was an especially important opportunity for a former classroom teacher like myself, who, as a teacher, had felt unduly constrained by what I believed were inappropriate and, often, indefensible state policies, especially policies involving standardized testing. Indeed, I had left the classroom for graduate school in part because I hoped to eventually help members of the policy community “get things right”! Now, it seemed, the opportunity to do this had arrived sooner rather than later because one of my professors—who I had easy access to—was, in fact, a major player in the state educational policymaking arena.

So, after briefly exchanging pleasantries with this professor in the corridor one day, I launched into a semi-rehearsed critique of the problems with standardized testing. He soon interrupted me by saying a reasonable facsimile of the following:

Look, I know all about the problems with standardized testing. What you don’t understand is that, when I am making state education policy, I need something like the Dow Jones Averages to be able to make at least somewhat informed decisions. You come up with something better than standardized test scores, and I will be happy to use it. But it has to give me a clear, relatively uncomplicated birds-eye view of what is happening in the state. In the meantime, I have to rely on standardized test scores. Making decisions with inadequate data is preferable to making decisions with no data at all.

I have been thinking about that statement for more than 30 years, and I still cannot decide whether I agree with the professor/policymaker’s comment about making decisions based on inadequate data being preferable to making decisions with no data at all. The other point he made—the point about policymakers liking things simple (the Dow Jones Averages was his metaphor of choice)—however, has been reinforced during my interactions with other members of the policy community, time and time again, as well as during times when I actually played the policymaker role.

An Academic Innocent Abroad (in the Seemingly Odd World of Policymaking)

During the bulk of my academic career, I worked as a faculty member at the Ohio State University which meant I was working 20 minutes away from where the state legislature worked. Over the years, while spending time in and around the state legislature, I got to know a legislative staffer who eventually became the director of Ohio’s Office of Management and Budget (OMB). Every two years while he served in as the OMB director, when it was time to prepare the biennial budget, the director asked me to come down and brief his staff on educational policy issues. “I have a lot of great policy analysts on my staff,” the director told me, “but none of them knows anything about education.”

So, I was invited to fill in the blanks about educational phenomena, but not before the director did a little personal coaching. The director, by this time, knew me well enough to know that I balked when asked to answer policymaker’s inevitable what-works question. I tended to say things like, “Well nothing works the same everywhere,” or “What works depends on what is happening in the context where a policy or program is being implemented.”

The director asked me not to do too much of that kind of talk with his staff. Staff members had to make policy recommendations for the entire state, I was told, and, consequently, they had to think in broad terms and not get immobilized by any differences that might exist between schools and/or the different communities in which the state’s schools were housed. There are always exceptions to the rule, the director noted, but policymakers are in the business of making the rules.

Later, after the director left his position with the state and started his own consulting firm, we often consulted as a team, especially for a brand new foundation interested in improving education. (He still needed someone around who understood education, he told me.) By that point, I had educated the former state budget director and my present consulting partner about the significance of contextual variation. I still found myself, however, having to respond to his questions about what works, though now the question normally was a bit more nuanced. “What works most of the time?” he now asked me, and he followed up with statements like the following:

I know things are different in different places, but certainly there are some patterns out there, aren’t there? I want to know—and I think our clients want to know—about the patterns so they know what types of programs to fund and which grant proposals to turn down. Even if there are exceptions to the rule, we need to know the rule to get anything done.

By this point, I also had adjusted my own thinking about the policymaking process. I understood what should have been obvious from the start: The role of a policymaker is to make policy and making policy requires the stripping away of complexity. Too much complexity—a focus on too many contingencies, too many storylines, too many “on-the-other-hands,” too much contextual variation—invites immobilization. Policymakers also must believe that their actions—that is, the policies they create—will cause certain things to happen. Elsewhere (Donmoyer, 2011) I have referred to policymakers’ belief in a relatively mechanistic world of cause and effect relationships as a functional fiction. Without this fiction, policymakers would have no incentive to do their job.

Changing Roles (Albeit Briefly): From Observer to Participant Observer of the Policymaking Process

The lesson about too much complexity leading to immobilization was reinforced on a number of occasions when I served in a quasi-policymaking role. To be sure, my stints as a policymaker occurred when I assumed administrative responsibilities in university settings, and, consequently, the policies I made or helped make affected far fewer people than the policies proposed by governors and enacted by state legislatures. Still, functioning in a somewhat limited policymaker role did allow me to move beyond observing the policymaking process from a distance; instead of simply being an observer, I became a participant observer in the policymaking process.

Here I will discuss two occasions when I took on a quasi-policymaking role in a university where I worked. Both occurred when I was a faculty member at the Ohio State University. One occurred at the beginning of my 20-year tenure there; the other happened during the final four years at Ohio State.

When I was hired at Ohio State, I was hired not merely as an untenured assistant professor; my job description also included serving as the director of a teacher certification program. I worked under a wonderful department chair who was the antithesis of your typical university administrator. In fact, he had an extreme aversion to standardization of any sort. Every person and every situation was unique, in his mind, and, consequently, much of the time, he resisted establishing anything that resembled policies. When I would propose a policy for the certification program I was supposed to make work, for example, he would inevitably think of the one hypothetical person in the often distant future who might not be well-served by the policy. I found myself in the unusual (for me) role of arguing for routines and rules because I realized that the complete absence of routines and rules made it impossible to run a program. The complexity associated with treating every situation and every individual as unique, in effect, immobilized me and everyone else involved with trying to make the program work. We desperately needed policies; indeed, for many of us, it seemed that a program without at least some policies to hold it together and choreograph programmatic activity was a program in name only.

During my final four years at Ohio State, I served as the director of the School of Educational Policy and Leadership, a role that required me, at times, to be a policymaker. Among the lessons that got reinforced when I took on this role was the one about simplicity being an antidote for immobilization. I recall one morning when the person I had hired to run my school’s student services office told me: “I really admire your ability to make decisions. You never hesitate; you never delay; you just can decide, and you decide so quickly!”

Now, anyone who knows me in real life—or even in my life as an academic—knows what a ludicrous statement this would be in other contexts. In other contexts, in fact, I am the ultimate Hamlet. I cannot help but consider alternative interpretations and explanations. Indeed, I am incapable of holding a grudge even when holding a grudge would be advantageous for me. Invariably I end up generating credible explanations for why someone who has hurt me or made me angry acted as he or she did, and, even if I conclude that I, personally, would not have done what the person who had wronged me did, the person’s actions become at least somewhat understandable and, consequently, at least somewhat defensible.

This proclivity to reject simplistic heroes-and-villains storylines and to think in more layered and complex ways is so extreme that one of my sons, when he was a hot-shot college boy, told me one day, “I bet even if you were discussing the evils of the holocaust, you would throw in a couple of “on-the-other-hands” that somehow made the Nazis feel good.” I think—at least I hope—he is wrong on the specifics here, but there is no doubt that he did pick up on a character trait that is pretty strong in me.

I am, in short, an on-the-other-hand kind of guy; simplified thinking does not come easily to me. Yet, when I played the policymaker role, I was at least temporarily transformed by the role I played. I could no longer play Hamlet. Often, there was simply no time to engage in on-the-other-hand thinking and, even when there was a little time to think, I had to eventually settle on a relatively simple narrative to act. Decisions had to be made, and decision making required a stripped-down view of reality. Too much complexity created too many doubts and suggested too many options, all of which could easily lead to immobilization.

To be sure, I like to believe that I was the sort of policymaker that was able to hold multiple and even contradictory ideas in my head at one time yet still act, and, when I am kind to myself, I suspect I may have been better at doing this than most people who operate in policymaking arenas. But, if I am honest, I must admit that my thinking while playing the policymaker role was considerably less complex than it is when I play Hamlet in the rest of my professional and personal life.

This Section’s Bottom Line

So, my experiences observing and interacting with policymakers and my limited experience with being a policymaker suggest that simple-mindedness is a virtual necessity in the policymaking process. Too many “on-the-other-hands” and too much attention to contextual variation and alternative possibilities produce immobilization. Too much doubt about the likely impact of a policy one is about to enact leads to inaction. Even if he had not died young and had lived to become a king, I am certain that Hamlet would never have been a very good policymaker (unless, of course, he, too, would have been temporarily transformed by playing the policymaker role).

Qualitative Researchers Embrace Complexity

The thinking of qualitative researchers normally is quite different than the thinking of policymakers. Although policymakers are drawn to and, essentially, require simplified pictures of social life, most qualitative researchers enthusiastically embrace complexity. This is certainly the situation with researchers who play the thick-description game. Geertz (1973/2000), the anthropologist who appropriated philosopher Gilbert Ryle’s notion of thick description and transported the concept to the realm of qualitative research where it is used to describe both what many qualitative researchers do and the products they produce, for example, writes of human beings being “suspended in webs of significance . . . [they themselves] have spun” (Geertz, 1973/2000). He also makes clear that these webs are nothing if not complex.

The task of the qualitative researcher, according to Geertz (1973/2000), is to try to make sense of, that is to explicate, this complexity. For Geertz, qualitative research is “not an experimental science in search of law but an interpretive one in search of meaning” (p. 5). Of course, the act of interpretation only adds to the complexity of the qualitative research enterprise, for, as Geertz notes, qualitative accounts of social life “are themselves interpretations, and second and third order ones to boot. (By definition, only a ‘native’ makes first order ones: it’s his [or her] culture.)” (p. 15).

Geertz (1973/2000), in fact, used a little story to capture the complexity that permeates what he refers to as the “science of interpretation”:

There is an Indian story—at least, I heard it as an Indian story—about an Englishman who, having been told that the world rested on a platform which rested on the back of an elephant which rested in turn on the back of a turtle, asked (perhaps he was an ethnographer; it is the way they behave), what did the turtle rest on? Another turtle. And that turtle? “Ah, Sahib, after that it is turtles all the way down.” (pp. 28-29)

In case his reader missed the point of this story, Geertz (1973/2000) quickly added the following:

Such, indeed, is the condition of things. I do not know how long it would be profitable to meditate on . . . [an observed cultural event]; but I do know that however long I did so I would not get anywhere near to the bottom of it. Nor have I gotten anywhere near to the bottom of anything I have ever written about . . . . Cultural analysis is intrinsically incomplete. And, worse than that, the more deeply it goes the less complete it is. It is a strange science whose most telling assertions are its most tremulously based, in which to get somewhere with the matter at hand is to intensify the suspicion, both your own and that of others, that you are not quite getting it right. That along with plaguing subtle people with obtuse questions, is what being an ethnographer is like. (p. 29)

Clearly, the sort of qualitative research Geertz described is not what policymakers have in mind when they fund researchers to tell them what works ⁴ and, consequently, which initiatives they should “scale-up.” Undoubtedly, more recent qualitative studies that bear the imprint of poststructuralist thought miss policymakers’ mark even more. In some cases, these studies consciously attempt to dramatize the uncertainty that Geertz alludes to by using a variety of rhetorical techniques that, among other things, intentionally undermine the authority of the author’s interpretation (See, for example, Lather’s (1994) discussion of poststructuralist conceptions of validity, and, in particular, Lather’s conception of what she calls ironic validity; and for a concrete example of this version of poststructuralist research, see, Lather & Smithies, 1997). In other cases, poststructuralists who engage in qualitative research adopt what anthropologists refer to as an etic or outsider stance and read the emic or insider interpretations of research participants through a number of different theoretical lenses developed in other contexts (Jackson & Mazzei, 2011). This tactic, of course, intentionally introduces even more layers of complexity into the already complex cultural analysis process described by Geertz.

The Antithetical Orientation of Other Academics

Interestingly, qualitative researchers’ enthusiastic embrace of complexity is atypical in the academic world. Philosopher Stephen Toulmin (1972), for example, noted long ago that those who work in most academic disciplines not only are oriented toward developing theory that, by definition, simplifies reality. They also generally do their work within simplified, ideal typical conceptions of the social world that are constructed, over time, by members of the disciplines, themselves. These ideal typical conceptions of the social world identify a limited (i.e., a manageable) number of variables that should be focused on and simplified narratives of social life that are accepted as givens by those who have been initiated into the perspectives of a particular discipline.

When there are irreconcilable disagreements within a discipline about how to simplify reality, Toulmin notes, the discipline simply subdivides: for example, linguistics divides into psycholinguistics and sociolinguistics; the University of Sidney creates two philosophy departments, one for Marxist philosophers, the other for philosophers of a more traditional stripe (Feyerabend, 1993/1988). Each subdiscipline then develops its own journals and its own criteria for success.

The point is, in much of academia, theoretical simplification is not only the end goal; some sort of a priori simplification also is required for academics to do their work. In fact, for most academics—at least those who work within the confines of most academic disciplines—working within a simplified ideal typical world constructed and sanctioned by a discipline or subdiscipline is a standard operating procedure and a key strategy for putting the discipline in disciplined inquiry (Toulmin, 1972).

This Section’s Bottom Line

In this section, I have noted that, unlike policymakers, most qualitative researchers enthusiastically embrace the complexity of the social world. I have also suggested that qualitative researchers’ embrace of complexity is atypical within the social science community as a whole. For most academics, disciplined inquiry entails (a) bounding research within predefined ideal typical conceptions of the empirical world that, in effect, mask empirical complexity and (b) making the desired endpoint of research the construction of theory that, also, intentionally simplifies reality.

In short, most academics have a vested interest in keeping complexity at bay, and, in this respect, at least, they are like members of the policy community. Given their shared preference for simplification, it should not be surprising that policymakers and many academics (especially those who embrace the use of quantitative research methods) are natural allies. At the very least, academics who have already simplified the social world’s complexity—through the a priori construction of ideal typical conceptions of the way the world works, through the theories they construct, and through the instruments they use to measure aspects of the social world—are unlikely to reject out-of-hand policymakers’ what-works question as unanswerable. This special issue, of course, is evidence that qualitative researchers think differently

The Art of the Possible

I realized early on that if politics is “the art of the possible,” this characterization applies even more so to the process of influencing what is inevitably a politicized policymaking process. After all, normally there is a rather significant power differential between policymakers and academics, especially when academics are operating on policymakers’ turf. Because of this, policymakers are likely to be in a position to dictate the parameters and framing of any discussion that occurs between them and members of the research community.

Earlier in my career, for example, I got a call from a member of the governor’s staff asking me about my assessment of someone being considered to head a major state initiative that had been proposed by the governor and that was about to be approved (and funded) by the state legislature. The staffer also asked me to recommend others who might be better suited to do the job. My immediate response was to criticize aspects of the governor’s plan. I very quickly picked up signals, however, that if I wanted to remain in the conversation—and if I hoped to influence the governor’s appointment decision—I had better stop playing the critic role.

I should quickly add that the problem was not that the governor’s staff could not take criticism; the problem was that my timing was off. My criticisms undoubtedly would have been welcomed and given due consideration had I voiced them while the governor’s proposal was being formulated. The staffer’s call came, however, just as the state legislature was about to legislate, too late in the process, in other words, to modify the governor’s proposal in any significant way (and, possibly, even in minor ways) without risking defeat of the entire proposal in the legislature.

So, I had a choice: I could stand on principle and continue to “tilt at windmills.” If I did this, I was certain that the call would end quickly and I would not be called for advice again, at least not about the specific matter being discussed in this particular phone conversation. Or I could stop playing the critic and keep the conversation within the bounds the staffer had defined at the start of our telephone conversation. I chose the second option and simply answered the questions the governor’s aide had asked. As things turned out, the person who the aide had called about did not get the nod, and the person I recommended as a better alternative ended up becoming the governor’s appointee.

Interestingly, a colleague of mine—a colleague, in fact, who actually knew considerably more than I did about the substance of the governor’s initiative and about people who could have done the job the aide was asking about—also was called for advice and also began his conversation by pointing out problems with the governor’s proposal. Unlike me, however, my colleague either did not pick up the aide’s signals or he chose to ignore them and proceed with his critique. The aide later told me that he and the governor simply ignored my colleague’s advice because my colleague, clearly, was not “on the governor’s team.” The aide also suggested that my colleague would not be contacted for advice by the governor’s office in the future.

Qualitative Researchers as Translators

So, part of practicing the art of the possible was realizing that anyone who wants to influence policymakers must work within policymakers’ frames of reference and to use policymaker’s language. Consequently, early on, I began to see myself as a kind of translator between the culture of qualitative and policymakers’ culture. I also had to come to grips with the fact that that a good deal of understanding inevitably would be lost in the translation efforts in which I would be engaged.

I knew, for example, that I, like many other qualitative researchers, might think that symbolic interactionist sociologists (Blumer, 1969) and practitioners of ethnomethodology also in the discipline of sociology (Garfinkel, 1967, 2002) were on to something when they argued that social action is not about cause and effect relationships but, rather, involved human beings constantly constructing and reconstructing meaning though social interaction. Blumer, for example, wrote, “A knowledge of this process [of constructing and reconstructing meaning] would be of far greater value for prediction, if that is one’s interest, than would any amount of knowledge of tendencies or attitudes [that could be viewed as causes that produce predictable effects]” (p. 98).

I also understood, however, that most policymakers would not have the patience to try to understand the distinctions Blumer and Garfinkel had articulated and would probably not know what to do with this knowledge even if they did understand it. So, despite what the qualitative research gurus, Lincoln and Guba (1985), had said about causal talk being inconsistent with a constructivist paradigm, I learned to talk in cause and effect terms when I am talking with policymakers. In fact, I came to regard the concept of causality as a functional fiction in the policy arena and reminded myself that, once one assumes that knowledge of the social world is always filtered through human constructions, we have no definitive way of deciphering fact from fiction, anyway (Donmoyer, 2011). The best we can do is select our conceptions of reality on utilitarian grounds, and, if a researcher wants to provide information that is useful to policymakers, he or she had better eschew using the language of symbolic interactionism or ethnomethodology and speak in cause and effect terms

The Benefits of Talking “Truth” to Power (in a Language Power Understands)

I quickly discovered the benefits of talking in a way that policymakers understood. I, like most qualitative researchers, am in the business of telling stories, and most policymakers are often persuaded by stories. I learned this lesson while still a doctoral student doing a modest little study that turned into one of my first research publications (Donmoyer, 1982). The study focused ex post facto on the passage of a what turned out to be a seminal piece of legislation that mandated that high school students must pass a state competency test to receive a diploma The legislation eventually was copied by many other states and, in fact, represents policymakers’ initial foray into what is now a common practice: mandating high-stakes testing in schools.

So I could add a document analysis component to what was primarily an ex post facto interview study, the state legislator who had introduced the legislation graciously gave me access to his files related to the legislation. What I found there was a letter from a parent that told the story of the parent’s special-needs son. The legislator later confirmed during an interview that this story was the only thing he needed to modify his legislation to allow school districts to exempt special-needs students from the competency testing program mandated by the legislator’s bill.

The power of a good story has been reinforced time and time again in my interactions with policymakers. Policymakers may value statistical data and analysis in their public discourse; in private, however, if you want to convince them of something, tell them a good story!

To be sure, the stories that normally are convincing are more like medieval morality plays than the layered and complex accounts championed by Geertz (1973/2000) and other advocates of thick description (e.g., Lincoln & Guba, 1985). And they certainly look nothing like the so-called messy texts (Tierney, 1999) that poststructuralist and postmodernist qualitative researchers write. When I become frustrated by the need to distill a complex storyline into a succinct and somewhat didactic anecdote, however, I remind myself that politics—and especially influencing politics—is, indeed, the art of the possible.

Managing the Costs of “Talking Truth to Power” (in a Language Powerful People Understand)

Clearly, there are costs—including some substantial psychic costs—associated with being a perpetual “translator.” I, for example, have used a variety of metaphors to characterize (until the writing of this article, only for myself) the sense of discontinuity I experience as I move back and forth between the qualitative research community and the community of policymakers. Schizophrenia was one such metaphor; I also, at times, found myself employing Aquinas’ distinction between the world of God and the world of man in my feeble attempt to make sense of what I was doing. (For the record, I always associated the culture of qualitative research with Aquinas’ world of God.)

To ensure that I do not go completely “native” when operating as a qualitative researcher in policymakers’ culture, I continue to spend considerable time writing for academic audiences and doing the sort of qualitative research that academics value. For example, not long before I was sent the page proofs for this article, another article I authored, which reported the rather complex and layered findings from a qualitative study my colleagues and I had conducted, was published in a another refereed journal (Donmoyer, Yennie-Donmoyer, & Galloway, 2012).

I also continue to read and attend conference presentations, and the books, articles, and conference presentations I attend to are by an array of very different types of academic authors. To be sure, I most certainly read and listen to qualitative and quantitative researchers, who, like me, are attempting to communicate to members of the policy community. But I also attend to the work of qualitative researchers who are blazing new methodological trails in the research world, the sort of qualitative researchers who have contributed to this special issue.

The latter sort of work has been especially helpful for me. Among other things, it reminds me not to go too native in the policymaking world and not to completely lose my methodological soul as I play the translator role. Beyond these general benefits associated with hanging out on the cutting edge of the qualitative research field on occasion, there are also at least three more specific benefits.

First, ideas I hear and read about when listening to or reading cutting-edge methodological articles and the substantive research “reports” that cutting-edge methods produce reinforce, sometimes quite dramatically, the idea that what we assume to be reality is only a cultural artifact. There are many other pathways to this insight, of course, but it always is helpful to be reminded in new and different ways that we inevitably inhabit an illusion. Such reminders are especially helpful when one is working in the policy arena where conceptions of reality—and the concept of reality, itself—are taken for granted.

The second benefit is a kind of corollary of the first: Many of the cutting-edge discussions of qualitative research I hear and read about remind me that reality is not only a cultural construction; they also are a reminder that cultural constructions invariably serve the interests of some groups and further disenfranchise others. This is actually a rather easy idea to translate into a language that policymakers can understand, though an adequate translation normally requires eschewing the use of abstract and highly theoretical talk and, instead, employing concrete examples about how certain terms (e.g., cultural disadvantaged students) and ways of framing issues (e.g., the decision not to include an assessment of bilingual ability in standardized tests of student achievement) privilege some and disadvantage others even before any empirical research begins.

Third, the sessions and articles about less-than- traditional thinking about qualitative research remind me that, even in academia, culture casts its spell and constrains as well as frames perception. One vehicle for doing this is language. Whenever I read texts that use a substantial number of words containing parentheses and/or slashes (e.g., (dis)empower; dis/empower), that treat trouble as a verb, and/or that use ideas from relatively abstract theory (especially theory produced by French scholars) the way clergymen and clergywomen use the Bible as a source of texts for their sermons, I know I am reading the language of postmodernism and that what I am reading will portray the process of research and other aspects of living in the world in rather predictable ways.

There is nothing wrong with this, of course. To the contrary, manifestations of unique cultures in the academy remind me to be even more tolerant when I am dealing with policymakers’ culture. To be sure, in most respects, the cultures of cutting-edge qualitative researchers, on the one hand, and conventional policymakers, on the other, could not be more different. But in one important respect they are the same: They each have socially sanctioned ways of talking and perceiving, and if one hopes to understand the culturally sanctioned ways of talking and doing things in either culture, one needs to operate the way most anthropologists, historically, at least, attempted to do their work.