Abstract
Response to: David Kasdan, “Great Books” of Public Administration, 1990-2010: Revisiting Sherwood’s Survey in the Wake of Reinventing Government
I continue to be disappointed with the time and energy capable scholars spend with what I refer to as beauty contest studies of public administration, unsystematic studies focused on determining who produces the most research, which scholars are most highly reputed and how they got to be so, or in the present case, determining the most “influential” books. Yes, of course, I recognize the shards of hypocrisy littering this judgment. As much of my own professional life is devoted to research evaluation, why should I begrudge others’ attempts to divine what is important or influential? More on this later.
I must in fairness say that this article is well written and engaging, so much so that I almost find myself questioning the wisdom of my general antipathy to insider ratings of public administration scholarship. Kasdan’s style is decidedly not the “dry academic” style that many of us use to fend off the droves of students and practitioners who might otherwise have some interest in our work. But neither is Kasdan’s style the peppy journalism of those bêtes noire of jealous academics, the “most influentials,” Osborne and Gaebler (1992).
We see in Kasdan’s article an ability to ask good questions and an admirable restraint from pushing easy answers. If we public administration scholars must be so inveterately self-reflective, then at least we should do it well. The author gives us many useful questions to deliberate, including ones pertaining to the changing role of books in the knowledge base of public administration, the critical and evaluative basis of books versus journal articles, and the meanings of “influence” and the relation of those meanings to quality. These points merit discussion. However, if the questions are good ones, the answers are not of much help. Critical flaws with the article reduce its ability to push the discussion very far. I am going to briefly enumerate these problems and then suggest a possible alternative to the sort of “beauty contest” style of scholarship so common in public administration.
First, I set a boundary condition. I provide no comments about the particular books identified by Kasdan’s respondents. In my judgment, this list includes some fine books (at least the ones I have read seem to me to be good ones). Like some of Kasdan’s respondents, I do not wish to participate in comparing books that seem to me incomparable and doing so according to ambiguous criteria.
A first point goes to the wellspring of beauty contests. In my judgment, Frank Sherwood’s article was not in need of replication but rather improvement. Kasdan’s article is neither a replication nor an improvement. As Kasdan notes, he does not yet have the apparent advantage of Sherwood’s extensive intellectual-social capital and, thus, draws from even fewer respondents and has less success in obtaining responses. But the broader problem is thinking that posing questions to award winners, journal editors, and editorial board members constitutes a suitable research method. Even if one finds such a method palatable, what do we make of these findings from only 18 respondents, 3 of whom provide no nominations and, indeed, resist providing nominations?
A second problem is confusion over what the respondents can tells us. If the author wants to know about budding bureaucrats, then asking professors to judge impacts on public administration graduate students is misguided. First, why not let students and former students speak for themselves? Survey them, not professors. Second, if one is really interested in readings that influence bureaucrats, it is well worth remembering that the vast majority of bureaucrats have never been enrolled in a public administration program. It would perhaps be interesting to examine differences in intellectual influences between public administration graduates and other bureaucrats. Note “intellectual influences,” not just books.
If one is interested in academic public administration, then it seems irrefutable that journals are the place to be. That is where the research is published, including, especially, the type the author refers to as scientism. I suspect Professor Kasdan’s use of this pejorative term is not intentionally pejorative. Scientism is a term that has long been used to denigrate scientific work that the critic feels is either obsessed with a false sense of objectivity or is rigidly unrealistic in its assumptions and procedures (Goldfarb, 1989; Maffie, 1995). The term is also used to disparage work in which social scientists use research methods that are claimed to be exactly like those used in the natural sciences (i.e., not reflexive or social constructed; Winthrop, 1959). It seems unlikely that many empirical researchers in public administration would characterize what they are doing as scientism.
One problem with Kasdan’s article is of such great magnitude as to sweep away all other problems. After asking possible respondents “What is the most influential book for public administration published from 1990 to the present?” Kasdan goes on to say, What is most influential is ambiguous by design—You may think of books that were influential to the field or to you, personally. An alternative way of thinking about influential would be “what books would you assign for a course on contemporary PA?”
This is so misguided that I barely know where to begin. Were I not gun-shy about possible scientism accusations, I suppose I would just say that no good could come of having an ambiguous dependent variable and leave it at that.
All of these meanings of influential are worthy of inquiry. Some might well want to know which books have had greatest impact (if that is what influential means) on, respectively, the field of public administration, individuals, and teaching—but not all at the same time. True, we could have perhaps learned something useful if Kasdan had even asked the respondent which meaning of influential he or she was using. But no such luck. This is what I think we can safely infer: A small, unrepresentative group of public administration scholars were asked about influential books in public administration; a minority of those asked actually identified some influential books (including at least one expressly disliked), but most of the respondents failed to indicate why the nominated books are influential or even what they meant by influential. That is a long sentence. Here is a shorter one. Ambiguity hinders scholarly work.
Were I providing a review essay on the topic “Flaws in Public Administration ‘Beauty Contest’ Research,” I have little doubt that I could just as easily hack away at other studies. Kasdan’s approach is no more egregious than others, and as I mentioned above, he is obviously very thoughtful and has a mind and a writing style that engages even when the “data” from his study have quite limited value. I think there are some interesting, important research issues to be uncovered in internal, reflexive studies of public administration, but the beauty contest approaches get at none of these. So, rather than railing further against the general approach this study represents, let me suggest a possible remedy.
My possible remedy to beauty contest scholarship requires a brief foray into the “other” professional interest I alluded to above. A significant portion of my work has nothing to do with public administration but rather pertains to research evaluation, including “scientometrics.” Scientometrics is the infelicitous term used for scholarship assessing knowledge products (articles, patents, licenses, algorithms) of engineers and scientists. In all likelihood, many readers of Administration & Society have not heard of this field, or if they have, it is probably peripheral to their research interests. However, depending on the way one marks boundaries, both the boundaries of fields and geographic ones, scientometrics is arguably a larger field than academic public administration. It has its own institutional life in such organizations as the International Society for Scientometrics and Informatics and professional journals such as Scientometrics 1 and Research Evaluation.
Among other research activities, scientometricians develop and apply bibliometric techniques. These are the people who gave us citation analysis, Hirsch scores, cocitation networks, and, my personal favorite, disambiguation techniques. 2 My own esoteric specialty in this generally esoteric realm is techniques for compiling quantitative data and predictive models from scientists’ curricula vita. 3
As a result of the work of scientometricians and research evaluators, we know a great deal about science outputs and impacts, including such factors as, for example, the differences in productivity rates during the course of a scientific career, trends in nanotechnology collaborations, and the relative rates of publication growth in the United States and China. Scientometricians have also given us a good deal of knowledge that is perhaps less than earth shattering. 4 The basic point is that we scientometricians take this stuff seriously.
Scientometric studies in public administration are quite uncommon. There are obvious reasons why. First, public administration is a field not on the radar screen of the National Science Foundation and National Institutes of Health program managers who fund most scientometrics studies. Many scientometric studies are quite costly because they often require special software and access to databases that are not public domain. Another reason why they are not common in public administration is that not many PA scholars have been trained in these techniques (I think I know most of them, and they number perhaps 10 or so). However, scientometric studies of public administration obviously are possible and, I am pleased to say, can even be welcomed in leading journals. True, proof of that last statement is quite limited. The only study I know of that uses data from public administration and applies relatively sophisticated bibliometric techniques is an article by Corley and Sabharwal (2010). Their article is decidedly not part of the beauty contest genre. Instead, their concerns are much the same as research evaluators have brought to the study fields in science and engineering. Most important, their work is part and parcel of a body of explanatory theory. Corley and Sabharwal seek to understand the social dynamics of research and impacts on productivity in general (i.e., not just the “giants of the field”). Their chief finding is an interesting one because it shows that public administration researchers swim against a strong tide. In public administration, productivity is associated with sole authorship, whereas in almost every field of the natural and social sciences, collaboration enhances productivity (Bozeman & Corley, 2004; Fox & Mohapatra, 2007) or is at least neutral in its effects (Duque et al., 2005).
What I like about the scientometrics-based studies is that they have some long-range utility as well as the ability to link up with explanatory theory. In my view, we do need to recognize scholarly leaders, researchers, book writers, mentors, and teachers, and that is why we have awards. Let us continue to give Waldo and Simon and Levine and other awards, but perhaps, we should impose a moratorium on using the award winners as research participants. If we move from the beauty contest approach, we can do some interesting and perhaps even important research on the intellectual state of the field.
Drawing loosely from the research evaluation literature, here is a quick list of topics that focus on intellectual trends in public administration but, at the same time, seem to have some theoretical weight. In the interest of space and the reader’s patience, I provide only three examples but could easily identify many more.
To what extent and in what ways does public administration research affect other fields and disciplines? True, we might not be delighted by the answer to this question, but if we can identify at least some pathways by which public administration research reaches other fields, then perhaps we can reinforce and expand these patterns of use. “Field penetration” studies are not common in research evaluation, but there are some studies to use as models (e.g., Ennis, 1992).
How do nations’ public administration research content and productivity compare? Most experienced public administration researchers know that some nations publish more public administration work than others, and they have a vague idea that there are important differences in style and content. But we could use scientometrics and research evaluation to go beyond intuition and vague ideas, and actually track changes in amount of production (e.g., Hicks, 2004) or thematic content (Franzoni, Simpkins, Baoli, & Ram, 2010). This type of analysis is not especially aimed at comparing numbers but rather at looking at workflows and international comparisons. In the case of public administration research, it is my intuition that international collaboration has increased greatly during the past decade or so. But is that intuition correct? If so, what have been the effects of increased international collaboration (e.g., Huang, Tang, & Chen, 2011)? Has increased collaboration affected research problem choice and research content?
What are the effects of external events on public administration research careers? A good deal of research in sociology of science and research evaluation seeks to understand researchers’ “life course,” including the effects of major events on scientific careers. For example, a colleague and I (Bozeman & Gaughan, 2007) used curricula vitae data and bibliometric data to chart the effects of research grants and contracts on scientists’ scholarly activity. Given the dearth of grants provided to public administration researchers, perhaps this is not the most fruitful direction for life course research in public administration. But perhaps, it would be useful to know about the research career and productivity effects of, for example, having a child, a government experience, or a career hiatus. Standard bibliometric and research evaluation techniques can shed light on such issues, and some of the people doing such work, but applying it to other fields, have been trained in public administration (e.g., Boardman & Ponomariov, 2007; Su, 2011).
Almost all the research evaluation issues that have been examined in science and engineering can be examined in public administration if there is interest, will, and training. If we public administration scholars decide to take this step, the results may prove more useful than the occasional unsystematic studies we now produce about research leaders and knowledge product ratings. Maybe I will even undertake to do some of such a study myself. If I do, I will certainly send it to Professor Kasdan for review. He may or may not view the result as scientism but, regardless, “turnabout is fair play.” I am sure he will have something interesting to say.