The Quality of Planning Scholarship and Doctoral Education

In observance of the thirtieth anniversary of the Journal of Planning Education and Research (JPER), I had the opportunity to participate with several former JPER editors and a frequent JPER contributor on a panel, A Retrospective on Planning Scholarship, at the 2011 ACSP Annual Conference in Salt Lake City. My comments at the panel began with reference to the main conclusions in Goldstein and Carmin (2006), which assessed planning scholarship based on articles published in the Journal of the American Planning Association (JAPA) over the 1963-2003 period.

We had argued in that article, using Stephen Toulmin’s (1972) categories, that planning was a diffuse discipline or field. This meant that there was still wide diversity in disciplinary goals, theoretical concepts, methods, and standards for judging scholarly quality and validity. There are both benefits and costs of being a diffuse discipline. The benefits are that as members of the community of scholars, we are more open to new ideas, new concepts, and new approaches to inquiry compared to many other disciplines, and hence more adaptive to change and more responsive to the changing external demand and needs for knowledge and expertise. There are downsides of being a diffuse discipline, however. These include academic planning programs often being in weaker positions within universities competing for resources and prestige, being considered (and treated as) a “minor” profession (Glazer 1974), having a chronic identity problem, and perhaps having a slower rate of disciplinary progress.

At least three reasons come to mind for why planning is a diffuse discipline. The first is that planning is at the same time a professional and an academic field, whose bifurcation leads to divergent goals. Second, planning has a long and deeply imbedded normative orientation, which requires modes of argumentation and standards for judging good and bad, in addition to judging the truthfulness of statements. Third, in order to “see thing whole,” much of our conceptual repertoire has been borrowed and adopted from a variety of other disciplines and fields, which introduces heterodoxy in approaches to scholarship.

Using refereed articles in planning and planning-related journals as the empirical base, there is little question that the quality of planning scholarship, on average, has improved since JPER began in 1981. In our 2006 article, we noted that the published articles in JAPA over time have become more scientific in terms of the intended aims of the articles. But if we only look at articles that get published in refereed journals, we get a biased picture of the state of planning scholarship. The improvement over time in the quality of planning scholarship may not be so impressive if we examine all the manuscripts that are submitted to planning journals and all the papers that are presented at ACSP conferences.

My impression, as a peer reviewer of planning journals, as an attendee of many ACSP conferences, and as a member of the audience and faculty search committees for a large number of job talks, is that a disconcertingly large number of manuscripts and presentations have major methodological flaws. Nor is this just my own idiosyncratic judgment, because a number of colleagues from other universities whose opinions I respect have commented to me about their own experiences as manuscript reviewers and listeners to job talks. To be sure, these problems are by no means the norm in planning scholarship. But unfortunately, they are frequent enough that as planning educators we should be concerned about the quality of training we provide our students in planning doctoral programs.

The concern I have about the research methods training our doctoral students receive is neither solitary nor new. Almost twenty years ago, in the Report of the Commission on the Doctorate in Planning to the Association of Collegiate Schools of Planning (Innes 1993), doctoral students and graduates reported that they feel “they are not well enough prepared in the research methods that they need [in order] to answer convincingly the research questions they ask.” It was commonly agreed by respondents to a survey of doctoral students and PhD program faculty that “learning to do independent research is the most important aspect of the Ph.D. programs [but also] the most difficult educational challenge” (Innes 1993, 169). More specifically, skills in economic analysis and statistics were cited the most as most important to postdoctoral success, with political analysis, project and program evaluation, and field work methods also mentioned (Report of the Commission on the Doctorate in Planning 1992, 28). The members of the commission went on to recommend as one of their key points,

Programs must give high priority to maintaining and improving the scholarly quality, rigor, and contribution of doctoral research. Students must have preparation in both quantitative/analytic and qualitative/field research methods as well as training in identifying research questions and designing research. (Innes 1993, 169)

So we might ask whether our doctoral programs are currently delivering the courses and other modes of training that should help to ameliorate the inadequacies in methods preparation identified in the Report of the Commission. If the answer is no, then why not? And if yes, what explains the “disconnect” between the training provided and subsequent performance as scholars? In either case, as planning educators, how should we address the problem?

As a starting point, I conducted an informal sample survey of ten doctoral programs in planning to see (1) what the current research methods requirements were in terms of coursework and (2) how doctoral students are expected to demonstrate competency in research methods on comprehensive, qualifying, or preliminary exams. Although the ten programs were selected to be roughly representative in terms of age and size of doctoral programs, the sample was not randomly drawn and hence is not intended to make inferences for the full population of doctoral programs. Its purpose was suggestive. A more complete and systematic survey might indeed be warranted as a next step, and should include how doctoral methods requirements might have changed since 1993. Comparisons with the doctoral methods requirements in other diffuse fields such as public policy, social work, or education might also yield valuable insights. For this Commentary piece, however, the more limited information about current requirements from the smaller convenience sample should suffice for suggesting whether the issues raised in the Report of the Commission have been addressed and for initiating discussion about how we might improve the methodological training of our doctoral students.

The results of the informal survey were illuminating. Of the ten programs surveyed, three require doctoral students to take quantitative methods courses, for example statistics, above the level of statistics taught in master’s-degree programs. Two of the ten require a course in qualitative methods. The other five programs require from one to three research methods courses, selected (with their adviser’s approval) from a menu of quantitative and qualitative methods, depending on interests and prospective dissertation research area. Of these five programs, two require one methods course, two require two courses, and one program requires three.

All ten programs require coursework in research design. Seven of the programs require a full-length course devoted to research design while the other three incorporate some research design into doctoral seminars that include other topics and activities. In addition, one of the programs has a required course on “The Logic of Inquiry” (a methodology course in the social sciences with a capital M) in addition to research design.

By and large, comprehensive (or qualifying or preliminary) exams do not cover formal research methods. Methods may be covered in terms of the student’s proposed use or application of research methods in his or her area of specialization or dissertation research. Many program directors commented that having a separate exam on research methods would be redundant with the final exams taken in their specific methods courses.

That a minority of programs surveyed require minimal coursework in methods courses, that only two of the ten programs surveyed require courses in qualitative methods, and very few programs now seem to require students to demonstrate competency in research methods on doctoral exams prior to undertaking the dissertation research, should all raise “red flags.”

The trend seems to be allowing students to choose the particular research methods courses they and their advisers feel they need, given their prospective dissertation research interests. In principle this policy sounds reasonable, but I fear it is leading to unanticipated problems. In following this policy, there is an assumption that students and their advisers know what types of methods they will need, not only for their dissertation research, but for their careers. We know that many academics change their research interests after completing their PhDs, over the course of their careers. These changes often require competence in particular research methods for which doctoral students were not trained. Yet the lack of previous training does not usually deter us; we simply “learn” informally or “by doing.” But I argue that without sound training and experience in the fundamentals while in doctoral programs, we end up not being able to select, utilize, and apply such methods in sound, critical, and rigorous ways.

The proliferation of a research design course in doctoral programs is a welcome development, but such a course should not be regarded as a de facto substitute for research methods courses. They are instead complements. Research design covers the logic of selecting among alternative approaches and strategies for answering a particular research question, but do not cover the necessary details for how to use and apply a particular set of techniques or approach. There is simply insufficient time in a one semester course. So, for example, one can discuss the strengths and weaknesses of using quasi-experimental designs, but a student will not be able to appreciate the implications of not being able to control for certain putative causal factors with a given design unless that student understands misspecification of multiple regression models. Similarly, students can learn the strengths and weaknesses of using case study and comparative case study designs (or other field methods) but they will not learn how to actually conduct them without getting the experience of, and feedback from, conducting them.

There are at least two mistaken beliefs that I perceive are shared by many planning doctoral students, and which may be reinforced by their faculty advisers. The first is that rigor and relevance are antithetical to one another; to be rigorous means doing research—and almost always quantitative—in a way that is only of use to other academics. This belief leads many students to eschew training in quantitative methods beyond what is minimally required.

The second mistaken belief is that qualitative methods are “easy” to learn and apply, requiring no special training or guided experience. The fact is that conducting high-quality and insightful case studies, historical research, or institutional analyses is not at all easy. Even after having received formal appropriate coursework, and advice from a dissertation advisor, these are skills that require multiple iterations of practice with critical feedback from already accomplished qualitative researchers.

So what should we do? I make the following suggestions, not as the last word, but to provoke further discussion about how we as a community of planning educators might be able to jointly address the problems raised in the Innes report. In making these suggestions, I am fully aware that they may run contrary to a possible trend of many doctoral programs easing rather strengthening research methods requirements, perhaps under pressure to shorten the time to degree.

An alternative or complement to comprehensive exams for demonstrating competence is for doctoral programs to offer students a research practicum where they have to analyze actual empirical data in order to answer a research question. In this setting, with “coaching” from the faculty, students can put into practice and learn from that practice a variety of methodological skills techniques prior to tackling the dissertation itself. A research practicum course along these lines might work best as the second part of a two-course research design and practicum sequence. It could provide options for students between a focus on the application of quantitative and qualitative methods. As an example, one team of students might be provided with a large secondary data set and then proceed to devise different, theoretically grounded research questions or hypotheses, applying a variety of statistical analysis techniques to test the hypotheses. Or another team of students could use the practicum to design and perform content analysis, coding analysis, or network analysis of a set of planning documents, or an existing set of narratives about a specific planning episode, or a set of observed interactions among actors.

It is not likely that all doctoral planning programs, especially the smaller ones, can afford to offer all of these courses internally. Sharing the instruction of methods courses with other departments in the social sciences such as sociology, political science, and public policy, or with other professional schools but that offer PhD degrees such as public health or social work, is one way to overcome these limitations. Obviously such joint arrangements would make sense only if there is a commitment to the same content and level of rigor.

It would be incorrect to interpret my comments as a call to devalue the acquisition of deep and broad substantive knowledge as a focus in our doctoral programs. I would be no more impressed with technically accomplished but substantively vacuous and practically clueless faculty candidates giving job talks than the opposite. Is it asking too much for our doctoral programs to provide the training to our doctoral students that will help them be literate in the broad array of theoretical and conceptual frameworks needed to understand complex issues, practically savvy, and methodologically sophisticated and astute? We should not accept anything less.

By not having our current generation of doctoral students well trained in research methods, we run the risk of having future planning faculty inadequately equipped to train and mentor the next generation of doctoral students, with the cycle repeating. By reinvigorating and taking more seriously the training of research methods in our doctoral programs, we will help ensure that the quality of planning scholarship will continue and surpass the strides it has achieved over the last thirty years.