Abstract
This paper proposes a model for assessing learning outcomes specific to planning studio courses. We begin by reviewing literature on learning outcomes found in education theory, and summarize a generalized model for outcomes assessment. Next, we review the planning literature on learning outcomes, then present a snapshot of contemporary learning outcomes assessment in planning studio courses as informed by content analyses of syllabi and interviews with studio instructors. Finally, we propose a studio-specific assessment model, and conclude with some recommendations for accreditation guidelines for learning outcomes assessment.
Introduction
The assessment of learning outcomes is a hot topic in the increasingly performance-oriented context of higher education, and accreditation mandates for professional degrees have mirrored this shift, placing increasing demands on their constituent academic programs to measure, analyze and, ideally, improve the educational experiences of their students. The planning profession is no exception. In the United States, the Planning Accreditation Board (PAB) recently revised its accreditation criteria to emphasize and require reporting on learning outcomes. In turn, many North American professional planning programs have responded by benchmarking and assessing learning outcomes of their programs as a whole, with most attention to the core curriculum. Admittedly, our understanding of learning outcomes, both as an academy and a profession, is in its nascent stages. The PAB Accreditation Document (2006) requires that programs report on learning outcomes in their self-study reports, yet it provides limited guidance on how to do this. Its own Outcomes Assessment Manual (2008), which addresses broader program goals centered on research productivity and community outreach, helps guide programs in repackaging existing feedback mechanisms (course evaluations, exit interviews, AICP pass rates) within the rhetoric of outcomes assessment, rather than leading a collective dialogue about what it means to deliver an effective planning education informed by an outcomes perspective.
Nevertheless, this dialogue has begun. The 2011 ACSP Administrators’ Conference included a panel on the challenges of outcomes assessment, and recently proposed changes to the Accreditation Document (to take effect in 2013) require that much more attention be paid to measuring program quality. Specifically, the proposed changes require that program quality be measured both by output measures (e.g., program degree productivity, faculty/scholarly contributions, and student learning) and outcomes measures (e.g., graduate satisfaction, employment, and accomplishments) (PAB 2012). A useful way to think about these distinctions is that output measures deal with the “inner loop” of feedback, assessing primarily student learning and academic achievement, while outcomes measures deal with an “outer loop” of assessment after graduation, once students have left the program.
Thinking through how we might educate planners from an outcomes perspective is a valuable exercise. Thus, our research was conducted to both contribute to the discussion on PAB requirements and to advance the wider discourse on learning outcomes assessment in planning. To inform this dialogue, we examine one small, but important, slice of planning curricula: required studio courses in U.S.-based planning programs. Participation in some form of studio-based learning experience is a requirement of most PAB-accredited planning programs, which suggests that studio courses offer a highly valued educational experience. As full-time ladder faculty who teach required studio courses and as program directors that have recently navigated the reaccreditation process, we have both confronted the challenge of how to evaluate our own and others’ studio courses within a learning outcomes framework. We have also reached out to our colleagues who teach planning studio courses—notably through a series of sessions on studio pedagogy at ACSP 2009 and 2010—and these conversations led in part to the research reported here.
Our findings suggest that studio courses are an informative lens through which to view learning outcomes assessment. Most studios primarily seek to “teach practice.” In studios, there is the potential to explore all manner of knowledge and skills, and there is also space to accommodate theoretical and ethical considerations, although these latter topics are less frequently identified as explicit goals. The capacious nature of studio would appear to offer great range in assessing learning outcomes, but in fact, we found that teaching objectives were only formally expressed in about 60 percent of syllabi examined. When adjunct faculty delivered the course, this percentage fell to 50.
Planning schools all define studio courses differently and there is little writing on the subject to serve as a primer for new studio instructors or for students new to studio-based learning. There are also varying degrees of program oversight for studio courses. Depending on whether the studio is required, and whether the course is taught by full-time versus part-time faculty, the degree to which these courses are integrated into the overall program goals varies widely.
For better or worse, then, there is no common understanding of studio as a distinct pedagogical practice in planning education, little in the way of common language used to discuss or describe studio courses, and few shared experiences to aid in shaping teaching goals or orienting our students to this unique pedagogy. It is against this backdrop that we sketch the contours of a framework for thinking about learning outcomes in studio courses as a means of both informing our understanding of studio pedagogy and the ongoing discourse on outcomes assessment.
This article makes three contributions that aim to inform and improve the student learning experience in studio courses. First, we lay out a general learning outcomes assessment model drawn from a survey of education theories. Second, we discuss the treatment of learning outcomes in studio courses in the planning literature and, in an effort to understand what and how learning objectives were expressed, also present the findings of a content analysis of forty-four syllabi for required studio courses in U.S.-based planning programs during the academic years 2009-2011. When not explicitly expressed in the text of a course syllabus, we revealed pedagogical intent by examining syllabi elements such as problem identification, frame and scope of inquiry, role of the client, plan-making process, and expected deliverables. We conducted follow-up interviews with eleven studio instructors, asking more specific questions about their learning objectives, outcomes, and performance indicators (see Appendix A for questionnaire). Third, we use this information to adapt the general assessment model to a studio-specific assessment model and point to important gaps between the two. Our concluding discussion summarizes the challenges associated with outcomes assessment in this environment and provides recommendations both for faculty members teaching studio courses and for the PAB.
A General Model for Learning Outcomes from Education Theories
Learning outcomes are observable, demonstrable, and measurable changes in abilities or achievements experienced by an individual student as a result of a learning experience (Allan 1996; Melton 1996; Spady 1988). Learning outcomes assessment has its origin in outcomes based education, a model of pedagogical structure that organizes curricula around explicit and detailed student outcome statements (Nusche 2008). One of the most fervent critiques of outcomes-based education is not about the use of outcomes themselves but about the fact that many desired outcomes—especially in relation to values, morals, and ethics—cannot be verified empirically through traditional assessment methods such as standardized testing.
Still, learning outcomes assessment goes some way in providing both accountability and guidance on teaching quality and student learning. In terms of accountability, higher education accrediting agencies require assessment above and beyond test scores and graduate placement, and state governments in the United States often directly evaluate the quality of instruction at public institutions (Kuh and Ewell 2010). When used in a curricular setting, learning outcomes assessments are intended to gather appropriate evidence, illuminate common patterns across student cohorts, and provide clearer guidance on specific strengths and weaknesses of individual courses. Benchmarking assessment results against established standards allows both institutions and programs to identify best practices and determine how they stack up against peers (Kuh and Ewell 2010).
Adoption of outcomes assessment policy is widespread: currently, about three-quarters of all higher education institutions (HEIs) in the United States use learning outcomes at the undergraduate level, and more than 80 percent of universities—including our own—have dedicated entire administrative units to student assessment alone (Hart Research Associates 2009). Four significant trends help explain the growth in HEI adoption of learning outcomes assessment:
Demands for quality information on student learning outcomes have increased from prospective students (and parents of students) as well as employers seeking qualified graduates (Nusche 2008). Taxpayers concerned with public funding and policy makers allocating scarce resources to HEIs want to know how well these institutions and programs fulfill their educational mission.
Stakeholders are also increasingly savvy consumers of educational rankings, and many desire to move beyond existing national ranking systems that focus predominantly on resource inputs and research outputs but provide no measurement of the degree to which institutions develop the knowledge and skills of their students (Nusche 2008, 5). Since these popular ranking systems—especially those published by US News and World Report—are often used as proxies for educational quality, leadership at traditionally underresourced institutions have begun to adopt and publicize positive learning outcomes assessments in the hopes of evening the playing field with their heavily endowed counterparts. The recent economic downturn, which has served to increase HEI enrollments while reducing public and private support, means that administrators must become better at assessing and using learning outcomes data to inform resource allocation and communicate how well their institutions are doing (Kuh and Ewell 2010).
Reliance on standardized testing methods or graduate placement data to assess undergraduate or graduate learning experiences has waned because of perceived selection biases and other contextual limitations (Nusche 2008).
The growing use of accreditation and other formal, external review processes as “quality assurance” approaches means that more HEIs are required to assess student learning regularly (Kuh and Ewell 2010). A recent survey of U.S. higher educational leaders revealed that accreditation represents the strongest driver for the establishment of an outcomes assessment program. A close second, however, is an internal commitment to improve and revise program-level learning goals (Kuh and Ikenberry 2009).
Learning outcomes assessment consists of three major components: (1) the learning outcomes themselves, (2) the outcomes assessment approach, and (3) and the outcomes evaluation design. In Table 1, we present a summary model of these three components, aggregated for heuristic purposes only, since it treats learning components as independent, while they are interdependent and overlapping.
A General Model for Assessing Learning Outcomes
The first section, learning outcomes, takes account of both cognitive and noncognitive learning, a distinction commonly made in the education literature. At its core, cognitive learning focuses on the development of skills and knowledge at the general or domain-specific level. Skills development evokes a procedural orientation while knowledge production emphasizes substantive learning. On the other hand, noncognitive learning refers to the development of certain values, beliefs and attitudes. This focus on analytical abilities emphasizes psychosocial and relational developments, social responsibility, and interpersonal and intercultural understandings (Volkwein 2003). Noncognitive outcomes can require extended personal reflection, and are thus less demonstrable through traditional testing methods.
For example, an introductory planning studio might have several intended learning outcomes that could include visual representation (cognitive, skills, procedural), zoning standards (cognitive, knowledge, substantive), and norms of peer interaction and social engagement (noncognitive, values, analytical).
The second component, the outcomes assessment approach, includes direct standards that are generally observable, measurable and absolute. Assessments of quality are criterion-referenced, meaning student learning is measured against preestablished criteria representing a minimum standard of desired performance. As such, cognitive outcomes can be assessed at a single point in time, as all students are expected to achieve a common level of knowledge or skills at the conclusion of a course or program. Conversely, noncognitive outcomes require indirect, before-and-after assessment styles relying on personal reflections of growth. Many note the inherent difficulty in evaluating such relative proficiencies, as there is little evidence that noncognitive qualities can be demonstrated or measured through directly observable behaviors (Nusche 2008). Such assessments are most often norm-referenced, meaning student progress is evaluated only in comparison to other students.
Using the same introductory planning studio example, the instructor might assess the visual representation outcome through a directly observable task, like asking students to render graphically a list of proposed zoning code changes such as new building height and setback requirements. Students would be assessed cross-sectionally (i.e., at a certain point of the semester), with skills evaluated using standardized criteria. The same instructor might use a before-and-after approach to assess the noncognitive “peer interaction” outcome. Instructors might measure this outcome by first observing the student’s interpersonal communication qualities at the beginning and end of term, with the assumption that some improvement should occur. Alternatively, a student might assess her or his own ability to navigate group dynamics before and after an intense group exercise.
The third component of the model refers to the outcomes evaluation design. Evaluation design differs from the previous component in that it moves beyond setting a framework or style of assessment to delineating the specific instruments and techniques for testing proficiency or competence on an individual outcome. Cognitive outcomes are usually subject to objective testing by a nonstudent using metric-based scoring methods such as rubrics and other standardized scoring guides. Testing formats include exams, projects, assignments, group work, and presentations. In contrast, the student herself normally evaluates noncognitive learning outcomes with open-ended statements of progress, since changes in values and attitudes are based on individual perceptions requiring subjective, personal assessment. Evaluation instruments in the noncognitive realm most often include surveys, questionnaires, and reflective journals.
Following the introductory planning studio example once again, visual representation could be evaluated objectively using a rubric (listing benchmarks related to attention to detail, craft or legibility), with instruments like a student assignment or presentations most appropriate for such an evaluation. Conversely, evaluation of the peer interaction outcome might involve asking students to reflect personally on the quality of their own leadership qualities, or having other students rate their peers’ proficiency. Such evaluations might be completed by requiring students to make regular entries in a project diary or through a peer evaluation assignment at the conclusion of the semester.
Learning Outcomes in Planning Studio Courses as Informed by Planning Literature
Since studio courses are required in the vast majority of planning programs in the United States, this suggests that as a pedagogical approach, studios offer learning opportunities that are unique and valuable when compared to traditional lectures and seminars. As evidence, according to Long’s article (this issue), students are required to complete at least one studio course—or “workshop,” “practicum,” “field-based,” or “client-linked” course formats—in 84 percent (sixty-two of seventy-two reporting) of PAB-accredited graduate planning programs in 2009. In these programs, required studio courses accounted for approximately 10 percent of total program hours, on average, or the rough equivalent of one to two standard courses. In terms of intensity of studio requirements in individual programs, the range is wide: more than thirty-eight programs require more than 10 percent of program hours spent in studio-type courses, seventeen require 15 percent or more, seven require 20 percent or more, and four programs require 25 percent or more of total hours in required studios. Only 16 percent of planning programs require no studio courses at all. Among top-ranked programs, the degree of confidence in studio-based learning is even higher: twenty-two of Planetizen’s “top 25 programs” require that students take at least one studio course to graduate (Planetizen 2009). See Appendix B for this list of programs.
The rhetoric of learning outcomes in studio courses typically involves descriptions such as teaching “synthesis,” “learning-by-doing,” and “reflection-in-action,” and aiming to expose students to the complexity of “real-world problems” and to initiate “professional socialization” (Chafee 1977; Schön 1983, 1987; Heumann and Wetmore 1984; Wetmore and Heumann 1988; Baum 1997; Forrester 1983, 1999; Long 2012 [this issue], among others). Among these objectives, “synthesis” is perhaps most commonly associated with studio pedagogy, where students apply judgment and values in selecting and applying analytical methods and plan-making procedures in the formulation and resolution of professional problems. “Learning-by-doing,” emanating from Schön’s cognitive foundation of “reflection-in-action,” is another hallmark of studio courses, where individual students and groups of students gain tacit knowledge through work on subsequent iterations “under the guidance and criticism of a master practitioner” (1985, 6). Introducing students to “real-world” and even “wicked problems” (Rittel and Weber 1973) is also an important objective of studio courses, although it is not unique in this setting: lecture and seminar courses can also effectively address these goals. Lastly, “professional socialization” is an important, albeit often implicit, goal of studio courses where students are introduced to the role of the planner in the plan-making process, as well as to a “community of practice” and its social norms. By learning to solve problems under a specific practitioner, students are exposed to the value and belief system of their instructor as expressed in his planning principles and mode of practice (Long 2012, this volume).
What is it about how studio courses are structured—including studios in design and architecture programs—that allow them to engage these unique learning opportunities? From our own reading on studios, as well as our experience teaching these courses, we see that they often begin with an open-ended problem, usually in the format of either a semester-long project or a series of shorter exercises that take account of current planning issues and afford students a choice in direction within the scope of the problem. Many studios also involve “clients” and thus raise questions of agency, constituency, power, and bias that underlay many planning problems. In some cases, the instructor “constructs” a client when she or he sets the problem. In others, actual clients are engaged, who participate in specifying the problem, provide students with access to experts and resources, and may even provide financial support for the course. Studios with actual clients offer the benefit of internship-like opportunities for students as well as access to a network of professional planners but are also often criticized for raising questions about faculty conflicts of interest and the use of student labor and intellectual property. Regardless of whether the client is real or constructed, studio problems are often physical in orientation and require deliverables that resemble professional outputs, such as land use plans and the like, but other kinds of products might also be required, such as policy analyses or community engagement strategies.
A key feature of studio pedagogy—and a key contributor to its unique learning outcomes—is the high level of interaction among students, instructors, program faculty, clients, and outside experts through informal exchanges that occur during class time and more formal exchanges centered on presentations by students at critical stages during the term. While there are some variations in the structure and focus of studio courses among planning programs, this objective of contact and interaction stems from the origins of studio pedagogy in our sister discipline of architecture. 1 But within programs, no clear pedagogical consensus exists about how studios should be administered or what their objectives should be. That this perceived indispensability to professional programs occurs alongside a dearth of standards for studio courses points to the need for a better understanding of learning outcomes associated with studio pedagogy.
Planning’s own accrediting body, the Planning Accreditation Board (PAB), is also grappling with assessing learning outcomes, with no specific guidance for the unique opportunities of studio courses. The Accreditation Document: Criteria and Procedures of the Planning Accreditation Program (PAB 2006) outlines eleven categories on which programs are evaluated, only one of which deals with curriculum. (As we stated earlier, a revised Accreditation Document to take effect in 2013 looks at both output [inner loop] and outcome [outer loop] measures but provides little guidance on how to measure student learning in general, let alone in the studio environment.) Within the curriculum component, the PAB’s basic model is to pair three categories of curriculum content—knowledge, skills, and values—with three categories of outcome measures—competence, satisfaction, and recognition. Of these measures, “competence”—which uses as evidence student papers, reports, exercises, assignments, exam scores, and presentations—is of particular interest to the case of studio courses because it is the only measure that can be readily controlled through curricular changes and course-level outcomes assessment. We can assess “satisfaction” through exit interviews, alumni surveys, and course evaluations and “recognition” through student awards, AICP test pass rates, alumni accomplishments, placements, and employment rates. Although these outcome measures are set to change along with proposed revisions to the Accreditation Document, they represent the most detailed thinking so far on outcomes assessment in planning programs.
For a learning outcomes assessment model to succeed, a more nuanced understanding of learning outcomes in studio courses is required. Given that there appears to be a consensus that studio is valued as a distinct pedagogical approach, then a more robust model derived from the educational literature combined with a deeper understanding of studio pedagogy can inform more than the action of faculty members who teach these courses; it could and should inform the PAB accreditation process.
Toward a Studio-Based Assessment Model: Learning from Syllabi Content Analyses and Studio Instructor Interviews
To develop an assessment model suited for studio courses, we turned to our many colleagues who teach studios, in addition to looking within our own studio-based programs to consider how we communicate learning outcomes to our students and to each other. We were able to identify a network of colleagues fairly quickly, particularly those leading physical planning and urban design concentrations in planning programs. We included the top twenty-five graduate planning programs in the United States according to Planetizen (2009) for their perceived curricular quality. We also considered geographic distribution, the mix of adjunct, junior and senior faculty, as well as locations in cities of different scales with different opportunities for practitioner involvement and community engagement. Most importantly, with a few exceptions, the largest planning departments with the largest number of faculty deliver these “top 25” programs. This is key to our study because larger programs are more likely to offer multiple studio sections and multiple studio concentrations, introducing variation into the sample.
We were able to make contact with chairs and program administrators in the twenty-two of twenty-five programs with required studio courses, and requested syllabi and faculty contact information for all required studios. This snowball sampling technique resulted in primary data from forty-four studio course syllabi. Full-time faculty taught thirty-two of the forty-four (73 percent) syllabi we reviewed, and twenty-six (59 percent) of these were tenured or tenure track faculty. We followed up with eleven instructors to confirm some of our findings and ask more about their teaching objectives, desired learning outcomes, assessment strategies, and performance indicators. Because of these nonrandom selection methods, quantitative analyses on the data were not performed; by extension, no claims can be made about our sample being representative of the population of studio course offerings in the United States. Nonetheless, there are insights to be offered.
Because of the unique nature of studio pedagogy—particularly the relationship with a third-party client and the inherent fluidity of the studio structure—instructors do not always develop traditional syllabi for their courses but instead maintain flexibility by producing a broad frame or structure for the course with supplementary material provided throughout the semester. When outcomes were not explicitly listed in the syllabus, we did our best to extract these outcomes by interpreting carefully the individual project description and required activities. This leads to an important point to which we return in the conclusion: that a syllabus includes explicit learning outcomes neither indicates the inherent quality of the course nor that it is necessarily “better” than one with a more broadly framed and flexible structure. Similarly, whereas clearly stated outcomes might help students and faculty become mutually cognizant of course expectations, their presence does not guarantee (or even suggest) that more students will achieve these outcomes.
Overall, our primary findings from examining course syllabi and talking with studio instructors kept with our earlier assumptions about the state of the studio in planning schools. Recall our argument that as a pedagogical approach, studio is not particularly well understood beyond notions of simulating practice. Our interviews and syllabi analysis showed that learning outcomes in studio courses were only identified in twenty-six syllabi (59 percent), although in fourteen (54 percent) of those cases, course objectives were primarily related to practical issues associated with the project site. Nor were learning objectives uniformly identified or described within programs or across departments. Different studio instructors state different learning outcomes and assessments in each course offering, and their outcome content is most often related specifically to the subject or case of the studio; for example, a studio focusing on affordable housing provision might include knowledge of low income housing tax credits as a learning outcome. This is a particularly important finding since studio topics found in our survey included such disparate content as neighborhood planning, affordable housing, climate change planning, form-based code development, and TOD station area planning.
Using the generalized model from the education literature (Table 1), we organize our findings using the same three main components: (1) learning outcomes, (2) outcomes assessment approach, and (3) and outcomes evaluation design.
Component 1: Planning Studio Learning Outcomes
We group the dozens of individual outcomes into six categories derived from the planning literature on the subject: communication, professional experience, learning-by-doing, problem-solving, teamwork, and ethics/values (Table 2). In this table, we include category frequencies and several examples of the wording of individual outcomes in syllabi.
Learning Outcomes Stated or Implied in Studio Syllabi
Some of the more common outcome types deserve discussion. With regard to communication, graphic and visual techniques were most commonly taught, since the studio is one of the only opportunities to train students on the foundational skills of physical planning and urban design. Although only fourteen syllabi (32 percent) listed outcomes related to teamwork, all but four of the forty-four total syllabi required significant group work during the term. Subsequent interviews with instructors revealed that teamwork-related outcomes are, in fact, important since most students enter planning programs with very different skill sets—some are proficient in GIS and presentation software, others are skilled orators or organizers, and still others have expertise in real estate finance and development economics. Given these discrepancies, the studio is seen to provide a one-of-a-kind opportunity for students with diverse training to work together to produce solutions to a vexing problem. Indeed, the studio provides one of the only opportunities in planning curricula for sustained peer collaboration, a hallmark of planning practice.
Professional experience was the most common outcome group noted, with more than three of four syllabi (77 percent) noting the importance for students to understand “real world” feasibility and worldview constraints—more specifically, the idea that clients might desire a product that might differ from the professional consultant’s opinion. An example of how this outcome plays out is useful. A studio instructor claimed that the most important outcome of his studio was that students learned to listen to the community and deemphasize their own personal worldview and expectations. This instructor was approached by a group of local farmers concerned about residential encroachment into prime agricultural land. The studio’s goal was thus to produce a more sustainable model of agricultural production in this region. While many students advocated strongly for small-scale, organic farms, it was later revealed that the farming community’s intention for limiting residential encroachment was more about scaling up production and using conventional farming methods to generate economies of scale. Although students were frustrated with the farmers’ worldview, they were able to at least partially bracket their own preconceptions and work with the client to suggest thoughtful compromises and middle-ground alternatives.
Component 2: Planning Studio Assessment Approach
We then reviewed syllabi and asked studio instructors what approaches they used in assessing student learning and the structure of coursework. All instructors use at least one of the following four assessment approaches, and in many cases they used a combination of approaches to assess satisfaction of a particular outcome: 91 percent (forty) use group projects, 82 percent (thirty-six) consult with external clients, 73 percent (thirty-two) have students make public presentations, and 59 percent (twenty-six) include individual work on assignments.
Assessment approach 1: Group projects
Group work was a mainstay in studio courses. The primary task for groups was to design a robust process including data collection, analysis, and plan production. Thirty-nine of forty syllabi requiring group work asked students to produce collectively a final professional report with tasks divided among individuals. Group projects necessarily involve balancing busy schedules and the delegation of tasks to group members with varying skills and expertise. When asked to expand on why group work was important, three interviewees suggested that recognition of one’s own “place” in a studio group was the most important outcome of the course. Heumann and Wetmore (1984) also offer that a “great part of a student’s education [comes] out of being together with classmates” (127). This physical propinquity—and the provision of shared, physical space in the studio classroom itself—provides opportunities to meet and discuss work formally and informally, approximating a true professional work environment. This arrangement also helps students recognize the iterative, long-term nature of solving “wicked” problems familiar to practicing planners. Most often, group work deliverables include the production of a plan itself, which can be evaluated based on a set of clear standards. In this regard, group deliverables can be assessed by instructors and external clients and juries using traditional, norm-referenced assessment standards found on rubrics and similar devices, whereas group work is best evaluated through self-assessments and peer comparisons of contribution quality.
Assessment approach 2: Work with external clients
That 82 percent of studios worked with clients suggests that the client relationship is one of the most indispensable components of the studio, and one of the most beneficial for achieving stated learning outcomes. One interviewee claimed the chief reason to work with a client is to increase student accountability, forcing students to be more attentive to deadlines and details because they are working for “more than a grade.” Similarly, two other faculty members noted that the studio is an opportunity for students to provide real benefit to communities; these real-world projects help students recognize and internalize the gravity of their intervention.
Assessment approach 3: Public presentations
Nearly as prevalent as external client work is the use of public presentations in studio courses. In some cases, students are asked to lead community workshops, focus groups, or other participation forums. In order to prepare students for these community encounters, instructors might require students to attend presentations made by planning professionals: three syllabi required attendance at a local city council or planning commission hearing. All thirty-two syllabi requiring a public presentation asked that students present in groups. Interestingly, although presentation quality could be assessed using direct, criterion-referenced methods related to content, clarity, or creativity, none of the syllabi noted the use of rubrics or other formal mechanisms. Seven of the eleven interviewees (64 percent) admitted they give scores based on an “overall perception” of presentation quality.
Assessment approach 4: Individual assignments
Faculty accustomed to teaching traditional courses are most familiar with the individual assignment. Assessment is based on absolute, criterion-referenced quality standards. In studio courses, students are often asked to finish a complete set of shorter assignments that build to a larger whole. Most involve several stages of project development, from site analysis and mapping to precedent studies and data collection to, at times, a concept plan for an individual site. This progression of separate assignments stresses the importance of deadlines and sequencing, and helps students recognize the skills in which they excel or need improvement. Ten of forty-four syllabi (23 percent) required readings on the plan-making process or on seminal texts on topics covered in the individual offering. One faculty member accustomed to teaching a first-semester planning studio requires all students to read and summarize planning canon from Jane Jacobs, Lewis Mumford, and Kevin Lynch, all in the first week of class.
Component 3: Studio Outcomes Evaluation Design
Lastly, we tried to determine how, specifically, instructors designed evaluation protocols. The main differentiating factor was the evaluating party; as such we group syllabi information into four categories. All forty-four syllabi indicated that instructors were primarily responsible for evaluation. Self-evaluation was required in ten cases (23 percent), peers evaluated fellow students in eight cases (18 percent), and external critics were asked to weigh in on grading 14 percent of the time. As earlier, our research indicated a propensity to combine several evaluative techniques within individual courses.
Design 1: Evaluation by instructor
Instructor evaluation is still the most popular technique and is generally most appropriate for individual assignments and even group presentations. Yet as mentioned above, not one faculty member used a rubric or standardized evaluation tool, which suggests these instructors rely instead on experience and “gut feelings” about the quality of work completed by individuals or groups of students. One interviewee simply claimed, “You know good work when you see it.” Because of this lack of standardized evaluation criteria, students in nearly all cases were assessed cross-sectionally at the end of an assignment or the course itself.
Although the education literature notes its rise in currency, two syllabi required students to develop an individual plan of action—and an evaluation contract—for the semester. In this model, the instructor signs off on the plan, and students are individually evaluated on how well they fulfill this plan of action. The instructor monitors progress on the contract and provides regular feedback to students. Although this process is clearly more time-consuming for the faculty member, it allows tasks to be clearly defined and evaluation methods transparently articulated. This technique prioritizes individual goal fulfillment over student competition and may be based on a longitudinal assessment of improvement from start to finish (see also Lusk and Kantrowitz 1990 for the use of individual criteria in studio courses). Although the student contract was not commonly used by studio instructors, two interviewees suggested informally that it might indeed provide a more appropriate model of studio learning outcomes evaluation.
Interestingly, the quantity of student–instructor contact—most studios have double the contact hours and credits of a lecture or seminar—as well as the intensity and depth of individual attention paid to each student—partly a function of both smaller studio class sizes and the semester-long, problem-based pedagogy—make outcomes assessment a much more subjective activity, as instructors can more confidently assess complex concepts like intellectual growth or professionalism.
Design 2: Self-evaluation
Ten courses also used personal reflection as a major evaluative method. The most common evaluation technique we saw was the individual journal as a means of self-reflection on work quality and growth. Three courses required students to write weekly reflective blogs and one asked students to maintain a weekly log of discussions that occurred. The consistency and regularity of the journaling task allows students to reflect on personal experiences and on their understanding of concepts, methods, and practice common to professional planning (see Roakes and Norris-Tirrell 2000 for use of journals in planning studios).
Design 3: Peers
Peers evaluated fellow students in eight courses (18 percent), seven of which occurred where group members were asked to evaluate each other’s contribution as a means of discouraging shirkers (see Grant and Manuel 1995 for more discussion on this method). The use of peer review keeps students accountable to their colleagues but also provides some means of relativity via student-to-student comparisons. In this regard, peer review allows a number of additional evaluation data points on which the instructor can rely in the final grading.
Design 4: Evaluation by external critics
As discussed earlier, studios often include a jury system for evaluation of final projects and presentations. In the six cases in which external reviewers formally evaluated student work, the jury consisted of the client himself or herself, who could assess confidently the value and quality of the proposed plan or intervention. In our own studio teaching experience, jurors are sometimes given rubrics or score sheets to evaluate work as it is presented.
A Planning Studio–Specific Assessment Model
How do these findings inform a learning outcomes assessment model specific to planning studios that fits within the general model from education theory? What is immediately clear is that studios are unique in their ability and intent to impart a combination of cognitive and noncognitive outcomes, and most outcomes identified in the studio context do not fit neatly into one of the content or learning categories in the general model (Table 1). For example, “problem solving” involves the development of critical thinking skills, knowledge of the appropriate processes to accompany project development, and recognition of personal values when confronted with varying viewpoints on a matter. One studio instructor indicated her two central outcomes were to help students understand how issues of politics and power play out “on the ground” and to be creative with solutions in the “safe space” of the studio. These lofty goals point to the uniqueness of the studio context and demonstrate that learning in studios can be at once procedural, substantive and analytical.
In terms of assessment approaches, instructors use a combination of direct and nondirect methods, but emphasis remains on traditional, direct, and criterion-referenced methods normally found in conventional, nonstudio courses. So even as instructors acknowledged that the studio course provides a unique learning experience complementary to seminars and lectures, they continue to transport a traditional assessment model into the studio environment. In fact, although nearly all syllabi listed assessment approaches, nearly all faculty interviewed seemed unclear on appropriate ways to assess, for example, student group work or external client relationships. Others were unsure of whether to test individual improvement before and after the course (i.e., longitudinal) or to hold all students to absolute benchmarks assessed at the conclusion of individual studio exercises (i.e., cross-sectional). We return to this point in the Discussion section.
With regard to evaluation, 100 percent of instructors assess the quality of student products, whereas only six (14 percent) attempted to measure personal growth throughout the semester. This runs contrary to the notion that the “whole atmosphere of the studio has to be one that focuses on learning and not so much on testing” (Lang 1983, 127). Nonetheless, thirteen syllabi (30 percent) used at least one evaluation technique that moved outside traditional, objective, instructor-derived scoring. Relying on combinations of peer and self-evaluation techniques, or subjective and open-ended grading, these faculty members acknowledge the different types of learning occurring in the studio context.
Our analysis resulted in the following studio-specific model, the application of which we discuss after Table 3.
Studio-Specific Learning Outcomes Assessment Model
Model Application
The model is not exhaustive and only includes potential approaches and designs for learning outcomes assessment. As such, a studio instructor might treat the model like a menu, first selecting cognitive and noncognitive outcomes then following it from top to bottom along the two main vertical channels, choosing appropriate assessment and evaluation procedures along the way. For example, one could measure a cognitive outcome such as problem solving, and in particular the formulation of defensible planning decisions, by requiring students to identify potentially catalytic development sites along an environmentally degraded river corridor. Groups of students could be given a discrete assignment—one of several throughout the term—to develop a clear set of criteria for site selection. Instructors and external experts familiar with the area could objectively evaluate the teams’ arguments based on the students’ successful use of accepted techniques like a SWOT analysis, grading for completeness and clarity.
Alternatively, an instructor could assess a noncognitive outcome like ethics/values by asking students to present at the beginning of the semester what a “socially just” planning intervention might look like. Throughout the semester, students would be expected to work with the studio client to modify this definition and develop a more precise set of measurable outcomes on which they then assess their own proposed interventions. Instructors could assess student growth on this outcome by requiring students to complete regular journal entries outlining how their own ethical/moral standards shifted based on interactions with the client or affected group.
Discussion
Studio outcomes, assessment approaches, and evaluation designs do not fit neatly into the general educational model. This may suggest one of two things: either the educational model is inappropriate or inapplicable in the planning studio environment or studio instructors are not as well versed as they should be in basic educational theory. The first claim makes sense, as this article and others in the symposium have proven that studio is different from traditional seminar or lecture courses, and the literature from which the general model was drawn fails to explicitly acknowledge workshop-style courses such as studios. Indeed, the generic learning outcomes assessment model assumes a certain pedagogical structure that just does not exist in this unique environment. The second argument, that studio instructors are not well trained in appropriate pedagogical technique, is also a possible explanation for the mismatch. Nonetheless, this mismatch demonstrates the need for a basic, studio-specific learning outcomes assessment model. This article has served to start that conversation.
Our analysis also reveals a clear mismatch between learning outcomes and the assessment approaches and evaluation designs used to measure them. While nearly all instructors used at least one or two noncognitive learning outcomes such as “professionalization,” “responsibility,” or “service,” all but a few used assessment methods more appropriate for evaluating competence on cognitive outcomes such as “communication skills,” “plan creation,” and “representational techniques,” as these are more easily measured with objective scores and are thus more defensible if challenged by students. When asked which PAB outcome measure they felt was most relevant to their own assessment, most felt that competence was the most easily tackled at the course level. Yet this term itself, defined by Kirschner et al. (1997, 151) as “the whole of knowledge and skills [one] can use efficiently and effectively to reach certain goals,” is related to the measurement of substantive and procedural outcomes situated solely in the cognitive realm.
In addition, many instructors collapse several outcomes into one, and each composite is assessed and evaluated using a combination of techniques. One instructor revealed that his most important learning outcome was to have students “demonstrate the ability to formulate compelling planning decisions appropriate to a specific context.” In our follow-up interview, he described that for his final project—a plan-making exercise—regardless of whether he or the client agreed with the solutions proposed by an individual or group of students, his only assessment procedure was to use direct, absolute standards to determine how well they were able to produce a defensible, logical rationale for said solutions. In other exercises throughout the term, he evaluates students on a combination of subjective participation scores and peer evaluations, as well as objectively scored, knowledge-testing exams along with three modules evaluated on the quality of deliverables like plans and presentations.
Studios are unique pedagogical opportunities. Four summary points are worth noting:
(1) The student–faculty relationship in planning studios makes assessment and evaluation more subjective.
In our experience, subjective assessment approaches and evaluation design are more common (although still fairly rare) in studios than in seminars or lectures. This might be attributed to the increased individual attention given studio takers due to smaller class sizes and increased contact hours. Although this factor might help instructors more easily assess student growth or effort throughout the semester, this same factor might paradoxically explain the lack of clear assessment standards and a reliance on “gut feelings” for evaluation.
(2) Evaluating outcome achievement is difficult because planning students generally possess wide-ranging skills and skill levels.
Some incoming planning students have significant expertise in graphic communication and software, whereas others have none at all. Some have worked as professional consultants and understand the business of planning; others have never worked in this milieu. The debate many studio instructors have, then, is with so many starting points, should we teach to a certain benchmark or assess individual growth from start to finish? In other words, perhaps our instructional intent is for all students to end up at some defined “level 10.” One student starts at level 3 and another at level 7. At the end of the course, the level 3 student ends up at level 8 but the level 7 student also only gets to level 8 of proficiency or competence. Who should receive a better grade? The unique student–faculty relationship outlined above can help smooth these waters, as can a combination of absolute, minimum-referenced assessments (e.g., assignments testing baseline knowledge of key planning concepts) with relative, peer-referenced assessment approaches (e.g., student growth during term demonstrated through individual contract).
(3) Instructors expect planning studios to satisfy cognitive and noncognitive outcomes: most times they can.
All studio syllabi listed both cognitive and noncognitive outcomes. Contrary to most traditional lecture courses, where knowledge acquisition is prioritized, or in practical workshops, where skill development is most germane, studios are expected to impart skills, knowledge, and values. They often can, and do, but measuring competence on each of these components can prove challenging.
(4) The unique features of the planning field itself provide for a pedagogical environment with both unclear standards and assessment approaches, especially with regard to PAB accreditation.
Planning is both a professional pursuit and a public service. All accredited programs in planning necessarily have a professional orientation because of standards set by PAB and by the American Institute of City Planners (AICP), especially in the latter’s Code of Ethics. Indeed, several instructors refer to the Code of Ethics when devising their course learning outcomes and several argue that students should complete the class with an understanding that all actions must attend to the “public interest.” Also, planning is a place-based practice, so appropriate solutions are always context sensitive. Therefore, absolute standards are difficult to set, maintain, or measure in professional practice itself, let alone in the studio environment (although moral absolutes like honesty, factual accuracy, and clear assumptions are still judged important in both contexts). And as mentioned earlier, the PAB model of assessment is problematic in that its outcome measures are nearly impossible to operationalize in the studio environment, where subjective outcomes like “translating theory into practice” are quite common.
Conclusion
Generally speaking, planning studios share some common goals (synthesis, learning-by-doing, reflection-in-action) and some common pedagogical characteristics (open-ended problems, real or constructed clients and teamwork). We show that studio courses are seen as important offerings in planning programs—84 percent require these classes—and they differ from other courses in the curriculum in a number of ways. And yet there are very few standards and no common understanding of what we expect students to learn from studio or what strategies, indicators, or evaluative measures faculty might use to deliver these objectives. Thus the assessment of learning outcomes is difficult, especially when based on capturing educational objectives that are appropriate and viable in the studio context. For example, we know that studio courses have the capacity, at least theoretically, to engage moral and ethical dilemmas, but perhaps because these outcomes are not commonly engaged in the nonstudio environment, there is little in the way of planning scholarship to guide instructors in executing this aspect of assessment.
This article relies on educational literature to produce a general model of learning outcomes, assessment, and evaluation that can guide faculty and administrators in all types of educational programs. We draw on a review of existing planning studio syllabi and interviews with instructors to modify and augment this general educational model (Table 1) into a planning studio–specific model (Table 3) that can help faculty and administrators understand the unique role of studio in the planning curriculum. We present evidence that studios represent a clear departure from traditional classroom-based courses; a single studio course is often expected to result in both cognitive and noncognitive outcomes and impart important skills, knowledge, and values to students.
In cases where learning objectives are simply not expressed—meaning that syllabi do not make clear how the learning outcomes will be assessed, and the studio instructors do not deliver this information through informal channels—then appropriate responses might include better training of studio instructors and production of more scholarship about studio learning outcomes and assessment approaches. Still, listing learning outcomes is no guarantee of quality. On the contrary, anecdotal evidence suggests that since central administrators at some universities began to require that all syllabi explicitly list course learning outcomes, some faculty “do the minimum” and list very generic outcomes just to satisfy the requirement. More research must be done to understand whether and how the presence of outcomes on a syllabus is related to actual achievement of said outcomes. Ultimately, this exercise suggests we need more rigor in both the clarity of learning outcomes themselves and in the manner(s) in which they are assessed in the planning studio.
The findings serve our ultimate aim of embracing learning outcomes—and PAB requirements thereto—as an opportunity and means to improving our understanding of what studio pedagogy can and cannot do. Moreover, our “output”-oriented approach can also ask questions about what kinds of learning outcomes might be best achieved in studio courses as compared to other kinds of instructional approaches. Yet our findings are exploratory at best. Our sampling frame is small, and our sampling technique nonrandom, thus our findings cannot be generalized to all planning faculty or programs.
Nonetheless, our exploratory findings serve as the basis of further discussion. This may be particularly true for the case of studio pedagogy, where the need to think more deeply about learning outcomes has emerged only recently. For example, our findings point to a high correlation between expressed learning outcomes (either in course syllabi or through conversation with the instructor) and a broader understanding of the pedagogical intentions of the program in its entirety. In this sense, well-defined learning outcomes were tethered at both the course level and the program level. Since no single program will satisfy its full slate of educational objectives through a single course, or a single pedagogical approach, this relieves faculty of trying to meet all program-level outcomes in single courses, or even groups of courses, such as the core curriculum.
Similarly, our interviews with faculty point to circumstances where the most coherent learning outcomes in studio courses seemed to be evident in programs that had made a formal effort to discuss educational outcomes at the program or departmental levels. In programs where the majority of faculty members are engaged in, and thoughtful about, educational theory, while also actively involved in deliberations about the curriculum, the assessment of learning outcomes was far more transparent to faculty, students, and to accreditation bodies. As Kuh and Ewell put it, “the two greatest needs to advance student learning outcomes assessment are greater involvement of faculty and more assessment expertise, resources and tools” (2010, 21).
This brings us back full circle to the PAB accreditation requirements, and the implications of our findings. Since much of the thinking about assessment of learning outcomes is directly tied to accreditation reviews—certainly this was the case for both authors as for many of the programs we interviewed—the PAB has an excellent opportunity to encourage more rigorous study of both the appropriate models in planning and the kinds of assessment techniques that best suit our profession. Proposed changes to the Accreditation Document hold some promise in their attention to program “outputs,” especially insofar as coursework, and thus planning studios, must now be explicitly assessed on its ability to provide graduates the array of knowledge, skills, and values required for successful professional work (PAB 2012).
At the same time, we recognize the difficulty of precision in the rhetoric of assessment, and the discomforts of a one-size-fits-all model. Studio pedagogy, with its lofty promises of synthesis amid complexity, might prove an excellent case study for exploring and bounding these tensions in educational outcomes.
Footnotes
Appendix A
Appendix B
Top 25 Planning Programs (Source: Planetizen 2009)
| 1. Massachusetts Institute of Technology |
| 2. Cornell University |
| 3. Rutgers, The State University of New Jersey |
| 4. University of California, Berkeley |
| 5. University of Illinois at Urbana-Champaign |
| 6. University of North Carolina, Chapel Hill |
| 7. University of Southern California |
| 8. Georgia Institute of Technology |
| 9. University of California, Los Angeles |
| 10. University of Pennsylvania |
| 11. Harvard University |
| 12. University of Michigan |
| 13. University of California, Irvine |
| 14. University of Illinois at Chicago |
| 15. University of Texas at Austin |
| 16. University of Cincinnati |
| 17. University of Florida |
| 18. University of Washington |
| 19. Virginia Polytechnic Institute and State University |
| 20. California State Polytechnic University |
| 21. The Ohio State University |
| 22. Florida State University |
| 23. University of Oregon |
| 24. Portland State University |
| 25. New York University |
Acknowledgements
We would like to thank the studio instructors and program administrators who responded to our survey. We would also like to thank the many presenters and attendees at the 2009 and 
sessions on the topic of studio pedagogy, for their insights and robust questions that developed our thinking on this topic. Finally, we would to thank our three anonymous reviewers, as well as JPER Editors Weiping Wu and Michael Brooks, for their constructive commentary.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.