Designing for Improvement in Professional Development for Community College Developmental Mathematics Faculty

Abstract

More than 60% of the nation’s 14 million community college students are required to complete at least one developmental mathematics class before enrolling in college-credit courses; however, 80% of them do not successfully complete any college-level mathematics course within 3 years. To address this problem, the Community College Pathways initiative was launched in 2009. A core element is the Pathways Faculty Support Program, professional development aimed at supporting faculty to develop knowledge, beliefs, skills, and practices for teaching Pathways courses. In fall 2014, a redesign of the program used improvement science methods to design an infrastructure for responsive, flexible professional development that is sensitive to varying and changing conditions, with mechanisms for identifying, designing, and testing ideas for improvement. Specifically, our team employed a user-centered design process that foregrounds the needs and work processes of Pathways faculty and a measurement system that drives the continuous improvement of the program.

Keywords

community colleges mathematics education professional development improvement science

Introduction

The academic success rate of developmental mathematics students at community colleges is alarmingly low. More than 60% of the nation’s 14 million community college students are required to take at least one developmental mathematics class before they are eligible to enroll in college-credit courses. However, 80% of the students who place into developmental math do not successfully complete any college-level mathematics courses within 3 years (Bailey, Jeong, & Cho, 2010). Many students spend long periods of time repeating courses and ultimately leave college without a credential. This means that millions of students fail to acquire essential mathematics skills and are unable to progress toward their career and life goals.

In 2009, the Carnegie Foundation for the Advancement of Teaching launched an initiative aimed at dramatically increasing the college-level mathematics completion rates of community college students who place into developmental mathematics. This initiative, called the Carnegie Community College Pathways, engaged a network of practitioners, researchers, designers/developers, and institutional leaders to design and implement a reform effort that addressed challenges to students’ mathematical success systemically. The approach includes profound changes to curricular pathways, content goals, lesson materials, classroom pedagogy, student motivation and engagement, and student advising and placement, in addition to attention to faculty development. The centerpiece of the initiative is two curricular pathways—Statway® and Quantway® (referred to above collectively as the Pathways)—currently adopted in approximately 50 colleges across the United States. These pathways are designed to accelerate students’ progress through their developmental mathematics sequence and a college-level course in a single year, as compared with sequences involving from three to five semester-long courses (including college-level mathematics). Statway integrates developmental mathematics and college-level statistics, whereas Quantway covers developmental mathematics and college-level quantitative reasoning.

This article reports on a key element of the Pathways initiative—faculty professional development aimed at preparing and supporting faculty teaching Statway or Quantway for the first time. Despite their primary mission as teaching colleges, community colleges typically have poor infrastructure and weak culture for supporting professional development about instruction. Prevailing modes of professional development typically involve one-shot workshops that do not provide meaningful and sustained opportunities for professional learning for instructional improvement (Bailey & Smith Jaggars, 2015; Huber, 2008). Furthermore, we know that high-quality professional development is critical for sustained positive impacts in system reform (Desimone, 2009; Fishman, Marx, Best, & Tal, 2003; Supovitz, Mayer, & Kahle, 2000). This means that a key driver for dramatically improving student outcomes using the Pathways is faculty professional development designed to support instructors’ developing the knowledge, beliefs, skills, and practices for teaching mathematics for understanding in the context of Statway or Quantway.

In spring 2014, a pilot program was launched to prepare and support first-time faculty teaching the Pathways—with mixed results. A comprehensive redesign effort was initiated in fall 2014, using methods and tools from improvement science. Specifically, we (a) used a user-centered design process in the redesign of the professional development program foregrounding the needs and work processes of new Pathways faculty and (b) developed a measurement system that drives the continuous improvement of the professional development program. This article presents an analysis of the design and development work involved in the program redesign, illustrating a case of the application of improvement science methods to the problem of faculty professional development in this particularly challenging institutional context. The case highlights how improvement science approaches enable the development of an infrastructure for responsive and flexible professional development that is continuously improving, that is, continuously sensitive to varying and changing conditions, with mechanisms for identifying areas for improvement and for designing and testing ideas for improvement, as well as structures for capturing and sharing learning across the network of instructors.

The Challenge: Changing Community College Mathematics Teaching

For many community college faculty, teaching Statway or Quantway is a radical change from their established classroom practice. In stark contrast to traditional community college mathematics instruction, Pathways pedagogy is grounded in research on teaching for powerful mathematical understanding that foregrounds students’ productive struggle with substantive mathematical problems, opportunities for rich mathematical discourse, making explicit the conceptual connections that structure course content, and the role of collaborative learning in promoting mathematical sense-making (Bransford, Brown, & Cocking, 1999; Hiebert & Grouws, 2007). In addition, the curriculum and pedagogy also integrate interventions and practices from social-psychological research designed to support students’ engagement and persistence (Dweck, 2006; Dweck, Walton, & Cohen, 2014; Hulleman & Harackiewicz, 2009; Silver & White, 2013). However, teaching in community college mathematics classrooms largely resembles the content-focused, knowledge transmission model so prevalent in undergraduate instruction, especially in lower level, remedial courses (Bailey, Jaggars, & Jenkins, 2015; Grubb, 1999; Grubb & Gabriner, 2013; Kember & Gow, 1994). Even in those classrooms in which student participation is emphasized, instructional activities tended to focus on factual and procedural knowledge as opposed to conceptual content and mathematical sense-making (Mesa, 2011). In addition, instructors who may be open to alternative methods of instruction, such as “learner-centered” models or models of instruction that are more facilitative in nature, are often skeptical of the efficacy of these approaches for developmental students, who they perceive as weakly prepared and resistant to such strategies (Grubb & Gabriner, 2013).

Furthermore, although community colleges faculty’s responsibilities involve teaching and service rather than an emphasis on scholarly research, community college faculty are no more likely to have had pedagogical coursework or training than faculty in research institutions (Grubb, 1999). In mathematics departments, it is much more common for community college faculty to have master’s or doctoral degrees in mathematics than any degree or coursework in education (Grubb, 1999). Thus, many faculty teaching the Pathways have had little experience with, or even exposure to, instructional practices other than lecture or teacher-centered discussion.

The core challenge, then, is to support new Pathways faculty as they make sense of and take up practices that are likely out of their comfort zone. However, despite their mission as teaching colleges, community colleges are not necessarily hospitable environments for professional learning opportunities with the potential to significantly change teaching practice in a sustained and generative way—for example, ongoing, coherent professional development experiences structured around collaboration, inquiry, and reflection (Bailey et al., 2015; Grubb, 1999; Huber, 2008; Hunzicker, 2010). First, community college faculty’s teaching responsibilities limit opportunities for professional growth due to heavy teaching loads. As Grubb (1999) notes,

The dark side of being a [community college] teaching institution is that faculty have to teach much more—an average of sixteen classroom hours a week, 50 percent more than faculty in state colleges and more than twice that of faculty in research universities, with obvious effects on the time available to ruminate about what good teaching might be. (p. 9)

Ironically, the very instructors who may need the most support for instruction have the least opportunity for it. Furthermore, in the increasingly utilitarian climate caused by shrinking budgets, an emphasis on economic participation as a goal, and louder calls for accountability, priorities for scarce resources are not often given to forms of faculty development that often can be perceived of as inefficient or even luxurious (Tinberg, Duffy, & Mino, 2007). In developmental education, the situation is even more challenging. Seventy-six percent of instructors in developmental education are adjunct, part-time faculty, who are typically assigned even heavier teaching loads, often spread over multiple campuses (Center for Community College Student Engagement, 2014). In addition, adjunct faculty frequently do not even have access to whatever resources for professional development that exist for full-time faculty.

The Faculty Support Program (FSP): A Professional Development System for Pathways Faculty

Taking into consideration the many barriers to quality professional development faced by community college instructors, in 2013 the Pathways initiative launched the development of a professional development system called the Faculty Support Program that prepares new Pathways faculty to teach Statway or Quantway and supports them during their first year of teaching. The design of the program was informed by the FSP Design Framework, a set of principles derived from research on effective professional development, primarily from K-12 settings (Garet, Porter, Desimone, Birman, & Yoon, 2001; Guskey, 2002; Hawley & Valli, 2007; Hunzicker, 2010; LeMahieu, Roy, & Foss, 1995):

Program structure provides for sustained opportunities for professional learning.

Learning activities are job-embedded, supporting emergent problems of practice.

Learning activities are context/discipline-specific.

Learning activities provide opportunities for collaborative reflection.

Learning activities are centered around artifacts/representations of classroom practice.

The context of the Pathways network created specific challenges for program design. The faculty members in our network are spread throughout the country, and there are increasing numbers of adjunct faculty teaching the Pathways. Thus, the work of promoting community and collaboration across long distances (and time zones) and among instructors with limited time is complex and expensive. In addition, unless mandated by a particular college in its implementation of the Pathways, the FSP is voluntary, which means that, for most faculty, participation is outside of their professional obligations. Relatedly, there is tremendous variability in the availability of resources for and interest in supporting faculty learning to teach the Pathways on the part of college administration. Thus, there are few incentives to participate other than instructors’ personal commitment to effectively teaching Statway or Quantway and their interest in learning new ways of teaching. Finally, the Pathways network is expanding rapidly; the network has tripled in size (from 1,551 enrolled students in 2011-2012 to 4,126 in 2013-2014) in the last 4 years and is projected to grow at an even faster rate in the next 4 years (Sowers & Yamada, 2015). Scaling the program while remaining accessible and effective is a tremendous challenge.

Therefore, the FSP must be flexible and responsive to the needs of faculty and clearly demonstrate value for time spent to faculty (and their administration) while also providing meaningful opportunities to learn about and through their experiences with these new teaching practices. As Guskey (2002) notes with respect to K-12 professional development,

What attracts teachers to professional development . . . is their belief that it will expand their knowledge and skills, contribute to their growth, and enhance their effectiveness with students. But teachers also tend to be quite pragmatic. What they hope to gain through professional development are specific, concrete, and practical ideas that directly relate to the day-to-day operation of their classrooms (Fullan & Miles, 1992). Development programs that fail to address these needs are unlikely to succeed. (p. 382)

This observation is even truer for Pathways faculty.

Initial Design and Launch of the FSP

The pilot of the FSP was launched in spring 2014 and ran through spring 2015. The goals of the program at this time were twofold. First, the program sought to prepare faculty to teach Statway/Quantway at a basic level of efficacy. It focused on the Statway/Quantway curriculum design, the pedagogical model and principles underlying the instructional approach, social-psychological interventions and practices addressing specific factors affecting developmental mathematics students’ persistence and success (known collectively as Productive Persistence), and technical aspects of running the online and in-class components of the courses. Second, the program aimed to support faculty in their first year of teaching, with particular focus on practices and interventions for the first few weeks of class designed to establish a classroom culture supporting Productive Persistence (called “Starting Strong”), collaborative inquiry into problem-centered instruction, and sharing and addressing emergent problems of practice.

The program included multiple modalities for faculty engagement and was comprised of the following:

a set of online activities, hosted in Google sites, designed to engage faculty in key concepts, practices, and information important for teaching the Pathways that extensively involved analysis of videos of Pathways instruction and video-based interactional activities (using Zaption, a web-based tool for creating and engaging in interactive learning activities using video) as well as curriculum analysis;

face-to-face intensive workshops, such as at the National Forum (the annual meeting of the Pathways network) that provided hands-on opportunities to engage in the Pathways learning and teaching experiences; and

mentoring from experienced Pathways faculty members (called Faculty Mentors) who work with faculty in all aspects of planning and teaching, including ongoing just-in-time support.

Specific learning activities and resources were developed based on the FSP Design Framework as well as an initial assessment of the foundational practices and most salient challenges of effective Pathways instruction by the Carnegie Foundation staff and experts in classroom instruction. An instructor’s access to the FSP began upon his or her assignment to teach Quantway/Statway, lasted through his or her first year of teaching (roughly 12-15 months), and consisted of about 30 to 35 hr of involvement over all components.

Learnings From the Initial Year

During the program’s inaugural year of implementation, we learned that our initial assumptions and theories about how best to support new Pathways faculty needed revisiting. A developmental evaluation¹ of the FSP was conducted in fall and winter of 2014. The results of the evaluation, as confirmed in subsequent interviews with Faculty Mentors and new faculty, suggested that unwanted variation existed in the performance of all the components of our system for supporting new faculty. Fewer than half of new faculty teaching in the 2014-2015 academic year attended the workshop, primarily due to a lack of funding or late notification about teaching assignments. Online engagement analytics (from Google sites and Zaption) revealed that the use of the online activities and resources varied widely between faculty, and even grew sparse as new faculty began teaching during the fall term. Finally, surveys designed to check in with new faculty—before, during, and after their first year of teaching—to gauge their feelings about the available sources of support indicated that these online resources were scarcely used and engagement with the FSP structures was variable. Instead, new faculty largely relied on Faculty Mentors and (if accessible) experienced Pathways faculty at their own colleges for support in their teaching of Statway or Quantway.

In addition, through our interviews with faculty, we learned that the outcomes of the FSP were not clear to important stakeholders—primarily Faculty Mentors and new faculty. Initially, a major outcome of program completion for faculty was certification by Carnegie signifying a faculty member’s readiness to teach Statway or Quantway. We had believed that this certification, although not necessarily a formal accreditation, would be of value for community college faculty, particularly adjunct faculty, as evidence of their professional growth and professional preparation to teach the Pathways generally (i.e., across colleges). In addition, the initial aim statement of the program, an articulation of the program’s measurable goals, said that the program would be successful if all new Pathways faculty teaching for the first time felt supported regarding and valued the Pathways’ instructional approach. Neither the outcome for faculty nor the program aims resonated with stakeholders. For example, improvement science specialists with whom the team consulted suggested that the aim statement be centered around student outcomes. This vision of quality was not shared with faculty members and Faculty Mentors involved in the design of the program. For them, retention of faculty was the preferred aim of the FSP, and they reasoned that to retain faculty, they must feel supported and must value the Pathways’ instructional approach. In addition, despite initial concerns about levels of faculty engagement in the FSP, the program aim statement did not address how it could be designed in such a way that faculty find responsive to their needs. The lack of a single, widely shared aim of the program left the designers and implementers of the FSP without a shared vision of what constitutes high-quality system of faculty support, which in turn made it difficult to gauge whether the program was successful.

This evidence of the lack of engagement in the FSP preparation and support structures, as well as the program’s lack of a shared vision of quality forced us to reexamine the initial design of the FSP, resulting in a comprehensive redesign project. This redesign is the focus of the present case, highlighting the application and utility of methods and tools of improvement science in service of designing a system that was more responsive and flexible than the initial FSP and reflected a shared vision of high-quality professional development.

The Tools of Improvement Science

We saw that what is required is a professional development system designed for continuous learning and practice improvement, with faculty needs, resources, work processes, and teaching contexts at the forefront. Traditional mechanisms for feedback and evaluation of professional development programs, particularly summative evaluations as are often used in institutions of higher education, are blunt tools; they typically inform program designers and facilitators about whether or not a program “worked” or how well it was received (Guskey, 2000), but do not provide actionable formative feedback that can inform ongoing improvement. They also generally do not provide insights into their users—the faculty—and faculty’s needs, resources, and constraints to the extent necessary for designing responsive and relevant learning opportunities.

Improvement science provides a set of tools, approaches, and methodologies for designing and improving systems that are user-centered, responsive to varying and changing conditions, and are structured for continuous learning. These tools and methods allow designers (and users) to learn rapidly about how the system is functioning and undertake small iterative tests of change that refine elements and processes of the system—what Bryk, Gomez, Grunow, and LeMahieu (2015) refer to as the work of “learning by doing.”

As presented in the introductory section of this issue (LeMahieu, Edwards, & Gomez, 2015), improvement science is guided by six principles (Bryk et al., 2015): (a) Make the work problem-specific and user-centered, (b) variation is the core problem to address, (c) see the system that produces the current outcomes, (d) that which cannot be measured cannot be improved at scale, (e) anchor practice improvement in disciplined inquiry, and (f) accelerate improvements through networked communities. These principles provide the conceptual framework for the design of the professional development system for Pathways faculty.

In the specific context of this work, improvement science animates an orientation toward the design of professional development that recognizes and honors practitioner expertise and lived experience, the sharing and collaborative refinement of knowledge and practice, and rigorous (and continuous) accountability, not only to content-related goals but also to goals related to faculty access and engagement, as well as the responsiveness of the program. In the sections that follow, we first describe in general terms the specific tools, approaches, and methods of improvement science that we used, then illustrate their use in a detailed case describing the redesign of the FSP.

Methods and Approach

The redesign of the FSP used a user-centered design process. The purpose of this process is to forefront the users by understanding deeply their needs and work processes, examining how existing products and services fall short in meeting those needs and complementing those work processes, and designing new products and services that better match users’ needs and work processes (Norman, 1988).

Design-based methods and tools from design organizations such as the Hasso Plattner Institute of Design at Stanford and IDEO (Brown, 2009) informed the five phases of the user-centered design process we used: Empathy, Define the Problem, Ideate, Prototype, and Test. Table 1 presents the purposes of each phase and examples of possible tools used in the phases. It is important to note that the phases, although presented here as a linear sequence for narrative clarity, are meant to be deeply iterative and recursive, with phases often occurring in conjunction with one another. This is true for design thinking generally, with phases of design thinking meant to be “thought of as a system of overlapping spaces rather than a sequence of orderly steps” (IDEO, 2015a).²

Table 1.

Phases of the User-Centered Design Process.

Phase	Purpose	Possible tool(s)
Empathy	Understand deeply the needs and behaviors, including work processes, of the users.	Open-ended interview protocols designed to prompt a conversation about the user’s experience, not guide it.
Define the Problem	Synthesize learning from the empathy phase, including the specification of problem statements that describe insights and challenges.	Affinity clustering, a technique for sharing and sorting design team members’ insights and takeaways to identify themes (LUMA Institute, 2012).
Ideate	Generate ideas meant to solve the problems outlined in the defining phase.	“How Might We?” statements, which ask a question about how to address a more specific component of the problem statement.
Prototype	Create low-cost artifacts to use as a basis for testing ideas.	Crude physical representations, such as mock-ups of webpages on notebook paper or a flowchart of a process.
Test	Use prototypes with users to better understand their needs and iterate on the design of a prototype as more is learned about the user.	Think-aloud testing, an interview in which a user is tasked with using (or pretending to use) a prototype and narrating his or her thoughts as he or she does so (LUMA Institute, 2012).

Source. Hasso Plattner Institute of Design (2013).

Empathy Phase

The purpose of the Empathy Phase is to understand deeply the needs and behaviors (primarily the work processes) of the users of a product or service, so that the product or service’s design can be improved or redesigned to better fit the needs and behaviors of the user. Common tools to serve the purpose of this phase are open-ended interview protocols about a user’s experience with a product or service. These protocols are not meant to guide nor direct the conversation, but to prompt an open-ended conversation about the experiences of the users (Hasso Plattner Institute of Design, 2013).

Define the Problem Phase

The purpose of the Define the Problem Phase is to synthesize the scattered learning that took place during the Empathy Phase and focus it. The result of this phase is the crafting of problem statements—concise statements that serve to provide the team with a common understanding of the kinds of challenges a design process will address (Hasso Plattner Institute of Design, 2013). One tool that a design team can use is affinity clustering, a technique for sharing and sorting each member’s key takeaways from the Empathy Phase based on similarity. This tool allows teams to find themes in their findings and to prepare problem statements based on themes that emerge (LUMA Institute, 2012).

Ideate Phase

The purpose of the Ideate Phase is to generate a variety of ideas to address the problem statements we are seeking to solve. The Ideate Phase is characterized by a flaring of ideas that are not limited to what is feasible or reasonable. The main goal of this phase is to generate a bank of solution ideas, not yet to evaluate whether or not they are actually possible. A primary tool for carrying out this phase is the use of “How Might We?” statements. These statements seek to ask questions about how to address a certain aspect of a problem, which facilitates the generation of ideas. The Hasso Plattner Institute of Design at Stanford provides an example using a problem with the ground experience of passengers at an airport and, in particular, parents who must wait hours for a delayed plane with children. “How Might We?” statements for this particular challenge included the following: “How might we make the airport a place kids would want to go?” or “How might we entirely remove wait time at the airport?” By picking different aspects of the problem, the “How Might We?” statements facilitate idea generation with the right level of specificity (Hasso Plattner Institute of Design, 2013, p. 3).

Prototype Phase

The purpose of the Prototype Phase is to develop testable artifacts based on ideas that a team is considering pursuing. Early in this phase, prototypes are meant to be crude representations of ideas, created with minimal resources (e.g., time and money). These prototypes allow designers to test what is or is not possible with users, while ensuring that any design failures are quick and cheap (Hasso Plattner Institute of Design, 2013). Prototypes can take many forms (particularly in their early stages); for example, design teams may use physical drawings (on regular notebook paper) to create mock-ups of online spaces, skeletons of a table of contents to represent a set of resources, process maps and flowcharts to facilitate a walkthrough of activities, and so forth.

Test Phase

The purpose of the Test Phase is to test prototypes with users to ensure that users’ needs and work processes are forefronted. The Test and Prototype Phases are iterative—the testing of prototypes informs the redesign and revision of said prototype, and that redesigned and revised prototype is then tested again (Hasso Plattner Institute of Design, 2013). A common tool in the Test Phase is think-aloud testing, in which the testers of a prototype would be asked to use it—or in some cases, pretend to use it—and narrate their thoughts and feelings as they do so (LUMA Institute, 2012). In addition, personas—fictional representations of specific types of users—allow for design teams to test prototypes from the perspective of users. Personas, which are often in the form of short narrative descriptions of a fictional character, are meant to give teams a clear idea of a type of user and consider them when designing (Wood, Wendell & Holtzblatt, 2004).

Our implementation of the user-centered design process iterated through these phases, though not in a lock-step sequence. We moved flexibly between phases of the work, for example, probing an observation from the Empathy Phase more deeply to aid in the development of a problem statement, or empathizing in the Prototype Phase to better inform design details, or ideating further in the face of new learnings from testing.

The core team undertaking the redesign, the FSP Design Team, included four Pathways faculty members and two Carnegie Foundation staff members (Ann Edwards and Carlos Sandoval). The faculty members were selected in part to represent the different types of faculty across the network—Quantway and Statway, new and experienced in the Pathways, full-time and part-time/adjunct, and from colleges in different areas of the country—and in part based on their prior experience supporting faculty colleagues and expressed interest in faculty development and improvement methods. The team engaged in all the phases of the user-centered design process collectively, meeting approximately every 2 to 4 weeks via Google Hangouts and convening 3 times for day-long face-to-face working meetings. As is detailed below, data were shared via Google docs (primarily in the Empathy Phase, but data were also collected in the other phases), and analysis of the data was done collectively in video conferences or in our face-to-face working meetings (across all phases).

Methodologically, the user-centered design process allowed the team to capture, document, and interpret multiple forms of data pertaining to faculty needs, work processes, opportunities and resources for learning, and perceptions about professional development for the purpose of acting upon those learnings to design and improve the FSP. In short, the goal of this work is to generate knowledge that informs design and continuous improvement. Thus, the case that we report here is not a traditional study of teacher learning in professional development or of a specific model of professional development, but rather an examination of the processes through which a professional development system designed for improvement is developed and implemented.

The FSP: Implementing User-Centered Design

To illustrate the application of the principles of improvement science to our core challenge—providing high-quality professional development to community college developmental mathematics instructors—we detail here how the FSP Design Team, during the 2014-2015 academic year, used the user-centered design process to redesign the FSP in response to significant initial failures.

Empathizing With New Pathways Faculty

Empathizing with users of a product or service is a core aspect of user-centered design. To best serve the needs of a user, it is critical to understand what those needs are. In our context, this meant developing a detailed understanding of faculty members’ work processes as well as the faculty experience in teaching the Pathways for the first time. This orientation toward empathy allowed us to see this redesign as better conceived as one that is done with faculty as much as for them.

As part of the Empathy Phase, the team interviewed a total of 32 faculty and administrators—both in and out of the Pathways network. The interviews were conducted by the six team members. Ranging from 30 to 90 min, the interviews were designed to probe what the new faculty had experienced, what challenges they encountered, and how they felt when teaching the Pathways for the first time. They also probed faculty experiences in seeking and receiving support for teaching. This process gave particular attention to a specific type of user in the Pathways—adjunct faculty. Adjunct faculty, we learned, faced challenges distinct from other faculty.

While many full-time faculty in the Pathways received financial and institutional support to pursue professional development opportunities provided by their college and the Carnegie Foundation, adjunct faculty received minimal support to do so. We also learned that adjuncts are likely to teach at multiple colleges simultaneously due to their own personal financial needs, financial constraints across many community college campuses, and caps set on the number of courses they are able to teach on a single campus. This greatly challenges the time they have available to seek out and take up professional development opportunities. As the Pathways Program has spread, we have found that an increasing number of faculty teaching Statway and Quantway are adjuncts. Given that this type of user in our network is growing, and that this type of user has unique financial and time constraints, the team developed an adjunct faculty persona—a fictional person who represents a group of users developed from interviews, research, and testimonials—to consider the experience of adjunct faculty members at colleges and how that affects their experience teaching the Pathways for the first time.

Our findings were consistent with, and expanded on, our initial learnings about the FSP, and included insights about faculty’s engagement in the FSP and in the network, sources of support, work processes, and instructional challenges. These served as the team’s design considerations and constraints. They are as follows:

New faculty did not engage with the FSP’s online resources; the activities in the resources were too long, not responsive to their needs, and often inaccessible.

Faculty Mentors and new faculty members’ Pathways colleagues at their colleges were accessible and were generally responsive to their requests for support. For this reason, new faculty relied on Faculty Mentors and local colleagues considerably more than they did the online resources.

Faculty felt more comfortable seeking out support—as well as receiving it—from someone they had met in person. This was especially true for faculty who attended the National Forum, where new faculty had opportunities to connect with their mentors.

New Pathways faculty did not feel connected with other network faculty, making it difficult to seek out support and share ideas with one another and creating significant barriers for collaborative reflection; faculty felt that interacting with other new faculty in person was critical for fostering connections they did have with their peers.

Faculty members’ primary resource in preparing to teach were the instructor notes provided in the curriculum materials—detailed explications of the teaching of each lesson that include suggested instructional moves, rationales for specific tasks or questions in the lesson, possible student solution strategies, and insights into how to engage students in the key mathematical ideas of the lesson.

New faculty found it difficult to facilitate group work in a way that allowed for their students to productively struggle with the rich mathematical tasks in the lessons.

New faculty found the pacing of the curriculum challenging—They expressed frustration with covering all of the modules.

Sustaining Productive Persistence beyond the initial Starting Strong weeks proved difficult for new faculty.

Overall, this phase provided us with a deeper understanding of the users of the FSP, as well as their needs for support to teach the Pathways, thus allowing us to better define the problems we sought to address, the next phase of our design process.

Defining the Problems to Solve

Having gathered data from interviews and surveys in the Empathy Phase, the FSP Design team next sought to clarify and focus our learning in the Define the Problem Phase. Although much was learned during the Empathy Phase, the purpose of this phase is to more meaningfully and concretely define the problems the design team will address. In our case, this meant developing problem statements that describe the issues that faculty face in teaching the Pathways for the first time—and the problems we faced as a team in supporting those faculty—to clearly articulate and scope problems that new design may seek to address.

To do this, members of the team were asked to inductively establish themes describing their learnings from the Empathy Phase. Each team member reviewed data from the interviews they themselves conducted as well as across all of the interviews, identifying and articulating themes. The team then collectively reviewed themes, refining these insights into key takeaways that served as the basis for the formulation of design problems. For example, one theme was that when faculty sought out support while teaching, they often sought out experienced Pathways faculty at their institution. Another theme was that faculty placed a particularly high value on face-to-face support. The team then rearticulated the takeaways into problem statements that more clearly defined problems to be solved through design.

For example, the themes above were the basis for the problem statement: “Colleges may not have support systems in place for new Pathways faculty.” It is important to note that many problem statements are broad, such as this one, and can be interpreted and parsed in many ways. The next phase serves to address specific components of these broader statements. This problem statement recognizes the role that institutions, and experienced Pathways faculty at those institutions, can play in supporting new faculty. Although faculty may receive support from local colleagues—which is not the case for all new faculty, particularly those who do not have access to experienced Pathways faculty at their institution—formal structures to support faculty may not exist. These ad hoc and informal systems of support make it difficult to support and improve them. In addition, because the support that faculty value most is face-to-face, this problem statement also recognizes that local support structures can provide spaces for faculty to receive face-to-face support.

The team generated six problem statements. It is important to note that the statements are not mapped one-to-one directly from the themes that emerged from the Empathy Phase; rather, they emerged from synthesis of the themes.

A sense of community to engage with and support one another does not exist for all new Pathways faculty.

New Pathways faculty may not have (or may not be aware of) enough opportunities to become comfortable with Pathways instruction by watching more experienced Pathways faculty teach.

New Pathways faculty may not have many opportunities to get feedback and mentoring while teaching the Pathways for the first time.

New Pathways faculty may not begin teaching Statway with a strong statistics background.

Colleges may not have localized structures in place to support new Pathways faculty.

New Pathways faculty may have a hard time with various core components of Pathways instruction.

Although the Empathy Phase allowed the team to learn a great deal about new faculty and the ways they are—and can be—supported, drafting these problem statements gave the team common, clear problems to address in our design work. Consolidating our learning about new Pathways faculty members and their needs using these statements serviced our work in the next phase: ideation.

Ideation

With clear problem statements drafted, the team moved onto the Ideate Phase of the design process—a phase described as a “flaring” in the process because of the many ideas that are brainstormed to address those problems. First, the team crafted “How Might We?” statements based on the problem statements. “How Might We?” statements serve as mechanisms for transitioning from defining the problem to ideation; they seek to surface specific aspects of a given problem statement and allow for the generation of ideas that address them in the particular. For example, consider the case of the problem statement about new faculty not receiving localized support. This is a complex problem, involving many components, which would have made generating ideas to address this problem difficult. Instead, the team crafted the following “How Might We?” statement to generate more focused ideas: “How might we support Pathways colleges in building local structures for supporting new Pathways faculty?” Ideas were then generated to address this question—again, without regard for feasibility or constraints. In fact, one radical change idea was to have the FSP team, along with a Faculty Mentor, fly around the country to help Pathways colleges build and run local faculty development programs. Although this innovative idea in its raw form was resource-intensive, aspects of this idea were adapted and combined with others as will be described in the Prototype Phase.

Overall, the team brainstormed 57 ideas addressing various “How Might We?” statements across the problem statements. These ideas varied widely, including, for example, modifications to existing online resources, developing new activities involving video-based coaching, engaging new faculty in community-building exercises, and building local faculty development programs, among others. It is important to note that the goal of the Ideate Phase is not to move forward with all of these ideas, but simply to generate a wide range of ideas. Brainstorming these ideas and sharing them with one another fostered discussions that allowed us to build off of one another’s ideas and revise them to include aspects of ideas that might otherwise seem unfeasible.

The Ideate Phase provided the team with innovative ideas about how to redesign the program to better meet the needs of new faculty. Although not all ideas generated were directly acted upon, all helped to inspire the innovative products and services that were designed and developed in the prototyping and testing phases.

Prototyping and Testing

The Prototype and Test Phases begin with the careful selection of a few ideas and move toward the design of crude representations or prototypes of these selected ideas. The prototypes are then tested with users, which, in turn, informs the redesign of the prototype as well as the refining of designers’ understanding of the users’ needs. Thus, it is important that considerable resources are not used on the initial mock-up of prototypes that may not be functional for its users; this means that initial prototypes can come in a variety of forms, and often are simple or crude renderings of the design idea.

Having brainstormed nearly 60 ideas among six team members, the team moved forward into prototyping and testing. Because of the team’s limited capacity, only a small number of the ideas were pursued, although aspects of many of them are incorporated in the ideas we did decide to prototype. We used the following selection criteria: (a) The ideas must be feasible given the resources we had at our disposal, (b) the ideas must have significant potential for impact in supporting new Pathways faculty, and (c) team members were personally interested in working on the ideas. In total, 11 ideas were selected for prototyping, and the team designed six initial prototypes.

To ground this process of prototyping and testing in an example, consider the aforementioned problem of providing new Pathways faculty with local support (see Figure 1). After examining the 57 generated ideas, 1 idea that we pursued involved the FSP team and a Faculty Mentor—seven total team members—traveling to colleges and aiding them in the design and development of their own local professional development programs. Although this was extravagant and unrealistic on its own, aspects of this idea were combined with others in the process of selecting ideas to prototype. For example, instead of the entire team being present for extended periods of time at colleges to help design local workshops, the FSP team and the college’s Faculty Mentor would provide remote, ongoing support using a set of base resources for designing local professional development workshops (a separate idea generated in the ideation phase); only the Faculty Mentor would be funded to travel, and only to the workshop itself.

Figure 1.

Local faculty support handbook: Ideate, prototype, and test phases.

These ideas would become the foundation for a prototype dubbed the Local Faculty Support Handbook. This handbook included the base resources for helping colleges develop systems of local support, such as sample workshop agendas, session plans, and protocols for ongoing team meetings, as well as protocols and articulated processes for using the handbook with college administration and faculty leadership. To test the prototype of the handbook, members of the team role-played in a scenario in which the Faculty Mentor coached a new faculty member responsible for coordinating support for other new faculty at his or her college. This activity was conducted multiple times over multiple sections of the handbook, and this testing led to the design (and testing) of prototypes of protocols meant to provide guidance for how to use the handbook and thereby develop and implement a local FSP. It also deepened our understanding about what new Pathways faculty need to be supported in their first year of teaching the Pathways. And this, in turn, informed other ideas and prototypes—for example, that although local workshops are important, so too were the online resources, which are best used as supplements to and “refreshers” of the topics covered during local workshops, instead of as a primary preparation resource. Second, that prior to attending a local preparation workshop, college faculty could be given a broad overview of the Pathways program that serves to motivate faculty to teach Statway or Quantway.

In our design process, the Prototype and Test Phases provided us with a way to test our ideas at low cost—by beginning with crude prototypes and refining them toward useful product as we continued to test—while also learning more about new faculty members and ensuring that our design products met their needs and fit into their existing work processes.

Six ideas were fully developed that resulted in significant modifications to the initial FSP:

The team drafted a standard contract for membership in the Networked Improvement Community requiring release time from regular course-loads for new Pathways faculty to present to administrators, allowing them to utilize the time to prepare and receive support to teach the Pathways for the first time.

The National Forum workshop for new faculty was redesigned to emphasize community building among new faculty.

The FSP began focusing its efforts on helping colleges develop local faculty support structures, primarily through the Local Faculty Support Handbook.

We designed an online “minicourse” to prepare new Pathways faculty. The content of this online minicourse is similar to that of the National Forum designed for faculty who are unable to attend the workshops.

We developed a set of just-in-time online resources—including Zaption tours, written advice from Faculty Mentors, and classroom activities that provide faculty with support designed to help them during their first year of teaching.

Finally, an online Faculty Mentor has been added to the program to support all new faculty in the Pathways using the FSP’s online spaces.

The User-Centered Design Process and Seeing Our System

To support the selection and organization of the prototypes as well as anticipate the impacts of proposed change ideas, the Design Team created and used system maps of the FSP. System maps are similar to process maps³ but represent the multiple interdependent processes that make up an entire system. They are core tools in helping continuous improvement teams see their systems and understand how a system can be changed to produce the outcomes the team wants the system to produce. By making explicit all structures, processes, routines, and decision points that comprise a system, improvement teams can locate gaps—and efforts to address those gaps—on the system map (Bryk et al., 2015).

These maps—initially developed in the developmental evaluation of the FSP and used continuously throughout the redesign—made explicit the components and processes that comprise the program. For example, the FSP system map⁴ includes the processes, roles, and tools involved in the identification/intake of new faculty to the FSP, initial preparation for teaching, mentoring, and first year support, among many other support and data-related processes. The representation also showed how the multiple processes in the system were linked to one another, highlighting the dependencies in the system.⁵ Thus, the map allowed us to see how change ideas affect specific processes at specific junctures, cascading impacts on related processes, and, ultimately, potential impacts on outcomes. As prototypes were tested and then implemented, the system maps were modified to reflect the changes to the system of faculty support.

Continuous Improvement of the FSP

As mentioned earlier, the initial lack of clear outcomes or definitions of quality for the FSP hindered our ability to evaluate the performance of our system of faculty support. Although we were able to piece together an idea about the performance of the FSP from various data sources and by seeing our system using system maps, a lack of a clear definition of quality faculty support made it difficult for us to do this systematically and routinely; it also made it difficult for us to target or focus our improvement efforts.

The user-centered design process helped us redesign the FSP to better meet the needs of its users—new faculty. It also provided us with a clearer idea of what constitutes a “quality system” for supporting new Pathways faculty. For example, we learned that the program must be efficient for stakeholders; this meant that the program could not require significant resources to engage in preparation and support structures. Faculty would not engage in preparation and support structures if they knew that doing so would require more time and energy than they could feasibly spend, particularly those with extreme time constraints (such as adjunct faculty). Thus, engaging with new faculty and understanding their needs and work processes intimately proved monumental for beginning to define a quality FSP.

Our learning about what quality looks like for our system of faculty support needed to be organized in a coherent manner, and in such a way that allowed us to measure the quality of our program across a network of 50 colleges. To do this, we turned to the Institute for Healthcare Improvement’s (IHI) system-level measures approach. Developed to assess the quality of health care and determine improvement priorities across a complex network of hospitals, IHI devised a suite of system-level outcome measures organized around a set of quality dimensions. This measurement system served as the basis of a reviewing of the quality of health care the network of hospitals provided, as well as targeting improvement efforts (Martin, Nelson, Lloyd, & Nolan, 2007).

Based on this model, the Design Team developed a set of system-level measures (13 in total) organized around the five dimensions characterizing a quality FSP (see Appendix A):

The program must be effective—The success of the program must be tied to improvements in faculty members’ knowledge, skills, and dispositions as well as student outcomes. One measure in this dimension is student success rate within the Pathways program.

The program must be efficient—Valuable resources, such as time and money, must not be wasted when engaging in preparation and support. This is a dimension that applies to all stakeholders, from new faculty to Faculty Mentors to the FSP team. A measure in this dimension is time spent engaging with online resources in the midst of a term while teaching the Pathways; faculty members often have time constraints, and designing resources that are time-intensive have proven overly burdensome.

The program must be responsive to the needs of new faculty—The resources that are provided must be designed in such a way that serve faculty members’ needs when those needs emerge, both prior to and during their teaching of the Pathways. A measure in this dimension is the new faculty members’ rating of the helpfulness of the support they receive.

The program must support a sense of community—New faculty are comfortable engaging with other new faculty, Faculty Mentors, and the FSP team. A measure in this dimension is the level of communication between new faculty and their Faculty Mentors.

The program must be faculty-centered and faculty-owned—Faculty are involved in—and willingly support—the continuous improvement of the FSP. A measure in this dimension is the response rate to FSP surveys.

Collected from a variety of sources, including surveys, online analytics, and student outcomes, the 13 system-level measures constitute the FSP measurement system. The team facilitates quarterly reviews of the program using a dashboard consisting of these system-level measures (see Appendix B for an illustration of the use of the system-level measures). Moving forward, the FSP team will review the system-level dashboard for the entire network of Pathways colleges to assess the performance of the program and to determine improvement priorities. Meanwhile, Faculty Mentors and faculty leads will review a system-level dashboard for new faculty that they are primarily responsible for supporting; they will then discuss potential ideas to test based on what they noticed about their dashboards and what they want to improve. With the FSP team’s guidance and coaching, Faculty Mentors and faculty leads will test those ideas after the program review and throughout the term.

Organizing ourselves in this way allows us to systematically accelerate our learning about how best to support new Pathways faculty. Using a common measurement system aligned with a cohesive vision of quality, and having routines, protocols, and processes for reviewing the performance of the FSP, allows us to continuously examine and improve the program so that it can keep getting better at meeting the needs of its primary users: new Pathways faculty.

Discussion

User-Centered Design and the Principles of Improvement Science

User-centered design is an approach that reflects and embodies principles of improvement science. Specifically, this case of the implementation of user-centered design in the service of developing high-quality professional development for Pathways faculty illustrates four of the six core principles of improvement science.

Seeing our system

Using a user-centered design process in the FSP redesign allowed us to see our system—the FSP—from the vantage point of our users. It also illuminated what we learned from our empathizing, definition, and ideation work, highlighting strengths, weaknesses, and gaps in the processes, products, and roles that constituted the system. We used system maps to visualize our “old” system for faculty support and locate its strengths, weaknesses, and gaps. We considered design changes based on what we had learned and used the system map representations to see how those design ideas could change the system. In this way, using system maps to see the system enabled us to organize and prioritize changes to prototype and test as well as provided a means for hypothesizing the impacts of changes to related processes and outcomes. The development and use of system maps also supported the articulation of a clear definition of quality, which we incorporated into a revised system map that represents the ideal system for faculty support, which is both informed by—and informs—our design products.

We cannot improve at scale what we cannot measure

The absence of a comprehensive measurement system for the initial FSP both hid weaknesses and failures in the system and made identifying targets for improvement and seeing the effects of any improvement efforts impossible. As a result, we felt it necessary to undertake a resource-intensive redesign process, which resulted in not only a better designed more user-centered program but also a system of measurement that drives ongoing and continuous improvement of the program. The system-level measures allow us to systematically and regularly assess the quality of the program, thus enabling us to use “small, iterative cycles of improvement” targeted to specific aspects of our system. We must be able to locate where improvement is needed based on the evidence from our measurement system; thus, the system-level measures are a critical design consideration.

Make the work problem-specific and user-centered

Although the process of developing the initial FSP involved faculty, it did not do so in a systematic and structured manner. Furthermore, we made assumptions about faculty’s needs and work processes that proved inaccurate. User-centered design provided us with a method and set of tools to be able to understand our users and their work processes in a way that informed the redesign of all aspects of the FSP. Having learned this lesson, one dimension of quality for the FSP is that the program is continuously managed and improved upon in a way that is faculty-centered. In addition, the user-centered design process is centered around the generation of and focusing on and scoping of problems. In this way, the process compels attention to specific problems to be solved.

Accelerate improvements using networked improvement communities

In the redesign process, we interviewed faculty who engaged in various support structures, provided both by the FSP team as well as their colleges. Faculty almost always received some form of local support; however, what is available to faculty varies tremendously from college to college. While some colleges provide minimal opportunities and resources for preparation and instructional support, others run elaborate programs that included week-long preparation workshops and regular meetings during the faculty members’ first year of teaching. To help colleges build their internal capacity for faculty development for the Pathways and to provide some basis of consistency in the quality of the content and structure of the professional development experiences that faculty were experiencing, we concluded that we should design and make available resources for colleges to develop and provide consistent localized professional development for their new Pathways faculty. Furthermore, as colleges scale, local workshops become necessary to prepare faculty to teach the Pathways, as resources for sending groups of faculty to the National Forum are typically unavailable.

For these reasons, we felt it critical to learn from the Pathways’ network of colleges about the ways that faculty are supported in various contexts, and from those learnings, develop a set of resources for local faculty support structures. As described above, this set of resources is now known as the Pathways Local Faculty Support Handbook. It includes descriptions of a local preparation workshop, ongoing faculty team meetings, as well as protocols for both workshops and meetings—in addition to recommendations for the use of the handbook itself. This is a living set of resources that are regularly improved upon based on what we learned from colleges implementing systems of faculty support locally. Protocols and sections of the handbook describing various structures are revised routinely to reflect our most up-to-date learning. In addition, those responsible for developing systems of local support are involved in continuous improvement of their programs (and of the FSP more broadly) using the system-level measures and quality dimensions.

Flexibility in the Application of Improvement Science

Improvement science offers a flexible suite of tools and methods for the design and continuous improvement of systems. In the case of the FSP redesign, core tools included user-centered design processes, system maps, personas, and system-level measures. These tools were often used to meet emergent needs and were often brought back at different stages of the process. We underscore the flexibility of improvement science to respond to suggestions that its emphasis on disciplined inquiry necessitates a lock-step approach to its implementation. Although the six core principles of improvement science form the foundation of work organized as continuous improvement, there are varied approaches and tools for animating those principles. Indeed, differences in timescales of improvement activity, system complexity, and the diversity of contexts represented in a network shape choices about tools and methods. This is particularly important in the application of continuous improvement in education. Educational systems are multilayered institutions that operate on multiple timescales and that vary tremendously in their organization, structure, and context. A set of tools that might be well suited for improvement of instructional practice, for example, may not be the same as those well suited for improving enrollment and placement processes. Those seeking to undertake continuous improvement should be well advised of a variety of tools and their specific uses and purposes.

Implications and Conclusions

By utilizing improvement science tools and methods, in particular user-centered design and system-level measures, we have designed a system of professional development for community college developmental mathematics faculty that is structured to be effective, efficient, responsive, centered on community, as well as faculty-centered and faculty-owned. The changes resulting from the redesign process are not primarily directed at content (i.e., the professional development learning goals, the mathematical content, the pedagogical content, etc.); rather, they focus on the processes and structures of the program that affect faculty engagement and access. One key takeaway from the initial launch and the developmental evaluation of that initial pilot is that, in a nutshell, you may build it, but they may not (or perhaps cannot) come. Because the initial development process was so focused on high-quality learning experiences and engaging faculty in the content, little attention was paid to whether and how faculty accessed the offerings of the program and the factors that affected the degree of their engagement. However, considerations of access and engagement are critical, especially in the context of professional development for faculty in a nationwide network, offered outside of faculty members’ colleges, and undertaken on a voluntary basis. The user-centered design process allowed us to develop a system that brings those dimensions to the forefront and is structured for continuous improvement.

Through this process, we have also gained insights about professional learning in the community college context that suggest implications for the design of faculty professional development. As LeMahieu et al. (1995) note, professional development activities should be “embedded in the routine organization of the school day and year and viewed as an integral part of the life of the school” and represent a “mutual obligation” on the part of the system to provide and the individual to partake (p. 21). However, the structure of teacher’s work in community colleges does not typically include regular or routine opportunities for professional learning—colleges underfund faculty development, faculty do not often expect or seek high-quality professional development at their institutions, and many faculty, especially adjuncts, have little time to devote to development activities outside of their contracted teaching duties. Also, prevailing models of professional development in community colleges reflect an orientation toward professional learning that favors acquisition of “best practices” through workshops or webinars. Seeing the need for professional development that is sustained and ongoing requires a cultural shift in understanding professional learning as long term and incremental, and so, time is required for meaningful and sustainable change—on the part of both faculty and administration.

Thus, professional development for community college faculty needs to be designed with the specific work processes and structures of community college teaching in mind to motivate sustained engagement. For example, even where regularly occurring formal professional development is unattainable, leveraging local colleagueship to build more formal and routine structures for team collaboration and planning has been possible in our colleges. In addition, we are testing the hypothesis that providing multiple modalities for accessing content about key topics and issues in Pathways teaching, such as an online course, just-in-time support online and from mentors, scheduled webinars, and so on, will fit more effectively into the work lives of faculty and therefore improve engagement and value. Critical for improving all faculty members’ access to support is the consistent availability of high-quality local college-based professional development. We have found that college faculty and administration are willing to build local professional development structures and activities for the Pathways when provided with support from the Carnegie Foundation team around design, content, and facilitation. The redesign reflects a commitment to pursuing this strategy more broadly as a means of embedding professional learning opportunities in the contexts of instructors’ work. Finally, the redesign of the FSP emphasizes ongoing relationships in a community of faculty—new and experienced—that serves as a basis for continuing engagement about teaching, coupled with a series of more formal structures (e.g., workshops, online content) that anchor that sustained engagement in deeper dives into content and pedagogy.

Through our use of improvement tools and methods, we are testing the hypotheses that community building, anchoring relationships with mentors, availability of multiple flexible modalities for accessing information and resources, and creating local systems of support will reduce barriers for faculty to engage in the FSP and will result in a more effective learning experience. The measurement system that we have put in place will allow us to see whether and how those hypotheses are accurate. That is the great value of improvement science for our case and, by extension, for others concerned with teacher learning—It is a means by which a system for professional learning itself can learn and improve in response to the learners it serves.

Footnotes

Appendix A

Table A1.

Quality Dimensions, Outcomes, and Potential Measures of Faculty Support.

Dimension	Description	System-level measures	Data collection mechanism	Goal or “aim”
Effective	NF value and implement well the Pathways curricula	● Success rates	● Student success data (1/year)	● 100% of NF have comparable success rates
	NF value and implement well the Pathways curricula	● Retention likeliness scale	● FSP “check-in survey” data (4/year)	● 100% of faculty rate 5 or 6 on retention scale
		● NF confidence rating	● National Forum or minicourse or local prep workshop survey (+ end of term survey)	● 100% of faculty rate 4-6 on preparation survey items \| 100% of faculty rate 5-6 on end-of-term preparation survey items
Efficient	Stakeholders do not waste time and resources on finding and providing support	● Time NF spend on support activities	● Engagement data in Google Analytics and Zaption (ongoing)	● 100% of NF spend no more than 3 hr/month on online resources while teaching
		● FM time spent on mentoring per month	● FM invoices (1/month)	● FM spend no more than 8 hr per month on mentoring
Responsive	Stakeholders receive the support they need when they need it	● NF perception of support timeliness	● FSP “check-in survey” data (4/year)	● 100% NF rate 5 or 6 on scale of support timeliness
		● Perception of time spent on preparation	● FSP check-in survey data	● 100% of NF score 4 out of 6 or better on “time was well spent” item
		● NF perception of helpfulness of support	● FSP check-in survey data (4/year)	● 100% NF rate 5 or 6 on scale of support helpfulness
		● NF perception of helpfulness of support	● Interviews and focus groups	● 100% NF rate 5 or 6 on scale of support helpfulness
Sense of community	Stakeholders become model and active members of the Pathways NIC	● Faculty Lead NF community item● Online engagement in minicourse or CarnegieHub● NIC engagement likeliness scale	● Faculty Lead survey and FM roll call● FSP check-in survey data (4/year)	● 100% communicating participating with Faculty Lead or FM or engaging in online resources
Sense of community				● 40% NF rate 5 or 6 on NIC participation likeliness scale
				● 40% NF rate 5 or 6 on NIC participation likeliness scale
Faculty-centered and faculty-owned	Faculty support design, development, and improvement priorities are driven by faculty members	● Online engagement in minicourse or CarnegieHub	● Google, Canvas, and Zaption analytics	● 90% engage in online resources
Faculty-centered and faculty-owned		● Number of faculty on FSP Design Team	● Observation (ongoing)	● Six NF are active FSP Design Team members
		● Percent of NF teaching classified as “communicating & participating”	● FM roll call	● 100% NF teaching classified as “communicating & participating”

Note. What is quality faculty support, and how do we measure whether we have reached this definition of quality? NF = New faculty; FSP = Faculty Support Program; FM = Faculty Mentor; NIC = Networked Improvement Community.

Appendix B

To illustrate the use of these quality dimensions and system-level measures, below is an assessment of the quality of the 2015 Pathways National Forum workshop for new faculty. This 2-day summer workshop, which launches the Faculty Support Program for a cohort of new faculty, provided an introduction to the core concepts in Pathways instruction and intensive, hands-on preparation for teaching. As the starting point of the program, occurring prior to the start of the term, not all data sources were available. Bolded system-level measures in Table B1 indicate measures that were directly used in this assessment. The measurement system is designed to include leading and lagging indicators, with indicators occurring at different timescales (e.g., once a term, 3 times per year, etc.). As such, the system-level dashboard is a dynamic representation of the health of the program, reflecting the appropriate leading and lagging indicators.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by the National Science Foundation, Directorate for Education and Human Resources.

Notes

Author Biographies

Ann R. Edwards is the director of Learning and Teaching at the Carnegie Foundation for the Advancement of Teaching. She directs curriculum, instruction, and professional development in the Community College Pathways, a systemic reform initiative aimed at improving developmental mathematics outcomes nationwide. Her research interests include mathematics teacher learning and professional development and issues of equity in mathematics education.

Carlos Sandoval is the research coordinator for Advancing Quality Teaching in the Community College Pathways initiative at the Carnegie Foundation for the Advancement of Teaching. He is primarily responsible for improving support systems for Pathways faculty.

Haley McNamara is currently the postbaccalaureate fellow for Advancing Quality Teaching in the Community College Pathways initiative at the Carnegie Foundation for the Advancement of Teaching. Her academic interests lie at the intersection of education and psychology, specifically how to translate research into meaningful, practical classroom interventions.

References

Bailey

Jeong

D. W.

Cho

S. W.

(2010). Referral, enrollment, and completion in developmental education sequences in community colleges. Economics of Education Review, 29(2), 255-270.

Bailey

Smith Jaggars

Jenkins

(2015). Redesigning America’s community colleges. Cambridge, MA: Harvard University Press.

Bransford

J. D.

Brown

A. L.

Cocking

R. R.

(Eds.). (1999). How people learn: Brain, mind, experience, and school. Washington, DC: National Academies Press.

Brown

(2009). Change by design. New York, NY: Harper Business.

Bryk

Gomez

Grunow

LeMahieu

(2015). Learning to improve: How America’s schools can get better at getting better. Cambridge, MA: Harvard Education Press.

Center for Community College Student Engagement. (2014). Contingent commitments: Bringing part-time faculty into focus (A special report from the Center for Community College Student Engagement). Austin: Program in Higher Education Leadership, The University of Texas at Austin.

Desimone

(2009). Improving impact studies of teachers’ professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181-199. doi:10.3102/0013189x08331140

Dweck

C. S.

(2006). Mindset. New York, NY: Random House.

Dweck

C. S.

Walton

Cohen

(2014). Academic tenacity: Mindsets and skills that promote long-term learning. Seattle, WA: Bill & Melinda Gates Foundation.

10.

Fishman

Marx

Best

Tal

(2003). Linking teacher and student learning to improve professional development in systemic reform. Teaching and Teacher Education, 19(6), 643-658. doi:10.1016/s0742-051x(03)00059-3

11.

Fullan

M. G.

Miles

M. B.

(1992). Getting reform right: What works and what doesn’t. Phi Delta Kappan, 73(10), 745-752.

12.

Garet

Porter

Desimone

Birman

Yoon

(2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915-945. doi:10.3102/00028312038004915

13.

Grubb

W. N.

(1999). Honored but invisible: An inside look at teaching in community colleges. New York, NY: Routledge.

14.

Grubb

W. N.

Gabriner

(2013). Basic skills education in community colleges: Inside and outside of classrooms. New York, NY: Routledge.

15.

Guskey

(2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press.

16.

Guskey

(2002). Professional development and teacher change. Teachers and Teaching, 8(3), 381-391. doi:10.1080/135406002100000512

17.

Hasso Plattner Institute of Design. (2012). An introduction to design thinking process guide. Stanford, CA: Author.

18.

Hasso Plattner Institute of Design. (2013). Bootcamp Bootleg. Stanford, CA: Author.

19.

Hawley

W. D.

Valli

(2007). Design principles for learner-centered professional development. In Hawley

W. D.

(Ed.), The keys to effective schools: Educational reform as continuous improvement (pp. 117-137). Chicago, IL: National Education Association.

20.

Hiebert

Grouws

(2007). The effect of classroom mathematics Teaching on students’ learning. In Lester

(Ed.), Second handbook of research on mathematics teaching and learning (pp. 371-404). Charlotte, NC: Information Age.

21.

Huber

(2008). The promise of faculty inquiry for teaching and learning basic skills. Stanford, CA: Strengthening Pre-Collegiate Education in Community Colleges.

22.

Hulleman

C. S.

Harackiewicz

J. M.

(2009). Promoting interest and performance in high school science classes. Science, 326, 1410-1412.

23.

Hunzicker

(2010). Effective professional development for teachers: A checklist. Professional Development in Education, 37(2), 177-179. doi:10.1080/19415257.2010.523955

24.

IDEO. (2015a). The field guide to human-centered design. Canada: IDEO.org. Retrieved from http://www.designkit.org/resources/

25.

IDEO. (2015b). Our approach: Design thinking. Canada: IDEO.org. Retrieved from http://www.ideo.com/about/

26.

Kember

Gow

(1994). Orientations to teaching and their effect on the quality of student learning. The Journal of Higher Education, 65(1), 58-74. doi:10.2307/2943877

27.

LeMahieu

P. G.

Roy

P. A.

Foss

H. K.

(1995). The characteristics of effective professional development. Newark: Delaware Education Research and Development Center, University of Delaware.

28.

LeMahieu

P.G.

Edwards

A. R.

Gomez

L. M.

(2015). At the nexus of improvement science and teaching: Introduction to a special section of the Journal of Teacher Education. Journal of Teacher Education, 66(5), 446-449. doi:10.1177/0022487115602125

29.

LUMA Institute. (2012). Innovating for people: Handbook of human-centered design methods. Pittsburgh, PA: Author.

30.

Martin

L. A.

Nelson

E. C.

Lloyd

R. C.

Nolan

T. W.

(2007). Whole system measures (IHI Innovation Series White Paper). Cambridge, MA: Institute for Healthcare Improvement.

31.

Mesa

(2011). Similarities and differences in classroom interaction between remedial and college mathematics courses in a community college. Journal on Excellence in College Teaching, 22(4), 21-55.

32.

Norman

D. A.

(1988). The Psychology of Everyday Things. New York, NY: Basic Books.

33.

Patton

M. Q.

(2006). Evaluation for the way we work. The Nonprofit Quarterly, 13(1), 28-33.

34.

Silver

White

(2013). Pathways to improvement: Using psychological strategies to help college students master developmental math. Stanford, CA: The Carnegie Foundation for the Advancement of Teaching.

35.

Sowers

Yamada

(2015). Pathways impact report. Stanford, CA: The Carnegie Foundation for the Advancement of Teaching.

36.

Supovitz

Mayer

Kahle

(2000). Promoting inquiry-based instructional practice: The longitudinal impact of professional development in the context of systemic reform. Educational Policy, 14(3), 331-356. doi:10.1177/0895904800014003001

37.

Tinberg

Duffy

Mino

(2007). The scholarship of teaching and learning at the two-year college: Promise and peril. Change: The Magazine of Higher Learning, 39(4), 26-33. doi:10.3200/chng.39.4.26-35

38.

Wood

Wendell

Holtzblatt

(2004). Rapid contextual design: A how-to guide to key techniques for user-centered design. San Francisco, CA: Elsevier.