Leveraging Multiple Theories of Change to Promote Reform: An Examination of the AAU STEM Initiative

Systems Theory

Systems theory examines the interrelationship of various subsystems within organizations and how organizations are interconnected (Birnbaum, 1988). In systems theory, change is most likely to be achieved when all aspects of the system are adjusted in coordinated ways (Kezar, 2013). For example, to change or affect the use of evidence-based teaching practices, professional development alone will not work as classroom practices are also tied to incentive systems, departmental norms, facilities, campus priorities, and student expectations (Austin, 2011; Kezar, 2013).

In addition, AAU adopted an open systems theory of change that explored beyond the impact of the internal college system to other actors and players that affect the campus. Open systems theory emphasizes how change processes are affected by external organizations, groups, and forces (Gumport & Sporn, 1999; Hearn, 1996). Various studies of change have adopted a systems approach, but fewer studies in higher education adopt an open systems theory, particularly as it applies to areas such as teaching and learning, which are more often considered the domain of academic professionals (Kezar, 2013). Open systems theory tends to be applied to issues such as cost containment, Title IX, increasing regulation, and even diversity; these are areas where higher education seems less open to internal levers for change and in need of external pressures or force (Kezar, 2013; Zemsky, 2013). Yet, AAU predicted that in research universities, various external groups shape norms around teaching, including disciplinary societies, national groups and agencies such as the NSF, or disciplinary accreditors (particularly in groups such as engineering) that have been active in shaping teaching.

As noted above, the AAU developed its Framework to guide institutional commitment to facilitate change in undergraduate STEM education. The Framework uses systems theory in describing linked parts of the campuses that need attention to ensure sustainable change, including the type of leadership commitment and resources needed for change, infrastructure in terms of professional development, changes to promotion and tenure processes, alignment of reward structures with teaching, and new facilities, classrooms and laboratories, and development of pedagogical, curricular, and learning goals and objectives. In addition, the open systems approach is evident in the objectives for the initiative that include and operate at multiple levels: (a) project demonstration sites, (b) the AAU network made up of the entire population of AAU membership institutions, and (c) the broader higher education system with national organizations and groups such as disciplinary societies, accreditors, and national reform groups.

Organizational Learning

Another prominent theory of change driving the initiative was organizational learning (Argyris & Schön, 1996). Organizational learning is typically associated with a branch of literature on change that examines the role of cognition and mental processes within change. Organizational learning is the study of whether, how, and under what conditions organizations can be said to have learned (Boyce, 2003; Kezar, 2005). Within this stream of research, organizations are seen to change when individuals learn and collectively reorient the way that they approach conducting work, in this case teaching undergraduate STEM courses (Huber, 1991). This is often seen as an approach to scaling change because of its emphasis on mind-sets that undergird work, which can promote lasting change across a wide swath of the organization (Kezar, 2013).

Organizational learning scholars tend to look at information and data available within organizations; the way that information is shared; facilitative mechanisms for learning, such as teams, networks, or data dashboards; sharing of best practices; and, more generally, looking at the ways that individuals within organizations examine or understand concepts (Kezar, 2005). Studies of organizational learning focus on issues, such as acquiring information and ideas, interpreting data, turning information to knowledge, knowledge recall and memory, and ways to sustain learning by embedding it into the organizational structures (often termed knowledge management; Argyris & Schön, 1996; Garvin, 1993; Huber, 1991). Research from multiple disciplines demonstrates the infrastructure that needs to be in place to support organizational learning (improved data systems, knowledge management systems), best practices for fostering learning (better formats for displaying information to different stakeholders, team practices that help interpret information to create knowledge), and the leadership and culture (creating a culture of trust, leaders supporting risk-taking and innovation) required to ensure that organizational learning occurs (Argyris & Schön, 1996; Kezar, 2005).

The AAU promoted organizational learning in a variety of ways, including having participants adopt practices from each other’s campuses; spreading learning across the network through presentations, webinars, and meetings; and pitching the eight demonstration sites as laboratories to try out new practices that they hoped would inform and be adopted by other campuses. In examining the five main objectives of the AAU initiative, four of them relate to aspects of learning, including developing means for information sharing about best practices in STEM reform, working with institutions and departments to train and professionally develop faculty, supporting project sites to implement the framework, and the development of the framework itself, which could be seen as a learning tool. The Initiative was also very informed by the collection of data. Surveys were conducted to inform each campus of how widespread use of evidence-based teaching practices was on their campus. Another major goal of the project was the development of measures that could help institutions assess their progress and learn how to improve implementation using data to inform their actions. And a major part of the Initiative that emerged over time was the use of data analytics to follow students and better understand their patterns of performance to inform improvements in curriculum. Most of these practices could be characterized as actions that fit into organizational learning.

Network Theory

Another important theory for scaling change is the utilization of networks and diffusion theories of change (Rogers, 2003). Many empirical studies demonstrate that changes in behaviors and mind-sets have been a result of networked diffusion models in which innovators utilized new practices and had some success, and then others became aware of the practices and utilized them as well (Rogers, 2003; Tenkasi & Chesmore, 2003). Networks create the informational channels for change ideas to flow, for a community to innovate in safe spaces, and intellectual capital and knowledge about how to implement change (Tsai, 2000). Webs of relationships are often the chief determinant of how well and quickly change efforts take hold, diffuse, and are sustained (Daly, 2010).

Researchers have identified several key ways that social networks lead to change. First, social networks offer a set of mechanisms that enable change—through communication systems, knowledge transfer, alteration of schema or mind-set, shaping of attitudes, increasing of problem solving, and accountability (Ahuja, 2000; Borgatti & Foster, 2003; Kraatz, 1998; McGrath & Krackhardt, 2003; Szulanski, 1996; Wasserman & Faust, 1994). Second, two outcomes of social networks have been related to change—learning and social capital (Borgatti & Foster, 2003; Burt, 2000; Kilduff & Tsai, 2003; Tenkasi & Chesmore, 2003). Many researchers have found a strong linkage between learning and social networks, and learning has been strongly linked to changes in behavior (Tenkasi & Chesmore, 2003). Networks also provide social capital that facilitates the change process (Burt, 2000), varying from knowledge about how the organization works, to influence within the organization, to an organization’s finances. Third, change often involves risk-taking that can be less problematic if it is done collectively rather than individually (Valente, 1995). If one knows many of their peers are going to engage in the same behavior, then one is more likely to also engage in this behavior (Rogers, 2003; Valente, 1995).

The AAU STEM Initiative utilized networks to build relationships, connect various aspects of the system they knew were necessary for change, disseminate information, facilitate learning, and eventually scale change. The AAU STEM Initiative was specifically designed as an effort that would connect eight project demonstration sites within a network, and these sites would meet with the larger set of AAU institutions to create a broader network to help foster change. Indeed, this broader set of institutions was named the AAU STEM Network. Over time, the AAU also created several subnetworks of department chairs, faculty within various STEM disciplines, and directors of centers for teaching and learning, for example. They also networked with other groups, such as the HHMI and the NSF and other networks such as the Cottrell Scholars, Bayview Alliance, and Center for the Integration of Research, Teaching and Learning (CIRTL).

IT

IT describes the impact of broader forces and organizations outside college campuses themselves on change, including accreditation, disciplinary societies, state policy makers, and national agencies such as the NSF—also known as the organizational field (Powell & DiMaggio, 1991). Examples include national associations pushing for a more similar general education curriculum, accreditors pushing for assessment of student learning outcomes, or presidents altering their mission to be more research-oriented in pursuit of prestige to be more like elite institutions (Boyce, 2003). In addition to the organizational field, the societal field also influences change on campuses. The societal field encompasses pressures that are further afield (such as economic changes or political regulations) yet still shape campuses and members of organizational fields such as accreditors that have closer contact with campuses and often help deliver and temper societal forces. Both societal and organizational fields affect campuses, but in varying degrees and depending on the power of the fields in any given time period. The power of fields is dynamic and changes over time.

IT leverages influence as a way to motivate change. This phenomenon often plays out through isomorphism, where institutions mimic each other; in particular, less prestigious institutions often emulate more prestigious ones (W. R. Scott, 2008; Taylor & Morphew, 2010). But influence can take many forms, such as more normative approaches where certain values or approaches are legitimized (W. R. Scott, 2008). In these normative approaches, shaping norms, language, discourse, and logics that underlie organizations are often a focus of influence. Therefore, change is a result of a new schema or set of norms, embedded in language, transferred through discourse, and entrenched in institutional structures over time through a system of institutional logics—a driving rationale for how things should operate (J. W. Meyer & Rowan, 1977). Legitimacy is also critical in that only certain players in the field (generally the most prestigious ones) are seen as being able to drive new norms, standards, and logic (W. R. Scott, 2008).

AAU realized that as a prestige organization, it may have the ability to alter norms and discourse of its own institutions and perhaps more broadly influence other campuses, as well. It developed a new set of values—that AAU institutions would be as excellent in teaching as in research—and worked to influence AAU campuses through dissemination and articulation of this new value system. In addition, it believed that based on its prestige and legitimacy, it could obtain the attention of leaders and help them shift priorities. AAU also thought that it could help align various messages from different organizations in the organizational field, which could also build pressure for changes.

This article explores specific ways in which these theories were enacted, focusing on how particular strategies could reflect multiple theoretical approaches to change. We also examine the efficacy of using multiple theories of change and their potential synergies and conflicts.

Data Collection

Data Analysis

The qualitative data (documents, interviews, and observation) were analyzed using HyperResearch (a qualitative software program) that helps manage and analyze large amounts of qualitative data and eases the coding process. Data analysis used Kincheloe and Berry’s (2004) process of bricolage in qualitative analysis. This approach involves utilizing multiple theoretical perspectives to best understand a complex phenomenon, taking a multi- or interdisciplinary approach. This approach avoids reductionism by not taking single theoretical or narrow approaches to interpretation. Instead, various theories are overlaid to interpret data both separately and then simultaneously. For example, the researchers first examined a strategy such as annual meetings and how it could be understood through concepts and themes as part of organizational learning or networking separately. The next step would be to look at this strategy with all the theories together to see how certain themes may run across theories but have different meanings based on the theory. Social capital, for example, plays different roles in each of these theories, so its full capacity is seen when examined analytically by multiple theories. Bricolage acknowledges the complexities of social life and the need to bring in many different interpretive frameworks to best understand complex social processes. Analyzing processes such as change lend themselves to bricolage as the phenomena do not fit neatly into a psychological, sociological, or organizational disciplinary box. Given our emphasis on multiple change theories, bricolage also allowed us to look at change from these multifaceted perspectives. We used the different lenses of IT, systems theory, organizational learning, and networking to analyze the data, all of which address change from different vantage points and frame the AAU STEM Initiative from different perspectives. We also examined the interplay of different theories as we interpreted the data, looking at how theories complemented or contradicted one another.

We utilized situational coding, part of grounded theory, to examine different change theories and combine data from what we observed, from interviewees, and from documents to examine different change theories and what was occurring (Clarke, Friese, & Washburn, 2017). In situational coding, researchers identify the situation of interest—for us situations when multiple theories of change were deployed—then analyze those situations for nuance around that way the phenomenon emerged, including key actors and key characteristics or features. In the situational analysis, the activities, synergies, and challenges emerged that we describe in our findings.

Trustworthiness and Limitations

This study design draws on the most valid and systemic approaches to studying scale of change, emphasizing interviews and observations that help understand the key underlying mechanisms, such as influence and incentives as well as barriers to scale (Dede, 2006). Several approaches to trustworthiness were utilized. First, observations and document analysis were carried out for over a year prior to the interviews to ensure that the researchers were well aware of the context and initiative prior to interviews to maximize interviews being as valid as possible. Second, observations, documents, and interview data were compared to provide multiple data points in support of any finding, thus triangulating findings. Third, multiple researchers coded data, ensuring that the themes were identified by multiple individuals. Fourth, we convened and received feedback from an advisory board on our design, observation and interview protocols, sample, and interpretations of the data. In terms of limitations, we had limited physical exposure to project sites for impromptu conversations with campus participants. We selected a variety of individuals to speak with who had different positions and involvement but skewed toward those more involved with initiative.

Embedding Multiple Theories Into Project Activities

We review three project activities (the initial Request for Proposals [RFP], site visits, and annual meetings) that reflect the use of multiple theories of change. It is not that these activities inherently reflect these various change approaches, but rather that AAU deployed them in a way that imbued them with these different theories of change. We documented 11 activities that utilized multiple theories, but due to space purposes, we review three in this section. These three were chosen because they are commonly used strategies and so have practical advantages, were representative of all project activities in terms of their reflection of multiple theories of change, and provide the clearest examples of how multiple theories were deployed. The other eight are listed in Table A1 (see the appendix).

Synergy of Multiple Theories of Change

Another way that multiple theories of change operated was in terms of synergy across various theories and how they complemented or supported another area. AAU did not consider these theories as operating separately but typically in service of each other, and they tried to capitalize on synergies. We highlight two of these synergies; space does not allow for a full review of all synergies, but the ones we describe below exemplify the way that theories interacted in support of each other. While we focus on dyadic relationships here, theories were also connected in triads and quadruple connection points; however, dyadic relationships were most common.

Conflicts and Negotiation of Multiple Theories of Change

While different theories of change can be synergistic, they can also come into conflict. This is one of the potential pitfalls of multiple theories of change—that the deployment of one approach may negatively shape or affect another strategy in play. Therefore, it is important to examine the ways that different theories of change can interact and affect each other. While AAU was fortunate to experience few instances of such conflict, we did see one major example in which the influence/IT strategies (particularly harnessing competition) appeared to negatively affect learning. AAU was aware of how their influence and power as an organization might suppress the openness and vulnerability needed for learning. They worried about whether campus participants would be honest and forthright about struggles; indeed, that feared lack of openness did occur.

AAU staff understood that part of the AAU culture is a sense of competition between campuses to “outdo each other.” The AAU leveraged this competition to motivate change, as a leadership team member from AAU described, “we know our campuses are highly competitive, want to be the best. They have this inclination to compare so we can use that.” Many interviewees noted that the competition between AAU campuses did indeed drive change on their campuses, with provosts pushing their project teams because they “heard another AAU institution had created active learning classrooms or new technologies,” for example. A faculty member at an AAU project site referred to this competition as a catalyst for change:

Competition is rooted in our institutions. After the AAU meeting in Laguna, I got this call—they heard what was happening at [two other institutions] and they wanted to know how we compared—what we were doing. It just throws us into motion.

While competition did spur some changes on AAU campuses, it also worked against efforts to spread changes and have learning occur across campuses. Each of the project sites developed an individual project and approach to change that they deeply believed in. Because they had very strong beliefs about their project being best suited for their campus, this may have created a reluctance to learn from the other sites or adopt practices. The sense of competition exacerbated their concerns about learning from and borrowing from other sites. Interviewees noted that if they were to borrow ideas from another site, it would reinforce the value of the other campus’ work; in the competitive culture set up between the AAU campuses, this was not desirable. One faculty member described how competition may have affected learning and adoption of new practices at the project sites: “I think we all just came in strongly believing in our ideas and maybe we felt borrowing ideas from the other campuses would somehow support them and we are competitive with each other.” This idea of competition affecting learning was referenced by an administrator: “taking an idea from another campus is to suggest you do not have equally good ideas on your own campus—this sort of one-upsmanship–well it does not breed uptake of competitors’ ideas.”

Furthermore, at several of the AAU strategic leadership planning meetings, the Initiative leaders described their concern about campuses’ sense of competition preventing them from problem solving together, learning, and sharing. They worried that campuses were missing out on learning within a broader network because of competitive behaviors witnessed at various gatherings, meetings, and webinars/phone conferences. An AAU staff member’s reflection on competition impeding learning in these settings captured this sentiment:

If they will not share information with each other [this was a reluctance to share survey and benchmark data], they are missing out on the whole idea of us connecting and networking them to learn from each other. We cannot hope to have learning when they compete.