Mixed-Methods Inquiry in Public Administration: The Interaction of Theory,Methodology,and Praxis

Abstract

Although the methods wars that were entrenched in most of the social science disciplines well into the 1990s have purportedly ceased, methodological diversity remains rare in public administration. We suggest that the productive interaction between methods, theory, and praxis in the field of public administration requires further methodological integration. Through a review of articles published in Review of Public Personnel Administration (ROPPA) since 2011, we show that the promise of mixed-methods design has not fully materialized. We explore avenues for implementing mixed-methods research designs that are particularly appropriate for examining the behavior and motivations of public service personnel by identifying examples of articles in ROPPA that used a single-method approach but could have usefully leveraged a diversity of methods to answer their primary research question. We then support our case for the use of integrated mixed-methods with a study of the institutionalization of a performance assessment system in an international organization.

Keywords

mixed-methods public administration performance management

Introduction

The methods wars that were entrenched in the 1970s up to the late 1990s in most of the social science disciplines did not spare the field of public administration and public management and are still lingering (Riccucci, 2010). Hard-lines about the nature of knowledge, the appropriate form of empirical inquiries remain particularly vivid today. Although some scholars have favored qualitative and historical approaches (e.g., Kaufman, 1960; Riccucci, 1995; Stivers, 2000), quantitative methodologies have remained dominant in public administration. To a large extent, enduring differences between perspectives creates intellectual stimulation in scholarly conversation, but there is also scope for exploring possibilities of further integration between different methodological traditions. This warrants a balanced and honest discussion, as called for in this symposium.

In the foreword to this special issue, Sanjay Pandey highlights the need for enhancing the discipline’s appreciation of the interplay between theory and method, including the need to be more reflexive about how the domains of theory and methods interact. In this article, in addition to the imperative of theory-development (episteme), and of thoughtful methodological design (techne), we are also concerned with the third approach to knowledge accumulation in the social sciences—practiced as a source of practical and context-dependent knowledge which is action-oriented (phronesis). The extent to which the scholarly field of public administration’s focus should be on informing practice or theoretical development remains up for debate. In this article, we do not refute the idea that the field of public administration benefits from theoretical advancements that may not immediately provide direct input to practice (e.g., Frederickson & Smith, 2003; Raadschelders, 2011; Riccucci, 2010). In the spirit of Lewin—who famously argued that “there is nothing more practical than a good theory” (Lewin, 1952, p. 169)—we take the three imperatives of episteme, techne, and phronesis as equally necessary. Adopting all three elements may require a shift in thinking about what constitutes rigorous evidence and how to obtain it.

The multiplicity of views regarding the goal and nature of social science in general, and public administration in particular, reflects diverse opinions about the relative strengths of different methodological approaches. Although the search for universally valid rules and principles is typically assumed to entail the application of hypothetico-deductive methods inspired from the natural sciences, an action-oriented social science highlights the importance of first framing the questions appropriately, and then considering the appropriate methods, as a fundamental part of the research endeavor. Action-oriented research is designed to produce knowledge that effectively informs public deliberations and praxis. Given the goal of informing practice, methodological considerations should be subservient to the goal of solving practical problems and generating useful evidence for action (Flyvbjerg, 2002; Harmon, 1981; Rorty, 1982). In this article, we explore how the discipline can embrace mixed-methods inquiry to support more phronesic-oriented knowledge creation.

The basic tenet of action-oriented social sciences is to answer questions that are salient to practitioners with the appropriate set of methods. One specific realm where theory, methodology, and praxis need to be particularly intertwined, and where dialogue among the three ought to take place, lies in the framing of research questions and the development of research methods to answer them. On this point, three lines of consensus seem to have emerged over the past decade. The first insists on letting the research questions guide the appropriate research design, and not the other way around (Donaldson, Christie, & Mark, 2015; Newcomer, Hatry, & Wholey, 2015).

The second line of consensus stems from the realization that many important targets of research in the field of public administration and public management are inherently complex, and require developing a granular understanding of the socioeconomic, institutional, and cultural systems within which public servants are embedded to explain the key phenomena of interest, for example, public sector motivation, effects of specific human resource management strategies, and management of performance (Riccucci, 2010). Reflecting the need to explore complex contexts, no single methodological strategy taken alone is likely to fulfill all desired knowledge development needs, and satisfactorily provide in-depth understandings of organizations and people’s behaviors, of institutions, and the intricate dynamics that underlie them. Consequently, research designs that adopt more than one method—or mixed-methods designs—are increasingly accepted as important to contribute to the accretion of practical knowledge.

A third area of agreement favoring increased methodological diversity has emerged from recent dialogues about making administration research more “impactful” and influencing change in public organizations or governance systems. Ensuring that research effectively addresses the users’ needs is giving increased consideration, and this may also entail adopting modes of inquiry that resonate with the users’ epistemological preferences and receptivity to particular methods (Greene, 2007; Head, 2015; Riccucci, 2010). Taken together, these three emerging lines of consensus support both increasing methodological diversity within public administration, and the use of mixed-methods inquiry in public administration research.

Nevertheless, even a mixed-methods study that makes strong methodological choices and strives to integrate findings will inevitably run into challenges that pertain to different philosophical foundations across methodological traditions, such as the meaning of “rigorous” evidence, and the tensions between “rigorous” versus “useful” and “impactful” research (Riccucci, 2010; Vaessen & Raimondo, 2012). These challenges may partly explain why the number of mixed-methods studies in the field remains limited, and have led some to question the idea that multimethods research is necessarily better (e.g., Ahmed & Sil, 2009).

There certainly has been significant progress in integrating increasingly more sophisticated, as well as more diverse, research designs in public administration research. In fact, there have been important methodological innovations in public administration and public management research. For example, experimental and meta-analysis methods are increasingly common in the discipline (e.g., Jilke, Van de Walle, & Kim, 2016; Perry, 2012). However, not as much attention has been given to matching multiple methods to address questions focusing on complex phenomena. The question that underpins this article is thus, “How can we harness the potential for implementing mixed-methods in public administration research?”

This article is organized as follows. First, we discuss the potential of mixed-methods inquiry in public administration and public management research, and how mixed-methods can enhance the dialogue between theory, methodology, and praxis. Second, we examine the fulfilled and missed opportunities for using mixed-methods in research on organizational culture, public sector motivations, and human resources through a review of the articles published in Review of Public Personnel Administration (ROPPA) between 2011 and 2016. Third, we offer a specific example of the integration of mixed-methods in researching the institutionalization performance assessment system in a complex international organization and its effects on the motivations of international public servants.

The Potential for Mixed-Methods Inquiry in Public Administration Research

It is first and foremost important to recognize that the distinction between quantitative and qualitative research methods stems from a conceptual simplification of the research process (Bamberger, 2015). It is indeed possible, and common, to conduct quantitative analyses on data that were collected using qualitative methods, and to conduct some form of qualitative analysis on data that were collected through, for example, sample surveys. Nevertheless, there is a special place in the methodological landscape for research designs that deliberately and purposefully seek to combine quantitative and qualitative approaches in one or more stages of a research process.

Mixed-methods inquiry encompasses a range of approaches that vary along methodological and epistemological continua. At one end of the continuum, many researchers, trained in a particular methodological family, for example, quantitative, or qualitative, may use multiple methods to fill some gaps in measurement. For example, researchers may provide illustrative (perhaps labeled as “anecdotal”) evidence to complement otherwise quantitatively described phenomenon. This can be seen as a minimum threshold for—or some may say paying lip service—to mixed-methods. At the other extreme, some researchers regard mixed-methods inquiry as a mental model, and go beyond the mere integration of qualitative and quantitative methods toward the integration of multiple disciplinary traditions (Bamberger, 2015). For example, Jennifer Greene (2007) defines mixed-methods as,

The planned and intentional incorporation of multiple mental models—with their diverse constituent methodological stances, epistemological understandings, disciplinary perspectives, and habits of mind and experience—into the same inquiry space for purposes of generatively engaging with difference toward better understanding of the phenomena being studied. (p. 30)

The Principles of Mixed-Methods Inquiry

In Greene’s (2007) all-encompassing definition of mixed-methods, the technical methodological mix is secondary to a more fundamental and substantive integration among theoretical traditions, and disciplinary frames of reference. Mixing methods is not seen as merely adding another set of data points, but presents a constitutive and essential part of the way the researcher goes about inquiry. In this respect, Greene builds on substantial work from Teddlie and Tashakkori (1998, 2003, 2009) and their seminal classification of mixed model designs in their ambitious Handbook of Mixed Methods. According to Teddlie and Tashakkori (2003), mixing can take place at three levels: (a) the overall purpose, and questions of the study can be mixed in nature or not; (b) methods for data collection, analysis, and inference can be mixed or not; and (c) the procedure for mixing, which can be concurrent, sequential, or a mere conversion (p. 689).

The prevalence of research using mixed-methods inquiry in public administration remains low. Looking at articles in flagship journals within a particular discipline is a simple and effective way to measure methodological domination. For instance, Pandey, Pansey, Breslin, and Broadus (2016) reviewed public service motivation (PSM) research published between 2008 and 2015, and applied a simple categorization to 106 empirical articles which they found in nine leading journals. They found that the overwhelming majority of articles used quantitative methods, only six articles used qualitative methods exclusively, and five articles used both quantitative and qualitative methods. The authors highlighted that “the qualitative analyses were presented as an adjunct to quantitative analysis with the typical rationale for such use being the need to provide depth and richness” (Pandey et al., 2016, p. 8). Their findings echoed a study by Groeneveld which showed that only 5.9% of the journals in the core PA journals use a mixed-methods design (see Groeneveld, Tummers, Bronkhorst, Ashikali, & van Thiel, 2015).

Matching Research Evidence With Epistemological Preferences and Information Needs of the Users

One of the particularities of public administration and management research is its applicability to directly (or indirectly) inform processes of change within organizations. Although theoretical and methodological values remain paramount, a third imperative is that of generating credible and relevant evidence to inform practitioners. In addition to designing research with intellectual rigor and methodological robustness (including construct, internal and external validity), public administration research can also be designed to uphold the standards of utility and applicability. Practitioners’ receptivity to particular modes of knowledge creation thus also matters. Through mixed-methods inquiry, researchers can offer breadth and depth of understanding, from providing average point estimates on treatment effects, to granular explanation on how and why specific subgroups are impacted differently by a particular human resources strategy. Providing more than one type of evidence is likely to be particularly valuable to diverse audiences. In Figure 1, we illustrate the complexity of the praxis challenge for public administration and management researchers. Our model traces the relationships between methodological decisions and the credibility of evidence, receptivity by various audiences, and the potential for research findings to influence change.

Figure 1.

The praxis challenge: Informing practice with credible evidence.

To simplify, researchers have control over the information they provide, and how effectively they communicate their research findings (Figure 1, Box 1). However, the information they provide may not be viewed as relevant or credible evidence by the practitioners. The trustworthiness of the evidence often lies in the eye of the beholder, and depends on the extent to which the source of the evidence, the reliability of the data and the quality criteria match the receiver’s epistemological preferences (Figure 1, Box 2). In addition, practitioners are embedded in groups and organizations whose cultures and values affect how the use and relevance of different types of evidence is viewed, and how different methodologies are favored. A host of organizational factors come into play to determine the extent to which learning, diversity of views and lenses, risk taking, and acceptance of errors are valued (Figure 1, Box 3.). Finally, although epistemological preferences, expertise and world view indubitably influence practitioners’ thinking (known as “slow thinking”), practical decisions are also often influenced by “fast thinking” or the reliance on heuristics, and other psychological considerations (Figure 1, Box 4). Ultimately, while researchers may not be able to fully know whether or not there is a match between the type of methods they used and the practitioners’ epistemological preferences and values, understanding and catering to practitioners’ information processing is important.

The Added-Value of Mixed-Methods Inquiry for Theoretical Development

In the “Introduction” section, we started delineating the value derived from using mixed-methods in public administration research. Here, we lay out some of the advantages of using mixed-methods, as opposed to single-method inquiry.

Two symmetrical canons of rigorous mixed-methods inquiry are (a) the framing of the research questions should drive decisions on appropriate data collection and analysis; and (b) methods should be appropriately matched to answer each of the research questions. These two precepts form the so-called “platinum standard” (Donaldson et al., 2015; Petticrew & Roberts, 2003; Sanjeev & Craig, 2010). However, in the face of the promotion of particular methods, this standard is not always met, and there is a tendency to mold research questions to fit the requirements of a preferred method. For example, one of the signs of the favoring of quantitative methods in a field manifests itself when research questions are uniformly worded in an inferential or correlational language. The rarity of addressing “why?” and “how?” questions in research articles—that call for in-depth understanding of contextual factors or of explanatory mechanisms, and are more appropriately addressed by qualitative inquiry—is another case in point. Expanding the theoretical boundaries of a given field requires addressing a wide spectrum of potentially important research questions, which include the full gamut of “why” and “how” questions that need to be addressed by a variety of methods.

Second, mixed-methods inquiry enables the researcher to look at a particular problem or research subject through multiple perspectives, thereby avoiding common researcher biases, the oversimplification of complicated issues, and methodological determinism (Bamberger, 2015; Creswell & Clark, 2011; Greene, 2007; Teddlie & Tashakkori, 2003, 2009). Mixed-methods inquiry has been found to add value in a number of research contexts. We want to highlight that other research designs, such as experiments or advanced modeling techniques, share some of these characteristics. Our objective here is not to claim epistemological supremacy for mixed-methods. Rather, we aim to make explicit some of the characteristics of mixed-methods inquiry that make the approach particularly amenable to the study of public administration, including public personnel administration:

Understanding the context-specific embeddedness of the research subject/object into the organization

Triangulating findings and strengthening inferences that are based on incomplete statistical data

Opening the door to innovative research designs, combined with more conventional research approaches

Bringing depth of understanding of common patterns of association and how they play out in particular subgroups or contexts

Making sense of inconsistent or unexpected statistical results

Strengthening the construct validity of concepts measured through surveys

Providing multiple perspectives on a given question or problem and catering to an heterogeneous audience with different epistemological preferences

Understanding behavioral patterns and the underlying factors that can explain them

Bolstering the potential for both analytical and statistical generalizability

In the next section we illustrate some of these benefits by proposing avenues for more systematically integrating mixed-methods inquiry into public personnel administration.

Opportunities for Mixed-Methods Inquiry in Public Personnel Administration

We reviewed articles published in ROPPA between January 2011 and January 2016 to classify the types of research approaches that the authors used. We went through two rounds of analysis. First, we read all the abstracts to filter out the articles that were not empirical. Out of a total of 105 articles reviewed, we identified 18 articles that were either solely theoretical or introductions to special issues. In our second review, we read thoroughly the “Introduction” and “Methods” section of each of the remaining 88 articles to extract information on (a) the specific research questions, (b) the stated objective of the researchers, and (c) the methods used. The latter variable was systematically coded and classified based on the following straightforward descriptions:

Research using solely quantitative inquiry: Articles that sought to assess effects, measure outcomes, or describe pattern of regularities through quantitative measurement and relied solely on quantitative methods, both at the level of data collection, for example, survey data, administrative data, and of data analysis, for example, regressions, principal components analysis, and structural equation modeling.

Research using solely qualitative inquiry: Articles that sought to understand behaviors, provide in-depth knowledge about a particular case, event or group of individuals, and relied solely on qualitative methods design both at the level of data collection, for example, interviews, focus groups, ethnographic observation, historical documents, and of data analysis, for example, qualitative or content analysis of documents and interview transcripts, in-depth case study, qualitative synthesis, rich description, and institutional analysis.

Research using mixed-methods inquiry: Articles that used at least two approaches and relied on two or more sources of data and/or analytical methods, that is, combining quantitative and qualitative methods.

The overwhelming majority of articles (85 out of 88 articles) used a single-method design. Among the single-method articles, the large majority exclusively used quantitative methods inquiry (69 out of 88 articles). The rest, 16 articles, used qualitative methods—most frequently document, historical or institutional analysis—and a minority used interviews and case study designs. In the articles published by ROPPA between 2011 and 2016, only three out of 88 empirical articles can be considered as using a mixed-methods design, meaning that they used a combination of multiple methods, stemming from different research traditions. Here, we succinctly describe these three articles.

Poocharoen and Brillantes (2013) compared merit systems in Asia and in the United States and carefully examined five dimensions of the merit systems, including recruitment criteria, corruption in recruitment and promotion, political influence, relative centralization of recruitment decisions, and merit-protection regimes. The authors used quantitative data from a large-scale survey that was administered by a team of academics in 2011 to civil servants in different ranks in China, Malaysia, South Korea, India, Taiwan, the Philippines, and the United States. A complementary set of qualitative data were generated for two countries, Thailand and the Philippines, stemming from a limited number of interviews with directors from the civil service commission offices, documents, websites, and the authors’ own experience in the civil service of these two countries. The objective of integrating a number of interviews with officials—into an otherwise survey-based research—was to add a level of depth in the conceptualization and understanding of the recruitment and promotion practices, as well as the challenges encountered by managers within different regimes.

Vrangbaek, Petersen, and Hjelmar (2015) systematically reviewed empirical literature on effects of contracting out on employees. They proposed an original systematic review design to synthesize the analyses and results of a large range of studies, irrespective of whether the research designs were primarily quantitative or qualitative. The authors noted,

Our systematic review method is inspired by a method developed by The Evidence for Policy and Practice Information Coordinating Centre (EPPI) centre (Gough, 2004; Gough, Oliver, & Thomas, 2012). This is suitable for systematically reviewing complex literature using various methods and data, including qualitative methods and data. (Vrangbaek et al., 2015, p. 9)

To uphold the standards of rigor of a systematic review, the authors developed a protocol with clear and transparent selection criteria to use in their review, including (a) whether the study was based on an appropriate analytical design compared to the analysis object and the availability of relevant data in the field, and (b) whether the data collection met research standards. In the quantitative studies, the data collection had to make generalization possible, and in qualitative research, the collected data had to be sufficient to support interpretations. They also considered whether the researchers had provided sufficient reasons for the choice of a particular method, and if the conclusions of the analysis were based on suitable methods with which to assess the effects of contracting on employees (Vrangbaek et al., 2015, p. 35).

Finally, the most structured example of mixed-methods found in the ROPPA articles reviewed was in Nesbit, Rimes, Christensen, and Brudney (2016). The authors researched the organizational implications and employees’ responses to being assigned the role and responsibilities of volunteer managers. Three clearly stated research questions underpinned the study: How do employees who have been assigned volunteer management duties view the volunteer management role? How do they view the role of volunteers in their organization? And how do volunteers’ managers’ perceptions of both roles affect their engagement in the volunteer manager role? The authors conducted a staggered/phased mixed-methods research design. They relied on extensive, semistructured in-person interviews with 34 volunteer managers at libraries. They also conducted a structured survey with the same population of interest. The goal of the design was to obtain an in-depth understanding of the participants, as well as quantifiable information, and to ultimately be able to triangulate information through a dialogue with the theoretical framework that guided the research.

Although only three out of the 88 empirical articles published by ROPPA deliberately relied on more than one methodological tradition, the potential for doing so was much wider. To illustrate the potential for mixed-methods inquiry, we selected three articles that were single-method inquiries when published. We identified the questions motivating the research and proposed some avenues for integrating some principles of mixed-methods to address the research topic. This thought experiment is not meant to be a critique of the existing research design, but simply a way to illustrate concretely how mixed-methods inquiry could work using existing research topics and examples.

In Jung and Rainey (2011), the main research question was formulated in a method-specific way, and sought to relate a measure of organization goal ambiguity and the public sector motivations of employees in a range of public agencies. The authors used the U.S. Office of Management and Budget’s Program Assessment Rating Tool (PART) results to develop a measure of goal ambiguity, and conducted a multinomial regression based on two distinct surveys. However, a number of questions were left unanswered by the article, in particular whether some other factors may explain why some programs have clear goals and others have ambiguous objectives that, in turn, may also explain the level of motivations of staff. One possible avenue for overcoming this challenge could have been to integrate a comparative case study approach—such as qualitative comparative analysis (QCA)— to the research design. QCA, which has been used in comparative politics research, is specifically designed to identify necessary causal packages for an effect (or outcome) to take place via a systematic comparison across cases. Utilizing QCA would have allowed the authors to take the contextual environment and inherent characteristics of programs and organizations into account. For example, they could have identified if more politically exposed programs were necessarily more likely to have ambiguous goals.

When assessing whether a particular human resource strategy or organizational factor impacts the motivation of public personnel, it is not sufficient to know that the policy or procedure has proven successful in a particular setting. To combine theory and praxis, research also needs to provide a fuller explanation of the necessary conditions that made this success possible. By identifying enabling factors, along with effect sizes, research findings are likely to be more useful to decision makers. An advantage of using mixed-methods inquiry is the development of middle-range theories that are embedded in particular bounded contexts, but have the potential for transferability to other contexts. To pursue this type of theoretical development with high practical value, it is important to understand the effects of contextual factors, such as organizational norms and culture, and the embeddedness of employees in groups and networks.

In the second example, Andersen, Pallesen, and Pedersen (2011) investigated the questions of whether PSM is higher in the public sector than in the private sector and whether the level of PSM depends on the task rather than on the sector in which the employee works.The methodological design revolved around a large-scale survey, conducted among 4,000 employees in a single occupational group—Danish physiotherapists—performing the same tasks in the private and public sectors. More specifically, the researchers used the abbreviated PSM scale developed by Coursey and Pandey (2007).

The granular quantitative analysis proposed by the authors yielded two main results. First, they found that the general level of PSM does not differ significantly between physiotherapists working in the public and in the private sectors. However, the use of the scale revealed a difference in the nature of PSM across sectors. Although physiotherapists employed in the private sector exhibit higher motivation toward individual users, publicly employed physiotherapists have higher levels of motivation linked to the general public interest and higher levels of compassion. The main findings of this study differ from previous studies that found a significant difference in the level of motivation between private and public sector employees. The authors propose that future research replicate their studies in other contexts to establish the level of generalizability of their findings.

An alternative or complementary approach would be to try to understand the context-specific factors that explain the results of this inquiry through embedding in-depth qualitative approaches within the design. For example, a subsample of the surveyed employees in both groups could be selected for in-depth interviewing to better understand why the type of motivations differs across sectors within the same profession. Some ethnographic methods, such as shadowing or observation could be leveraged to better elucidate some of the findings of the survey. Similarly, the question of whether ownership is a sufficient determining factor to establish the distinction between private and public sectors in this particular context could be explored through qualitative organizational inquiry.

In our third example, Groeneveld and Verbeek (2012) investigated the effectiveness of organizational diversity policies in improving the representation of ethnic minorities in organizations. They conducted a lagged dependent variable regression model based on data harvested from 8283 annual reports filed as a requirement of the Dutch law on the labor participation of minorities. One of the study’s findings is that the effect of managing diversity was rather modest. However, it is increasingly recognized as good practice in the assessment of policy impact to conduct a granular study of policy implementation. Embedding a qualitative design within this impact study would have gone a long way in explaining whether there were implementation failures of the policy, or whether there was variation in implementation across time or location. Any claim of causal inference on the efficacy of a policy is vulnerable without being backed by a description of how the policy was implemented on the ground.

The three examples presented above were selected to illustrate some of the benefits of expanding the breadth of methods used in public personnel administration research, and the resulting intersection between theoretical, methodological, and practical considerations. In the next section, we present a case of mixed-methods inquiry which was designed to apply the principles of mixed-methods discussed above. Integration took place at three levels: research paradigms (theory and epistemology), methodology, and findings, with important consequences for the practical knowledge generated through the research.

A Recent Example of Mixed-Methods Inquiry in International Development Administration

In this section, we go beyond discussing the potential and opportunities for the integration of multiple methods, and demonstrate its feasibility and added-value by presenting a fully fledged example of mixed-methods inquiry in public administration. The study researched the institutionalization of a performance assessment system in a complex international organization inspired by the New Public Management paradigm, and paid particular attention to the effect of the performance assessment system on the motivation of international public servants working in the organization (Raimondo, 2016a), a topic that had received little empirical attention. Discussing in detail the research findings is beyond the scope of this article, rather, the focus here is on the methodological design choices, what informed them, how they provided multiple lenses to look at a complex phenomenon, and how the findings were integrated theoretically, methodologically, and in praxis.

Theoretical Integration

Understanding the effect of Results-Based Monitoring and Evaluation (RBME) on the performance of a large international organization, and its effects on agents’ behaviors within the system, is a complex topic of inquiry which requires a multi-layered sense-making framework. A first aim of the research, not discussed in details in this manuscript, was theoretical integration. The first part of the research design consisted of weaving together theoretical insights from diverse theoretical strands, stemming from public administration theory, evaluation theory, and international organization theory.

Mixing theoretical paradigms is one of the most contentious and difficult part of a mixed-methods design (Creswell & Clark, 2011; Greene, 2007). Given that paradigmatic assumptions guide and direct practical inquiry decisions, these assumptions and the differences between them should be made explicit, and intentionally used together. The juxtaposition of diverse assumptions helps reframe or create new understanding. In practice, it was important to use a proper framework that helped organize the various strands of literature and achieve theoretical integration. The theoretical integration informed the formulation of three main research questions—providing a scaffold for the research design, making incremental analytical steps from description to explanation—and were framed as follows:

Research Question 1: How is an RBME system institutionalized in a complex international organization?

Research Question 2: What difference does the quality of RBME make in project performance?

Research Question 3: What behavioral factors explain how the RBME system works in practice?

As is the tradition in mixed-methods inquiry, the research questions were not predefined by a particular methodological preference, but rather driven by research gaps, theoretical foundations, and a well-informed problem statement.

Methodological Integration

The methodological decisions followed suit, and the overall research design was motivated by the precept of Pawson’s (2013) scientific Realism and its anchor in explanation building:

Theories cannot be proven or disproven, and statistically significant relationships don’t speak for themselves. While they provide some valuable descriptions of patterns occurring in the world, one needs to be wary of the fact that these explanations can be contradictory or artefactual. Variables do not have causal power, rather the outcome patterns come to be as they are because of the collective, constrained choices of actors in a system [and] in all cases, investigation needs to understand these underlying mechanisms. (p. 18)

The mixed-methods design thus followed a staggered approach: each phase and its method strategy informed the next phase, and checked the findings from the preceding one. The study started with a thorough mapping of the RBME system’s organizational elements, and its evolution over time. Through an in-depth review of archives and documents, the research identified the main agent-based driven changes, and the configurations of factors that influenced these changes. To achieve high quality in this type of documentary research, three main criteria of rigor were applied to the selection of the documents reviewed and to the extraction of information from these documents: systematization, transparency, and triangulation. For example, a systematic analysis of all past annual results and performance reports was performed using a content analysis software and the application of a coding system across all documents. The results of this inceptive research were then triangulated with evidence stemming from interviews with key informants within the organization as well as outside observers.

To answer the second research question, the mixed-methods design integrated a number of Propensity Score Matching models to measure the association between Monitoring and Evaluation (M&E) quality and project performance for more than 1,300 projects that were evaluated by an independent evaluation group within the organization. The quantitative approach identified patterns of regularity in the positive association between M&E quality and project performance (Raimondo, 2016b).

The third phase of the inquiry involved examining the behavioral make-up of the RBME system, with a particular focus on incentives and motivations shaping the behavior of project managers responsible for managing RBME in their projects, and for using the evidence from RBME during the project in the preparation of subsequent projects. This phase of the research followed common criteria of qualitative research (Silverman, 2011): the cogent formulation of research questions, the clear and transparent explication of the data collection and analysis, the theoretical saturation of the available data in the analysis, and the assessment of the credibility and trustworthiness of the results.

A concern with qualitative approaches is the possibility that the interviewees provide an answer to questions, not because they are accurate representations of their thoughts or past actions, but because it is the socially desirable answer. To address this challenge, the interview questions were neutrally worded, and all of the interviewees were assured of confidentiality. Staff members were also engaged in game-enabled research processes that helped with participants’ cognitive abilities in a relaxed, pressured-free environment. The game was used to tap into staff members’ experiential knowledge and to better understand group dynamics when operationalizing complex tasks and faced with challenging decisions.

The level of transparency of the analysis and interpretation of qualitative data is a critical element determining their credibility. To maximize traceability, a qualitative content analysis software was used, that allowed inductive coding and tracing back every single theme and finding emerging from the data to their original source in the interview transcripts. In the spirit of grounded theory, theoretical saturation on every theme that was mentioned in the final analysis was reached. Theoretical saturation is the point at which the dimensions of the theme and its account are fully described with sufficient evidence to capture its complexity and variation.

Advantages of a Mixed-Methods Research Design in the Example

There were three primary advantages to using a mixed-methods research design in this particular example: enhanced validity, but also nuance, of the evidence uncovered, as well as increased theoretical and practical contribution to the field.

First, the design enhanced the construct validity and the quality of inquiry inferences by using methods with offsetting biases. The term triangulation of findings is a core principle of mixed-methods inquiry. It is typically interpreted to mean that multiple methods echo each others’ findings in a way that provides evidential saturation for a particular finding. The rationale for triangulating is that all methods have inherent limitations, so that by combining multiple methods, a researcher can counterpoise biases, to check whether results eventually converge. Ultimately, the validity and credibility of the research findings is better than it would have been if only a single method had been used (Creswell & Clark, 2011; Greene, 2007). For example, the systematic content analysis of documents and in-depth interviewing strengthened the construct validity of the findings from the regression modeling. Similarly, the various modeling strategies and the preparatory structural equation model made the statistical validity of the propensity score matching model more robust.

Using multiple methods also strengthened the internal validity of the research findings and the robustness of any causal inferences. Although the quantitative approach identified patterns of regularity in the positive association between RBME quality and project performance, the qualitative approach was necessary to shed light on the behavioral mechanisms that triggered these patterns of regularity. In addition, by combining multiple methodological lenses, the study was able to take a configurational approach, and identify causal packages, as opposed to assuming simplistic single cause and effect relationships (Pawson, 2013; Rihoux & Ragin, 2009). It was indeed the combination of organizational factors, historical path dependence, managerial signaling, and behavioral responses by staff that could ultimately explain the patterns of regularity that surfaced in the quantitative study.

With regard to external validity, the study relied on a robust theoretical framework that emerged from an interdisciplinary understanding of the phenomenon. Combining theoretical robustness and rich descriptions of the contextual factors that explain the findings, the study reached a high level of analytical and context-bound generalizability (Rihoux & Ragin, 2009).

A second advantage of using mixed-methods was the opportunity to reach a high level of nuance in the findings. As noted by Greene (2007) one of the assets of mixed-methods inquiry is the possibility of generating empirical puzzles which provide clear path for further knowledge accretion. In our example, the various methods took due account of divergent perspectives, addressed emerging paradoxes, and delved into complex behavioral mechanisms. In the process of checking the evidence from different methods against each other, at times, the result did not necessarily converge. These “empirical puzzles” warranted further examination and informed the progress of the empirical inquiry via the appropriate methods. Concretely, the findings of the quantitative models raised a paradox: on one hand, the association between the quality of M&E and project performance was positive and statistically significant; on the other hand, M&E quality has remained historically low in the organization. Explaining this paradox warranted an in-depth qualitative look into actors’ motivation, incentives, and the norms and organizational cultural traits that could explain why M&E quality remained historically low in the organization.

Third, the research was able to increase its theoretical and practical contributions to the field by combining multiple theoretical strands. The interdisciplinary look at the problem unveiled complex conjunctions of material, cultural, internal, and external factors that affect processes of change in complex organizations, and advanced theoretical development regarding organizational change.

From a practical perspective, the research brought empirical evidence that contributed to questioning one of the core assumptions on which performance management in international organizations relies and is operationalized in the organization: the compatibility of the accountability and learning objectives of the evaluation function. By unpacking the RBME systems’ behavioral ramifications, this study was able to precisely pinpoint key areas of tensions, and to illustrate how a system primarily designed to uphold corporate reporting and accountability could crowd out learning. These findings had clear practical ramifications to inform the design of a new strategy in the organization studied.

Conclusion

In conclusion, we want to link the discussion laid out here in favor of more methodological diversity in the study of public administration and public personnel motivation to the topic of the symposium: theoretical determinism versus methodological determinism and the intricate interplay between theory and method. Let us return for a brief moment to the fundamentals of theories of knowledge in the social sciences.

The epistemological complexity within social sciences has long been recognized, and especially in the study of humans and society (Riccucci, 2010; Vaessen & Leeuw, 2009). One of the most rigorous arguments explaining why stable and cumulative social sciences aspiring to theoretical universalism is probably impossible resides in Dreyfus and Bourdieu’s “Tacit skills” argument (Bourdieu, 1977; Dreyfus, 1982). Made simple, the argument goes as follows: the aspiration for stable and cumulative social sciences requires a theory of human skills and behaviors, which is impossible because human skills and behaviors are context specific and do not answer to rules. However, a theory must be context-free to be general enough, if not universal (Flyvbjerg, 2002, p. 47). It follows that the study of human behaviors and society is unavoidably context-dependent. One of the practical consequences of this conundrum for public administration research, for both methodological choices and the classes of problems we study, is that context must have a very central position, and grand, predictive theory a less central one than in natural sciences. The imperative is thus to use modes of inquiry and methods that capture the particular and the context-dependency of human phenomena. Context sensitivity can be measured in various ways, for instance, using experiments (Van Bavel, Mende-Siedlecki, Brady, & Reinero, 2016), or, as we propose in this article, by mixing methods where appropriate.

Mixed-methods inquiry opens the door for combining the general and the particular, the universal and the context-dependent, and, ultimately, building both theoretical and practical knowledge. Building knowledge around cases and combining different methods to elucidate cases and accrue context-dependent knowledge is a promising avenue to “move from lower to higher levels in the learning process” (Flyvbjerg, 2002, p. 71). To paraphrase Anthony Giddens (1984), the challenge for public administration and management research is precisely to elucidate the nature of agents’ knowledge and motivation, their reasons for action, and the decisions made in multiple action-contexts.

By nature, research in the fields of public administration and public policy requires a balance between methods that are both understanding-oriented and action-oriented. In this article, we have recommended that the triptych underlying such a mission is to (a) frame the research questions to address the important problems, (b) find the most appropriate methodological combination to answer the questions, and (c) take utility and practicality into account when using the chosen methods.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Author Biographies

Estelle Raimondo is an evaluation specialist at the World Bank’s Independent Evaluation Group where she advises on methodological design, and a professorial lecturer in research methods at GWU. She received her BA from the Institute of Political Studies in Paris, her MA from Columbia University and her PhD in Public Policy and public administration from the George Washington University.

Kathryn E. Newcomer is the director of the Trachtenberg School of Public Policy and Public Administration at the George Washington University, where she teaches graduate courses on public and nonprofit program evaluation, and research methods. She is president of the American Evaluation Association (2017), and is an elected fellow of the National Academy of Public Administration. She received her BS and MA at the University of Kansas and her PhD in political science at the University of Iowa

References

Ahmed

Sil

(2009). Is multi-method research really “better?” Qualitative and Multi-Method Research, 7(2), 2-6.

Andersen

L. B.

Pallesen

Pedersen

L. H.

(2011). Does ownership matter? Public service motivation among pshysiotherapists in the private and public sectors in Denmark. Review of Public Personnel Administration, 31, 10-27.

Bamberger

(2015). The importance of a mixed methods approach for evaluating complexity. In Bamberger

Vaessen

Raimondo

(Eds.), Dealing with complexity in development evaluation. Thousand Oaks, CA: Sage.

Bourdieu

(1977). Outline of a theory of practice. Cambridge, UK: Cambridge University Press.

Coursey

D. H.

Pandey

S. K.

(2007). Public service motivation measurement: Testing an abridged version of Perry’s proposed scale. Administration & Society, 39, 547-568.

Creswell

Clark

(2011). Designing and conducting mixed-methods research (2nd ed.). Thousand Oaks, CA: Sage.

Donaldson

Christie

Mark

(Eds.). (2015). What counts as credible evidence in applied research and evaluation practice? (2nd ed.). Thousand Oaks, CA: Sage.

Dreyfus

(1982). Why studies of human capacities, modeled on ideal natural science can never achieve their goal. Proceedings of the Boston Colloquium for the Philosophy of Science. 1, 3-22.

Flyvbjerg

(2002). Making social science matter. Cambridge, UK: Cambridge University Press.

10.

Frederickson

Smith

(2003). The public administration theory primer. Boulder, CO: Westview Press.

11.

Giddens

(1984). The constitution of society: Outline of the theory of structuration. Cambridge, UK: Policy Press.

12.

Greene

(2007). Mixed methods in social enquiry. San Francisco, CA: John Wiley.

13.

Groeneveld

Tummers

Bronkhorst

Ashikali

van Thiel

(2015). Quantitative methods in public administration: Their use and development through time. International Public Management Journal, 18, 61-86.

14.

Groeneveld

Verbeek

(2012). Diversity policies in public and private sector organizations: An empirical comparison of incidence and effectiveness. Review of Public Personnel Administration, 32, 353-381.

15.

Harmon

(1981). Action theory for public administration. New York, NY: Longman.

16.

Head

(2015). Toward more “evidence-informed” policy-making? Public Administration Review, 76, 472-484.

17.

Jilke

Van

Walle

Kim

(2016). Generating usable knowledge through an experimental approach to public administration. Public Administration Review, 76, 69-72.

18.

Jung

Rainey

(2011). Organizational goal characteristics and public duty motivation in U.S. Federal Agencies. Review of Public Personnel Administration, 31, 28-47.

19.

Kaufman

(1960). The forest ranger: A study in administrative behavior. Oxford, UK: Oxford University Press.

20.

Lewin

(1952). Field theory in social science: Selected theoretical papers by Kurt Lewin. London, England: Tavistock Publications.

21.

Nesbit

Rimes

Christensen

Brudney

(2016). Inadvertent volunteer managers: Exploring perceptions of volunteer managers’ and volunteers’ roles in the public workplace. Review of Public Personnel Administration, 36, 164-187.

22.

Newcomer

K. E.

(2016, June 15). The organizational and political landscape for evidence-informed decision making in government. Paper presented at the 7th Annual Sino-U.S. International Conference on Public Administration, Beijing, China.

23.

Newcomer

K. E.

Hatry

H. P.

Wholey

J. S.

(2015). Planning and designing useful evaluations. In Newcomer

K. E.

Hatry

H. P.

Wholey

J. S.

(Eds.), Handbook of practical program evaluation (pp. 7-34). Jossey-Bass.

24.

Pandey

Pansey

Breslin

Broadus

(2016). Public service motivation: Prologue and prospects. In Raadschelders

Stillman

(Eds.), Foundations of public administration (pp. 314-332). Irvin, CA: Melvin and Leigh.

25.

Pawson

(2013). The science of evaluation: A realist manifesto. Thousand Oaks, CA: Sage.

26.

Perry

J. L.

(2012). How can we improve our science to generate more usable knowledge for public professionals? Public Administration Review, 72, 479-482.

27.

Petticrew

Roberts

(2003). Evidence, hierarchies, and typologies: Horses for courses. Journal of Epidemiology & Community Health, 57, 527-529.

28.

Poocharoen

Brillantes

(2013). Meritocracy in Asia Pacific: Status, issues, and challenges. Review of Public Personnel Administration, 33, 140-163.

29.

Raadschelders

(2011). Public administration: The interdisciplinary study of government. Oxford, UK: Oxford University Press.

30.

Raimondo

(2016a). The institutionalization of monitoring and evaluation systems within international organizations: A mixed-methods study (Dissertation UMI No. 10053381). Retrieved from www.proquest.com.

31.

Raimondo

(2016b.) What difference does good monitoring and evaluation make to World Bank project performance? World Bank Group Policy Research Working Paper 7726.

32.

Riccucci

. (1995). Unsung heroes: Federal execucrats making a difference. Washington, DC: Georgetown University Press.

33.

Riccucci

(2010). Public administration: Traditions of inquiry and philosophies of knowledge (Public Management and Change Series). Washington, DC: Georgetown University Press.

34.

Rihoux

Ragin

(2009). Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques. Thousand Oaks, CA: Sage.

35.

Rorty

(1982). Consequences of pragmatism. Minneapolis: University of Minnesota Press.

36.

Sanjeev

Craig

(2010). Toward a platinum standard for evidence-based assessment by 2020. Public Administration Review, 70(Suppl.), 100-106.

37.

Silverman

(2011). Interpreting qualitative data (4th ed.). Thousand Oaks, CA: Sage.

38.

Stivers

(2000). Bureau men, settlement women: Constructing public administration in the progressive era. Lawrence: University Press of Kansas.

39.

Teddlie

Tashakkori

(1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: Sage.

40.

Teddlie

Tashakkori

(2003). Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In Tashakkori

Teddlie

(Eds.), Handbook of mixed methods in social and behavioral research (pp. 671-701). Thousand Oaks, CA: Sage.

41.

Teddlie

Tashakkori

(2009). Foundations of mixed-methods research. Thousand Oaks, CA: Sage.

42.

Vaessen

Leeuw

(2009). Mind the gap: Perspectives on policy evaluation and the social sciences. Somerset, NJ: Transaction Publishers.

43.

Vaessen

Raimondo

. (2012). Making sense of impact: A methodological framework for assessing the impact of prizes. Evaluation, 18, 330-347.

44.

Van Bavel

J. J.

Mende-Siedlecki

Brady

W. J.

Reinero

D. A

. (2016). Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences of the United States of America, 113, 6454-6459.

45.

Vrangbaek

Petersen

Hjelmar

(2015). Is contracting out good or bad for employees? A review of international experience. Review of Public Personnel Administration, 35, 3-23.