Conducting commissioned research in neoliberal academia: The conditions evaluations impose on research practice

Abstract

Spanish

French

The article deals with the unease we experience during various commissioned research projects. On the one hand, as social scientists, we feel committed to conducting ‘good research’ that acknowledges quality criteria such as flexibility and transparency and in particular allows for musing and reflexivity to ‘discover’ new aspects of our research topic. On the other hand, we are situated in the context of present-day neoliberal academia. This means that our work is assessed according to a culture of audit characteristic for neoliberal management of universities that values publication indexes and fundamental research. At the same time, universities strive increasingly for third-party funding that favors commissioned research. This article discusses how commissioned research conditions our evaluations and research practice and how these conditions might conflict with the ‘good research’ we hope to conduct.

Keywords

Commissioned research ethics of research evaluation funding policies neoliberal academia quality of research

Introduction

The debate about good evaluation research has a long history and has always dealt with both questions of quality as well as questions of ethics (Morris, 2008). This debate was particularly intensified with Guba and Lincoln’s fourth-generation evaluation (Guba and Lincoln, 1989) and their accentuated phenomenologist stance. In the context of broader scientific criticism of positivist science and the search for alternative epistemologies, two primary claims confronted evaluation research. First, in order to acknowledge the impact evaluations have on involved stakeholders, researchers should respect the right of stakeholders to take part in the data production. This moves the perspective of the stakeholders to center stage. Researchers must be aware that an evaluation always includes the concerns of the ones directly affected by its outcomes (Guba and Lincoln, 1989; House, 1993). Second, the turn from a testing and verifying positivist perspective to a constructivist perspective ends with the notion of the scientist as an objective authority because the scientist is part and parcel of the production of this evaluation. This also entails a move from the verification of research questions to the discovery of unknown, unseen, or unacknowledged meanings, processes, and structures (Chelimsky and Shadish, 1997; Guba and Lincoln, 1989). In Guba and Lincoln’s (1989: 59) words, ‘progress in science … depends heavily on intuitions, flashes of insights, “vibes”, or mental experiments to provide the propositions that can be tested in authentic scientific fashion.’ This also entails, much like in grounded theory (Glaser and Strauss, 1998 [1967]), the construction of new theories by relying on qualitative data.

Despite the accentuated debate they provoked in the academic community at the time of their inception, today, most of these claims have become professional standards adopted by important professional associations such as the American Evaluation Association (AEA). For instance, it is widely accepted that good evaluations should rely on a mix of qualitative and quantitative methods (Cook, 1997). At the same time, the debate about quality and ethics in evaluation research must continue, as changes in the formulation of standards and a series of recent articles document (e.g., Schwandt, 2007; Simons, 2006). Another reason is the development of academia itself and of how the academic community and society at large value different types of scientific practice and their respective outcomes.

The general development in which current academic work and research are embedded can be described – in particular, with regard to our discussion – by two main aspects. On the one hand, academia and its management have adopted neoliberal values and practices (Castree, 2000, 2006; Power, 1994; Strathern, 2000). Promoted by the notion that competition increases productivity and the quality of the products, competition between individuals in academia as well as between departments, universities, disciplines, and countries (Sheppard, 2006) is now paramount. The logic of the market also comes into play with the assumption that valuable research will receive funding and that individuals and institutions that are successful in acquiring funding for their research produce valuable research. The market thereby assesses the quality of the research. Neoliberal academia is, in particular, also characterized by methods and instruments for the assessment of the performance of individuals, departments, and universities. A new audit culture (Castree, 2000, 2006; Elton, 2000; Strathern, 2000) has emerged for the assessment of the quality of research and of academic entities based on criteria such as the amount of third-party funding for research or the number of publications in peer-reviewed journals (Elton, 2000).

On the other hand, research and knowledge production in the academic sphere are linked to a wider knowledge economy including organizations outside the university. State agencies or private entities, such as NGOs, think tanks among others, also produce and control knowledge. The knowledge that is produced in universities is now part of a knowledge economy that seeks for and controls access to knowledge that is applicable for technical and social purposes. In this sense, the university has become a ‘pluriversity’, producing knowledge that is also geared towards application and driven by needs outside academia (Torres, 2011). Nevertheless, academia still plays an important part in this knowledge economy as it continues to engage in ‘basic research’, which can be seen as the last autonomous site of knowledge production outside the realm of the public and private sector. As Castree (2006: 747) puts it: ‘willingly or not, we are perceived as “experts” in the wider society: people equipped to produce sophisticated cognitive, technical, moral and/or aesthetic understandings of material and representational’ knowledge.

It is in this latter context in which we locate our unease. Tensions and animosities between fundamental and applied research have a long tradition (see, e.g., Kaufmann, 1969). Whereas the former is highly valued and leads to a higher reputation as a scientist, applied research is seen as a byproduct of fundamental research and not as a type of research on its own. It is rather a service to the community than ‘real’ research. This contrasts starkly with the importance applied research has in the growing necessity for universities and departments to acquire external funding. The same differences in value also apply to the outcome of both types of research. In particular, articles in peer-reviewed journals are highly valued whereas the typical outcomes of evaluation research, such as reports or articles written for a non-academic audience of practitioners, are not acknowledged as proper forms of scientific production (Castree, 2006; Elton, 2000).

This tension between applied and fundamental research is seldom discussed as a possible factor limiting commissioned research. One reason may be that commissioned research must cope with predefined research questions and has to be carried out by following a tight schedule while operating under power constellations related to particular social problems and associated interests (Von Kardorff, 2000). However, the abovementioned contextual aspects of neoliberal academia and its role in the knowledge economy have a strong influence in the long run as individual researchers and departments are ranked on the basis of their research outcomes.

In what follows, we discuss our experiences from conducting several commissioned evaluations while working in an academic environment. We are particularly interested in the various problems commissioned research – as a specific form of research – generates for researchers who want to conduct good research. We start with four vignettes that are characteristic of the constellations we experienced in the course of several projects of commissioned research we have carried out. The vignettes follow the temporal logic of a commissioned project, portraying moments in (1) contracting, (2) designing research, (3) the research process, and (4) presenting and/or applying results.

Contracting – Expectations and realistic realization

Commissioned research is always carried out under a contract that defines the available resources as well as the research goals. Moreover, behind such contracts often lies a hidden agenda. Because we had become quite familiar with the research field – the Swiss prison system – over the last few years, we always made efforts to identify important stakeholders such as prison directors, members of political bodies, parole boards, or the prison administration. We spent time meeting these people and talking to them beforehand. This gave us a better understanding of why the evaluation had been commissioned, what pitfalls might come up, and where hidden intentions or interests were located. Our projects deal with public spending and therefore attract general public interest and are carried out in the realm of the prison system, a field of research that in itself is very sensitive to public opinion (security), the rights of the inmates (confidentiality), as well as the rights and interests of prison staff and management. This means that compared to most social science research, the process of contracting is more complicated as it reflects pressures, fears, and intentions of involved stakeholders to a greater extent. Meeting stakeholders beforehand helped, on the one hand, to understand the aims of the commissioned research and, on the other, to define clearly the limits of the research. For instance, in research on policy and practice in the prison system, politicians and practitioners always want to know whether a particular measure is effective in reducing recidivism. However, because this can only be determined with a sophisticated research design (a vast amount of quantitative data and a long-term survey beyond the completion of the project), we were not able to provide answers to these questions. We always had to insist upon this limitation at the moment of contracting.

Designing the research – Resources and outcomes

Most competitive calls for commissioned research include specifications on resources and research goals. This means that market principles of cost-effectiveness play an important role. Thus, it is not only the quality of the design or the particular way questions defined by the client have been addressed by the applicants that counts heavily, but also the output (in numbers of interviews, different approaches, reports, and presentations) researchers can provide for the defined amount of money. This strong incentive encourages researchers to offer as much as possible for the approved budget, leaving them with little buffer for unexpected challenges. Therefore, the resources are rather restricted compared to the proposed output, a fact that can affect the quality of research and lead to self-exploitation.

The research process – Developing the subject vs. taking a pre-defined research perspective

Our understanding of the research question was partly shaped by our conceptual focus on the prison as a complex system with different groups of stakeholders characterized by diverging interests and resources. Therefore, our goal was to make these different perspectives visible in our research: inmates, prison staff, and management. Some stakeholders appreciated this approach because it allowed them to voice their concerns and to find answers to their own questions. Others, mostly representing project leaders and/or members of the funding agency, focused first on preliminarily defined questions and were not willing to allow the expansion of the research question or changes in its focus.

On the other hand, the research perspective is always limited by the constraints and aims set in the contract. This strong emphasis on the original research design and research focus meant that we were caught in the yearly routine of collecting massive amounts of data; in particular, we were not able to analyze the data in-depth because even qualitative data seem more reliable to the client when available in a large amount. After data analysis, reporting absorbed most of our resources to complete the contracted yearly reports. Although resources allowed us to fulfill the contract, they were never enough to seriously reflect on the evolving research process as a whole, its biases, and its lacunae.

The presentation and use of the results – Formative, summative, and legitimatizing function of results

Finally, the way results are perceived, interpreted, and used by the client can diverge from our notions as researchers of how it should be done. In the case of a formative evaluation, we periodically presented results intended to provide a sound basis for future adjustments and project improvement while the project was underway. Every year, we encountered the following situation: We met with the representatives of the project and the funding agency to discuss the results they had received previously in the form of a report. Our expectation was to discuss these results during the meeting with an emphasis on what could be useful for the project’s further development and improvement. Instead, we were confronted with harsh criticism. Project leaders picked up formulations in the report that showed weak points in their own work or that of collaborators. They insisted upon changing the wording of the report to obtain a more favorable version. Further, they questioned our professionalism in collecting data. The quantitative data seemed not trustworthy enough because we had asked inmates to assess changes in their own skills and behavior – data that were compared to the assessment of other people, such as prison staff. The qualitative data were criticized for not allowing the formulation of general conclusions because of low numbers. In addition, we became aware throughout the project cycle that our recommendations had been ignored or not taken seriously and though it had been contracted, the formative logic of the evaluation was not really prompted by those responsible for the project. The project leaders were trying to appropriate these results because they were using them for legitimization only. In their view, an ‘independent’ voice to endorse the project to support its reputation and ensure funding in the future seemed politically correct.

These four vignettes give insight into the tensions we encountered while conducting evaluations. Over the last seven years or so, our research has focused on topics related to the Swiss prison system. The team¹ has always been multi-disciplinary, comprising researchers with a disciplinary background in sociology, social anthropology, geography, and political sciences. This allowed us to benefit from the advantages of collaborative work to discuss methods and data, adjust findings, and gain a deeper insight into the research topic (Hostettler, 2012; Houston et al., 2009). Further, it enabled us to use a variety of methods in a mixed-method approach and take advantage of team members’ disciplinary and personal specialization and experiences in diverse methods. In different combinations, we use quantitative and qualitative methods, including surveys, semi-structured interviews, focus group discussions, observation, and document analysis.

In addition to ongoing fundamental research funded by the Swiss National Science Foundation, part of our research is of the contracted type. Among these contracts, there were three major projects that lasted for at least a year and involved substantial funding. Apart from several research reports and articles written for an audience of practitioners, we also published papers in journals (Achermann and Hostettler, 2007; Richter et al., 2011) and in edited books (Hostettler, 2012; Kirchhofer and Richter, 2012).

We start our discussion by outlining criteria of quality in social science and evaluation research. We focus mainly on criteria in qualitative research as our research was and is driven by a qualitative perspective, although we usually also include quantitative elements. Criteria discussed in the context of evaluation research reference the debates in quantitative and qualitative research. At the same time, they are geared towards providing good evaluation for the contracting body. Lastly, we explore the conditions commissioned research imposes against the criteria of quality discussed before and compare them with experiences we had. In particular, we discuss how commissioned research and its outcomes need to be assessed in the context of an academic culture that operates increasingly according to audits, publication indexes, and research grants.

Good research and good evaluations

Our experience in various evaluation projects provides the empirical background to discuss the question of how we are able to conduct good research with commissioned research. Good research – and in our case, good evaluations – requires some thought about what ‘good’ means. Whereas the quality criteria for quantitative research are commonly agreed upon and based on objectivity, reliability, and validity, the debate in the realm of qualitative research is more contested. Our own mixed-methods approach can be called ‘qualitative driven,’ resembling what Howe might call ‘mixed-methods interpretivism’ (Howe, 2010: 169). We thus limit our discussion of quality criteria to the realm of qualitative approaches in social sciences.²

In what follows, we focus on a proposition, including a catalog of criteria, that is widely accepted in German-speaking academia in which most of our own work is situated. The criteria Steinke (1999, 2000) proposed cover the research process as a whole. They start from the relevance of the research and end with aspects that deal with the final transmission of the results. The criteria are based on the notion that qualitative research needs to be utterly flexible in its empirical approach (Flick, 2000 [1995]; Steinke, 1999). Therefore, the ways in which methods of data collection and analysis are adapted to the topic of research become an important issue for quality concerns. As the methods are rarely standardized, rather varied, and usually adapted to the specific project, it is even more important to document the research process. The positionality requires the researcher to adopt a reflexive stance through the entire process. Researchers cannot ensure data collection and analysis that is not influenced by their subjective position. Therefore, the need to make data collection and analysis as well as the researcher’s part in these processes understandable to others is of great importance. Based on these thoughts, Steinke (1999, 2000) defined the following criteria:

Intersubjective understandability: This criterion is formulated analogously to the quantitative criterion of objectivity, which allows other researchers to test and confirm research findings. However, intersubjective understandability does acknowledge the positionality and influence of the researcher, so results can never be replicated exactly by others. It is therefore important to make steps of data collection and analysis transparent and understandable.

Indication of the research process and its methods, including operationalization and concepts: As noted above, qualitative research designs are well equipped to adapt process and methods to the topic investigated. On the one hand, this represents a strength of qualitative research. On the other hand, it implies that there is a need for explanation of why the chosen process and methods are indicated for the topic studied.

Limitation of the research: Results of qualitative research might not be generalized in the same way or to the same extent as quantitative research; nevertheless, information on the generalizability and the limitations of the research is important. Does the research only speak for the people interviewed/observed? Can it speak for an entire community? If not, what is the range of its results?

Coherence: This criterion is concerned with the coherence of the argumentative text of a report, article, or book, but also with the internal coherence of the research as such. It is not only the final product that needs to be coherent to convince an audience. Coherence is also needed at the level of theoretical concepts and empirical design.

Relevance: ‘Relevance’ has become a difficult word in recent years because it is often used to discredit research that does not produce directly profitable results. Relevance needs to be understood in a much broader way. A research project should not follow only the personal interests of the researcher. It needs to be relevant to other audiences, such as the public in general, but also specialized academic disciplines, stakeholders, or minority groups of a larger society.

Empirical grounding of theory building and testing: The founders of grounded theory (Glaser and Strauss, 1998 [1967]) highlighted that theory building and testing always require empirical data. Data collected in qualitative studies are usually a rich source for theory building and testing and serve as a basis for the development of new concepts. Therefore, its use should not be restricted to anecdotal embellishment.

Reflexive subjectivity: The positionality of the researcher requires a continuous reflexive stance. Neither the selected topic of research nor the way in which we approach it, think of it, conceive it, collect data about it, and analyze it can be disentangled from the person of the researcher. The researcher takes an active part in a qualitative research process and needs therefore to reflect upon his or her role in the process. This opposes the notion of objectivity because qualitative research acknowledges the subjectivity of the researcher and requires her or him to be reflexive in order to make research thoughtful and transparent.

Criteria formulated by other authors such as Tracy (2010) emphasized some aspects in greater detail. For instance, Tracy insists on the aspect Steinke calls ‘relevance’ by differentiating three related categories: ‘worthy topic,’ ‘resonance,’ and ‘significant contribution.’ What in Steinke’s account remains implicit but explicitly named by others is ethics in research (e.g., Naples, 2003; Tracy, 2010: 846ff.).

In general, however, most propositions for criteria of good research do not include a closer look at the conditions in which research takes place and the urge for researchers to produce innovative knowledge. In general, this question is scarcely debated because new knowledge only rarely emanates from formalized processes. As formalization is a key characteristic of scientific inquiry, it seems impertinent to seek innovation in arbitrary procedures (Breuer and Reichertz, 2001): ‘Looking at the history of science one finds that the examples of new insights that were brought to light while obeying the rules of the theory of knowledge and understanding are few. Often enough intuition, coincidence, self-interest and stubbornness engendered new ideas’ (Breuer and Reichertz, 2001: 7). Such an abductive way of reasoning (Reichertz, 2000) – in addition to inductive and deductive ways of reasoning – requires a certain freedom for the researcher to leave the beaten path. This abductive stance reminds us of the claim for discovery in fourth-generation evaluation made by Guba and Lincoln (1989). This is of no surprise as the debates about good research and good evaluation (research) are intimately linked.

The present standards of professional associations in the field of evaluation have included and amalgamated the various debates and accept a mixed-methods approach as mostly indicated for evaluations (Lincoln, 2009). Nevertheless, they mirror the specific context in which evaluations and commissioned research in general take place. The Swiss Evaluation Society (SEVAL) defined their standards in the 1990s. They are based on standards formulated by the Joint Committee by taking up discussions in the United States stimulated by the American Evaluation Association (Yarbrough et al., 2011). When the SEVAL published their standards in 2001, they were the first society in Europe to do so (Widmer, 2011). Since then, the standards have been used as guiding principles in a process of professionalization of the evaluation praxis in Switzerland and are by now widely accepted by evaluators as well as their clients. They encompass four basic dimensions of quality: usefulness, in particular, for the users of the evaluation, feasibility of the project, propriety, in the sense of respecting legal and ethical standards regarding the people involved in the evaluation, and accuracy regarding the process of research design, data collection, and interpretation in order to provide valid results (Widmer, 2011: 25–26; Widmer et al., 2000).

These standards mirror in the first place the concerns of both practitioners and clients of evaluations: evaluations are commissioned to serve a predefined aim, and they should answer a concrete question (such as whether a program is effective and efficient). They are geared towards legitimizing mostly political projects and programs (Widmer, 2011). This becomes clear when we compare the evaluation standards with the standards for qualitative inquiry discussed above.

Although they overlap in many aspects, because they both refer back to a code of conduct of good science, a couple of important differences remain. First, the evaluation standards focus clearly on the evaluation context and the need to be cost-effective and take into account the political field in which the evaluation is carried out to provide public or political support – an instrumental use of evaluation (Chelimsky, 1987) – for the evaluated project that also ensures the acceptance of its later results. Evaluations are always contracted, and evaluators must accept restrictions in time and resources as stipulated in the contract. Of course, some of these limitations apply also to other research contexts.

Second, an evaluation is measured according to its usefulness. While other research projects should also be useful, an evaluation always has a clearly defined public to address and a narrowly defined usefulness: it usually answers commissioned questions.

Third, an evaluation needs to evaluate; it has to judge whether things are good or bad, whether they match previously defined standards. Whereas many forms of qualitative inquiry seek to understand social processes, evaluations seek to judge (Simons, 2006). This contrasts strongly with what we might strive for as social researchers.

Finally, what is missing in these criteria is a reflexive stance and the claim for discovery. Although qualitative methods are being acknowledged – the standards of the SEVAL explicitly name quantitative as well as qualitative methods – their methodological and epistemological debates have not yet reached the discussions of quality criteria for evaluation. Rather, the standards echo the neoliberal discourse of the knowledge economy, which coopts academic research in order to produce knowledge that is useful to the organization that provides the funding. That good research needs flexibility to respond to unforeseen developments in the research process does not seem fully compatible with standards such as usefulness or feasibility.

Commissioned research and its limitations

‘Does research being a “good” in the economic sense, compromise its status as a “good” in the moral sense?’ (Ham, 1999: 275). When we conduct research in an economic context, with contractors who expect a certain product – often favorable to their endeavor – how can we as academics ensure that our inquiry fulfills the quality standards not only of the commissioners but also of academia? How can we succeed in this difficult task in the context of neoliberal management of academia? When researchers sign a contract, they accept many restrictions for their research in exchange for funding and less material aspects, such as public recognition or public usefulness of their research. Such a situation involves, as Ham points out (1999: 281),

[j]udgements as to whether or how much restrictions in contracts actually compromise the ethics of a commitment to truth, expressed in terms such as the freedom to report ‘honestly’, the presumption of negotiation ‘in good faith’, and a ‘trust’ in a shared desire for competent critique. It involves asking whether signing a contract with restrictions on content or publication is necessarily a threat to the researcher’s primary ethical commitment to tracking the ‘truth’, or is it a justifiable accommodation in order to make the ‘case’ manageable or the education system ‘better’; is it an act of professional misconduct or an acceptable sin of commission?

We discuss these questions along the four steps of the evaluation process we exemplified in the introductory vignettes: (1) the process of contracting; (2) the design of the research; (3) the process of research itself, including data gathering, analyzing, and preparation of the report(s); and (4) the presentation of the results and further use of them.

Contracting

As our experience in contracting grew, our capacity to assess particular settings in order to develop an adequate design and reasonable estimation of resources increased. We took care to write this experience into our contracts and to make roles and responsibilities of contract partners explicit in the contract. This also meant, as exemplified in the vignette, that prior to the actual contracting, we had to contact potential stakeholders and thereby become familiar with the setting of the evaluation. Moreover, clients often wanted us to provide information on output or outcome, such as the effect a program has on recidivism, even when the agreed design and available resources (particularly time) did not allow for a methodologically sound assessment of this relationship. From this we learned to invest time to explain to the client the contract disclaimers regarding outcome/output assessments. With time, we also became aware that we should take better care about securing our rights to the future use of data for academic purposes, and we started to include a special clause in the contract.

This takes us to the question of relevance and who defines what is a relevant outcome of research. As defined by Steinke (1999, 2000) and discussed above, relevance is, on the one hand, a difficult concept as it can be used to discredit research results. At the same time relevance should encompass a broad audience that includes the commissioners, but also other – maybe even unintended – audiences. From an academic point of view, relevance and its scope can often only be defined with regard to the results, meaning that it is an issue that becomes clearer towards the end of a research. Adding a clause to the research contract that allows the researchers to use the data for academic purposes is therefore only a first step, as possible other audiences, such as stakeholders not identified beforehand, might appear during the process of evaluation. Giving the researchers the authority over their data for further academic analysis and publication tries to bridge the divide between commissioned and therefore applied research with its outputs such as reports and the fundamental research with outputs such as articles in peer-reviewed journals. It follows the claim of current audit culture to acquire funding and produce at the same time the desired outputs. Gaining academic freedom would also mean ensuring that all the audiences who have an interest in the results gain access to them in a form that matches their needs.

The main problem at the moment of drafting a contract is that it represents a tightrope walk. On the one hand, there is the need to secure the necessary rights, in particular of authorship, and to secure as much flexibility and openness as possible. On the other hand, the commissioners require their questions to be answered and the answers to be grounded in data that seem reliable to a non-academic public, while researchers do not know at the outset of the research how the course of the research might alter the group of stakeholders, how additional questions might arise or whether changes in the empirical approach might be adequate.

Research design

As we were conscious of the fact that commissioned research leads to a contract that sets clear limits on the research process, we proposed a research design that served the interests of the client while respecting in the best possible way the criteria (Steinke, 1999, 2000) of a qualitative-driven inquiry. First, our reports gave a broad and deep insight in order to make the research process and its results understandable (intersubjective understandability). The outcome was that the reports were longer than the contracting bodies often would have desired. Second, our methods of data collection and analysis were adapted to the topic of research (indicative of the research process and its methods): as we mostly studied complex systems (prisons) with many different stakeholders, in the field, we opted for mixed-methods approaches that acknowledge various perspectives on different institutional levels. In the reports, we tried to represent the different voices of the various stakeholders, and often we had to face the critique that our reports were contradictory. Since we often evaluated new programs or pilot projects, we planned the collection of data through mixed-methods and a series of yearly repetitions of data collection in order to capture developments and changes and to document these by a sound stock of empirical data. Third, because the limitations of a project were usually obvious from the beginning, our evaluation would be limited to the project or program, and our data would only be applicable (at least in the course of the commissioned research) to the prisons and the programs researched. Fourth, the research design needed to be coherent in itself, not only because this constitutes a sound point of departure for research, but in particular because the design was part of the contract between evaluators and client, as well as for budgetary calculations. Fifth, the relevance for the commissioners was given at the beginning by the need to evaluate a program or pilot project. For us as researchers, the relevance could change during the course of the research, as we started to gain a deeper insight into the subject of research. Often we came across questions that seemed relevant from a scientific point of view, but had not been included in the contract. Sixth, all the commissioned research we conducted was based on strong empirical grounds in particular because the commissioners urged us to provide ample empirical evidence for the results. Finally, ethics always played an important role because research in prisons is a highly sensitive field in which information has to be handled with care. The protection of inmates as well as staff is of the highest priority.

At the outset, our evaluations seemed to be well designed, according to current criteria of quality. This ensured some flexibility for the researchers. We also reserved the right for later academic use of data and results. This might not be the case for many commissioned research projects. The limitations became visible afterwards and are, on the one hand, linked to the characteristics of qualitative research and its potential contradictions with expectations linked to contracted research. On the other hand, as exemplified in the vignette, commissioned research competes with offers by other research teams. Therefore, there is an incentive to offer more output for less cost, which reduces further the flexibility and openness demanded by qualitative research.

The vignettes on the research process and, in particular, on the presentation of the results show that it is important to discuss questions of openness and flexibility of the research process before first results are available. The moment of contracting and agreeing on the research design is a moment of relative freedom, as possibly undesired results are not yet on the table.

Research process

In the research process, evaluators move along a rather narrow framework defined by evaluation goals, research questions, applied methods, and available resources. However, this framework is less than stable once research begins, and one has to be prepared for the unexpected developments and institutional contexts that even the most careful planning could not foresee. And even when everything develops as planned, following a core topic in the evaluation process always allows insight into secondary or collateral topics that may grow in importance during the research process. Should it be explored? And how can evaluators know at what moment or under what conditions their research went off on the wrong track and should be stopped?

One of the core principles of qualitative research is the concept of ‘Gegenstandsentfal-tung’ (e.g., Steinke, 1999), which we could translate as an unfolding of the research topic in the course of the research process. This means that when starting a qualitative inquiry, researchers cannot plan the whole research process in advance. Rather, they need the flexibility and the autonomy to adapt the processes of data collection and analysis, as well as their reasoning, to the research topic that starts unfolding bit by bit in each interview, in each observation, with every form of data collected, and with every analytic and reflexive moment. Although we had planned for some flexibility in the research designs, our autonomy was restricted by the contract we had signed. During the course of research in one project, we did change individual interviews to focus group interviews because we felt that interviewing the same people every year led to many repetitions, whereas the focus groups helped to enhance differences and discussions that were already going on among the staff of the prison. At the same time, we felt obliged to collect the amount of data stated in the contract. Even though most of our data was qualitative, the sheer number of interviews was the factor that made our results seem more representative to the client of the evaluation. We therefore spent much time with data gathering that did not contribute to more in-depth knowledge as a theoretical saturation of data had already been achieved.

Flexibility in the research process is very much shaped by the expectations of the commissioners of the evaluation. Some changes in data collection, such as the change from individual interviews to focus groups, were accepted as a change from one qualitative instrument to another and did not conflict with the expected outcome and its believed scientific accuracy. Reducing the numbers of interviews because theoretical saturation was achieved would not have altered the results, but it seemed to change the weight of the gathered data in the perception of the commissioners.

As described in the vignette, the yearly routine of data gathering, fast analysis, and writing of long reports left us with little time to reflect on our positionality, the direction the research was taking, and new topics that were unfolding. Such an amount of data can only be analyzed and reported if one focuses very strictly on the research questions defined beforehand, which leaves no room for reflection, unfolding of the research topic, and general ‘musing’ or ‘guessing’ that could possibly induce abductive reasoning. In this sense, it was rather the characteristic of a qualitative research process that collided with the strict time schedule and output-orientation of the contracts we signed than an insufficiently quality-aware research design that limited the quality of our research. In other words, it is difficult under these circumstances to gain new knowledge and advance science with contracted research. This does not mean that the evaluations were of poor quality; they aimed to answer specific and predefined questions and succeeded in this aim.

But we were also the victims of our own success. In order to acquire funding, we had offered a dense research program in order to compete with other applicants. Acting as neoliberal subjects (Dowling, 2008) we had internalized the importance of acquiring funding. On the one hand, there is a need to discuss what the price of ‘good’ research can be and whether we as academics should avoid presenting research designs that respond too neatly to the market logic of a maximum of results (or data) at the smallest price. It might be necessary to intensify the ongoing discussions about what quality means in social sciences (in particular also for qualitative approaches), and, more importantly, make these discussions resonate outside the walls of universities. On the other hand, we should not forget that many researchers are not fortunate enough to work in permanent positions and therefore also need to acquire funding in order to secure their positions.

The presentation of the results

In general, the client is allowed to use the results of an evaluation in a way that serves the client’s particular interests, such as improving an ongoing program or project (formative), getting an outside view of past performance (summative), legitimating past or planned future action, or improving public relations. The client’s own interests influence to a great extent expectations of quality and content of contracted research. Such expectations may not coincide with what is otherwise good research or evaluation, and they may completely ignore the potential for development and innovation an external evaluation may provide.

In the vignette, we have shown how the presentation and further use of the results consisted of a moment where we realized the constraints limiting our research. As we have described above, the quality of the results was not assessed according to quality standards of research. Rather, the debate focused on the report as a product that would soon be published. The scientific search for ‘truth’ (Ham, 1999) or for ‘understanding’ of the topic under study collided with the contracting bodies’ interest in a legitimate political project. In commissioned research, when dealing with the results and the way they will be presented, we are not confronted with ethical issues such as in research in general. As the contracting bodies of the research usually have their say in what is published and how this is done, we face a problem of who has the authority over the results. A recent study among German evaluators showed that client influence over the results and the way they are presented in publications is a common issue (Stockmann et al., 2011). To put it differently, the problem of anonymity is not posed at the level of the individuals interviewed and researched. Rather, ‘problems with anonymity are more political than ethical as they relate to the demand of control in modern organizations. The consequence of such practice is that those who are protected are not necessarily those that are most exposed, but rather those who in their normal management of power experience a need to have control over information’ (Skaerbaek, 2007: 66).

Control over information is closely connected to the question of relevance. If the commissioners are left with the full control over the knowledge gained through research, then they will also decide what is relevant and to whom. Often the academic realm seems quite safe for the publication of additional outputs that do not touch directly the interests of the commissioners. The question of relevance and of control over what is relevant focuses rather on the communication to stakeholders and the general public.

Conclusions

In the face of a development that makes universities increasingly interested in external funding, researchers seek to acquire funding in the form of contracted research. What might seem at the outset a good way to finance research turns out differently from fundamental research that we usually associate with research conducted at universities. We think that it is important to stress these differences and to acknowledge the conditions contracted research poses on researchers and on the research process.

This article has developed this argument by first discussing quality criteria for qualitative research. We follow a discussion of qualitative methods because our projects usually have a mixed-method approach that also includes quantitative elements but is always ‘qualitative-driven.’ A comparison of quality criteria for qualitative research with quality criteria for evaluations clearly shows where the differences between fundamental research and contracted (oriented) evaluation research lie. Whereas the criteria for fundamental research ensure that the knowledge gained is transparently transmitted, the conclusions are empirically grounded, and the researcher’s positionality is reflected, the standards for evaluations focus much more on research as a service: it should be useful, feasible, correct, and accurate.

Our past experiences acquired in different contracted projects in the realm of prison research in Switzerland indicate that the limitations contracted research poses on qualitative inquiry are hardly palpable at the outset. They rather become visible and powerful as the research develops. At the outset, the research designs could be defined to fulfill quality criteria of qualitative research. Once we accepted the contract, a research topic, and a set of questions to be answered, we were always quite free to present a research design that was adapted to the research topic.

When the project started, we became caught up in a yearly routine of collecting data, analyzing it, and writing the respective report. This yearly rhythm led to a large production of data (e.g., in numbers of interviews conducted) and reports, thereby forgoing a deeper analysis and understanding of the research topic. In particular, we did not find the time to reflect upon the project, its course, the data gathered, and our position in the entire process. The distance and musing that is needed to generate new ideas (abductive reasoning) seemed impossible in such a tight and productive schedule. At the same time, the contract also obliged us to stick to the pace set out at the beginning. Though we were able to adjust some methodological issues, we still had to accomplish the expectations (for instance, in numbers of interviews) formulated in the contract and requested by the contracting bodies. Although we were using mostly qualitative methods, a high number of interviews seemed to make the statements more valid and reliable to the contracting bodies.

The presentation of the results led to another limitation. Apart from discussions about content and professional qualities of the researchers, the debate showed the diverging interests pertaining to the function of the results. Whereas for us researchers the results were meant to support the project leaders in adjusting and improving their instruments and procedures, the project leaders and the funding agency were mainly interested in the political legitimation our report was providing for them and for the project. The results seemed less important than what and in which form should be presented to the public.

Our discussion has shown that the moment of contracting is vital to define questions of authorship over the data and their products and to ensure a maximum of flexibility and openness. This can only be done in dialogue with the commissioners and by taking into account the setting of the research.

Although today it seems vital for universities and for researchers to demonstrate their capacity to acquire funding and complement their budgets with funding from third-party sources such as contracted research, this does not constitute a simple way of financing fundamental research. A good evaluation has its value, but it should not be seen as a substitute for fundamental research or a way of financing fundamental research. At the same time, nothing is gained in playing either of them against the other. To the contrary, in the best case, both approaches enrich each other and produce synergies in data collection, analysis, and interpretation.

Footnotes

Funding

This study received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Notes

Author biographies

Marina Richter holds a PhD in geography and a Habilitation in sociology. She is a researcher and lecturer in sociology, social policy, and social work at the University of Fribourg, Switzerland. Her research spans the fields of migration studies, gender studies, and prison studies. In a nutshell, she is interested in the characteristics of our contemporary plural societies. Currently, her research analyses, on the one hand, Bulgarian migration to Switzerland with regard to questions of social inequality and regional disparities. On the other hand, she is working on the situation of end-of-life in prison in the Swiss context. She has conducted research financed by the Swiss National Science Foundation as well as several commissioning institutions from the public sphere.

Ueli Hostettler is a social anthropologist (PhD University of Berne; Post-doc at the University of Florida, Gainesville and the University of Texas at Austin). He is professor at the PHBern – University of Teacher Education (Berne, Switzerland) and a researcher and lecturer at the Institute for Penal Law and Criminology at the University of Berne, Switzerland. Since 2003 he has been engaged in research in the field of prison studies and has established a research group at the University of Fribourg and, since 2015, at the University of Berne (). The research of this group is based entirely on third-party funding from various sources, among them the Swiss National Science Foundation, the Federal Office of Public Health, and other institutions and foundations.

References

Achermann

Hostettler

(2007) Femmes et hommes en milieu pénitentiaire fermé en Suisse: Réflexions sur les questions de genre et de migrations. Nouvelles Quéstions Féministes 26(1): 70–88.

Breuer

Reichertz

(2001) Wissenschafts-Kriterien: Eine Moderation. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research 2(3).

Castree

(2000) Professionalisation, activism and the university: Whither ‘critical geography’? Environment and Planning A 32(6): 955–970.

Castree

(2006) Research assessment and the production of geographical knowledge. Progress in Human Geography 30(6): 747–782.

Chelimsky

(1987) What have we learned about the politics of program evaluation? Evaluation News 8: 5–22.

Chelimsky

Shadish

(eds) (1997) Evaluation for the 21st Century: A Handbook. London, Thousand Oaks, CA and New Delhi: Sage.

Cook

(1997) Lessons learned in evaluation over the past 25 years. In: Chelimsky

Shadish

(eds) Evaluation for the 21st Century: A Handbook. London, Thousand Oaks, CA and New Delhi: Sage, pp. 30–52.

Dowling

(2008) Geographies of identity: Laboring in the ‘neoliberal’ university. Progress in Human Geography 32(6): 812–820.

Elton

(2000) The UK research assessment exercise: Unintended consequences. Higher Education Quarterly 54(3): 274–283.

10.

Flick

(2000 [1995]) Qualitative Forschung. Theorie, Methoden, Anwendung in Psychologie und Sozialwissenschaften. Hamburg: Rowohlt.

11.

Glaser

Strauss

(1998 [1967]) Grounded Theory. Strategien qualitativer Forschung. Bern, Göttingen, Toronto and Seattle: Hans Huber.

12.

Guba

Lincoln

(1989) Fourth Generation Evaluation. Newbury Park, CA, London and New Delhi: Sage.

13.

Ham

(1999) Tracking the truth or selling one’s soul? Reflections on ethics of a piece of commissioned research. British Journal of Educational Studies 47(3): 275–282.

14.

Hostettler

(2012) Exploring hidden ordinariness: Ethnographic approaches to life behind prison walls. In: Budowski

Nollert

Young

(eds) Delinquenz und Bestrafen. Zürich: Seismo, pp. 158–166.

15.

House

(1993) Professional Evaluation: Social Impact and Political Consequences. Newbury Park, CA, London and New Delhi: Sage.

16.

Houston

Hyndman

McLean

Jamal

(2009) The methods and meanings of collaborative team research. Qualitative Inquiry 16(4): 285–297.

17.

Howe

(2010) Mixed methods, mixed causes? Qualitative Inquiry 17(2): 166–171.

18.

Kaufmann

(1969) Soziologie und praktische Wirksamkeit. In: Schäfers

(ed.) Thesen zur Kritik der Soziologie. Frankfurt am Main: Suhrkamp, pp. 68–79.

19.

Kirchhofer

Richter

(2012) Resocialisation as a goal of the prison system: The example of education in prison. In: Budowski

Nollert

Young

(eds) Delinquenz und Bestrafen. Zürich: Seismo, pp. 167–191.

20.

Lincoln

(2009) ‘What a long, strange trip it’s been …’: Twenty-five years of qualitative and new paradigm research. Qualitative Inquiry 16(1): 3–9.

21.

Morris

(ed.) (2008) Evaluation Ethics for Best Practice: Cases and Commentaries. New York: The Guilford Press.

22.

Naples

(2003) Feminism and Method: Ethnography, Discourse Analysis, and Activist Research. New York and London: Routledge.

23.

Power

(1994) The Audit Explosion. London: DEMOS.

24.

Reichertz

(2000) Abduktion, Deduktion und Induktion in der qualitativen Forschung. In: Flick

Von Kardorff

Steinke

(eds) Qualitative Forschung. Ein Handbuch. Hamburg: Rowohlt, pp. 276–286.

25.

Richter

Kirchhofer

Hostettler

Young

(2011) Wie eine Insel im Gefängnis. Bildung im Schweizer Strafvollzug. Tsantsa 16: 50–60.

26.

Schwandt

(2007) Expanding the conversation on evaluation ethics. Evaluation and Program Planning 30(4): 400–403.

27.

Sheppard

(2006) Geographies of research assessment: The neoliberalization of geography. Progress in Human Geography 30(6): 761–764.

28.

Simons

(2006) Ethics in evaluation. In: Shaw

Greene

Mark

(eds) Sage Handbook of Evaluation. London, Thousand Oaks, CA and New Delhi: Sage, pp. 243–265.

29.

Skaerbaek

(2007) Who’s afraid of … ethical dilemmas in qualitative research. Nora. Nordic Journal of Women’s Studies 15(1): 64–74.

30.

Steinke

(1999) Kriterien qualitativer Forschung. Ansätze zur Bewertung qualitativ-empirischer Sozialforschung. München: Juventa.

31.

Steinke

(2000) Gütekriterien qualitativer Forschung. In: Flick

Von Kardorff

Steinke

(eds) Qualitative Forschung. Ein Handbuch. Hamburg: Rowohlt, pp. 319–331.

32.

Stockmann

Meyer

Schenke

(2011) Unabhängigkeit von Evaluationen. Zeitschrift für Evaluation 10(1): 39–67.

33.

Strathern

(2000) The tyranny of transparency. British Educational Research Journal 26(3): 309–321.

34.

Torres

(2011) Public universities and the neoliberal common sense: Seven iconoclastic theses. International Studies in Sociology of Education 21(3): 177–197.

35.

Tracy

(2010) Qualitative quality: Eight ‘big-tent’ criteria for excellent qualitative research. Qualitative Inquiry 16(10): 837–851.

36.

Von Kardorff

(2000) Qualitative Evaluationsforschung. In: Flick

Von Kardorff

Steinke

(eds) Qualitative Forschung. Ein Handbuch. Reinbek: Rowohlt, pp. 238–250.

37.

Widmer

(2011) Zehn Jahre Evaluationsstandards der Schweizerischen Evaluationsgesellschaft (SEVAL-Standards). Schweizerische Zeitschrift für Kriminologie 2: 23–30.

38.

Widmer

Landert

Bachmann

(2000) Evaluations-Standards der Schweizerischen Evaluationsgesellschaft (SEVAL-Standards). Schweizerische Evaluationsgesellschaft (SEVAL).

39.

Yarbrough

Shulha

Hopson

Caruthers

(2011) The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd edn. Thousand Oaks, CA: Sage.