Abstract
This article describes the theoretical basis and characteristics of a process-based, negotiated and generative approach to the evaluation of organizations. Illustrated by a case study, it highlights the inappropriateness of standardized instruments in the face of complexity and uncertainty, suggesting ways to improve the practice of evaluating dynamic complex systems. It underlines how conducting an evaluation within an organization is an opportunity to produce knowledge that reflects on and can potentially transform existing systems. Using an action-research method, ‘instructions to the double’, the implications for data collection and analysis suited to complex interventions and complex environments, are presented and discussed. In particular, the article describes the monitoring and evaluation of practices as unfolding processes, in which evaluative methods and tools are context dependent and subject to social, dynamic and contested mobilization of knowledge. The collective negotiation and production of knowledge in collaboration with practitioners working within the organization, is considered as the necessary condition for enhancing the formative and transformative role of evaluation and supporting reflexivity in professional practices.
Introduction
The focus of this article is evaluative practice in dynamic and complex systems. It highlights the implications in terms of reflexive learning and adaptation (Ling, 2012) informed by monitoring and evaluation. Assuming Weaver’s (1948) definition of complexity (within the context of social research) as the unpredictable interaction of multiple parts, the article argues that the separating-out and definition of the key uncertainties in a dynamic complex system (Ling, 2012; Stirling, 2010) need a focused description of how that particular system works (Martin and Sturmberg, 2009). This understanding of practice entails not only a detailed description of work life, but a deeper understanding of practices and their relationships as meaningful and order-producing activities (Chia and Holt, 2008; Nicolini, 2009a). Such an understanding includes a process of negotiation that facilitates agreement on what is to be evaluated (Pawson et al., 2005; Van Stolk et al., 2011).
In order to deal with complexity we argue that:
Evaluation constitutes a complex and articulated practice that, as Patton suggests, considers both individual and organizational dimensions and involves different stakeholders. The quality and effectiveness of the delivered services depends on organizational functioning and the way in which staff and other organizational actors interpret their work.
Evaluation can provide context driven knowledge that becomes a basis for a reflexive process of organizational change, development and innovation. Thus the assessment of organizational functioning and the outputs or services it produces (based on knowledge sharing, reflection and learning) can lead to the identification of problems and criticalities, enhancing adaptation and potential solutions.
Evaluation is not a neutral procedure capable of producing standardized results, but rather an inductive process that makes it possible to analyze the system of activities, the bundle of practices and the relationships in which practitioners are involved.
The purpose of this article is to illustrate how conducting an evaluation within an organization becomes an opportunity to produce knowledge thus reflecting on and transforming the existing system of activities and practices. The article proposes a combined practice-based approach (Nicolini, 2013) that both furthers the understanding of the dynamics of specific settings and detects the ways in which systems of activity interact, thus allowing the shaping and sharing of a vocabulary for appraisal (Gherardi, 2009a, 2009b). The connection between organizational practices and evaluation stems from the claim that professional life consists of a flow of interactions (Yanow and Tsoukas, 2009), in which people are absorbed and that provides a collective and negotiated evaluative orientation. This sort of self-understanding refers to a shared body of relevant knowledge that allows organizational actors to situate themselves in relation to specific standards of excellence, constraints, rules and obligations. These ethical and aesthetic judgments are socially defined and constantly refined while being practiced, through daily negotiation on what is considered correct or incorrect, right or wrong, tasteful or distasteful.
From these starting points, the research questions that this article addresses are:
How does an evaluation become a resource for action and transformative practice?
Which characteristics of domains and contexts provide conditions for using evaluative knowledge for new practices? and
What is the role of the researcher or evaluator in triggering formative and reflexive opportunities?
Evaluation as supporting reflexive learning and transformative adaptation
Situations in which organisational players (managers, professionals, practitioners, stakeholders) have to deal with the complexity and uniqueness of problems and events are becoming increasingly frequent in the current social and economic scenario. Furthermore, the competences of organizations are often judged on the basis of the capacity of their staff to handle matters perceived as new, unusual and unsolvable on the basis of existing knowledge (Weick and Sutcliffe, 2007).
Knowledge work and knowledge workers are today crucial, with knowledge simultaneously being the input, medium, and output of work-processes. Examining the production and circulation of knowledge is therefore a critical dimension in studying and understanding organizational dynamics. This has led to new ways of perceiving, producing and managing knowledge (Brown and Duguid, 1998, 2000; Gibbons et al., 1994), and new organizational models for appropriately managing knowledge-intensive organizations (Orlikowski, 2000; Tsoukas, 1996).
The perspective described above seems to provide a new lens through which both to analyze organizational phenomena (Gherardi, 2006) and to conceive of methods to evaluate organizational processes.
This article aims to highlight the connection between the (researcher’s) practice of evaluation and evaluative knowledge, believing that this connection represents an innovative perspective conceiving of evaluation as a practice-based approach. Through a case study, we seek to demonstrate that carrying out a good evaluation is also an opportunity to improve an organization, because of the structural link between shared evaluative orientations and broader aspects of organizational life. It also makes possible the transformative capacity of taken-for-granted activities, through a process of discussion and negotiation about the meanings attributed to events and situations, roles and visions as well as to problems. It is a work of relationship-building and mutual positioning, which underlines the difference between stakeholders involved and their different interpretations and use of evaluation knowledge (Cunliffe, 2002, 2009).
Evaluation is the more relevant and actionable when it is actually engaged with those who are being evaluated and is able to generate processes of development and learning related to real problems.
We conceive of evaluation as contributing to organizational transformation by mediating the inevitable political, cultural and relational implications that are characteristic of all evaluations. This, on the one hand, means taking into account the embeddedness of evaluation in organizations whose dynamics are shaped by stakeholders with their own interests. On the other hand, our conception of evaluation involves questioning the idea that evaluation research should directly serve the decisions of policy-making and practice (Hammersley, 2013). In this regard, as Contandriopoulos and Brousselle (2012) suggest, evaluation results need to be embedded in a real-life context in order to be relevant. In particular, they point out that the use and the usability of the evaluation depend on both the distance between actors’ different readings of problems, and the balance that they attribute to the cost/benefit ratio of investing the energy and resources that evaluation requires. The process and the results of evaluation practice, therefore, depend on how different and often divergent opinions, criteria and interests are combined in a context. It is this that determines the specific conditions of knowledge-use and exchange, including the use of knowledge produced through evaluation.
Evaluating and recognizing accountabilities
Our evaluation approach builds on various literatures concerning the nature of social and organizational evaluation. Thus we regard evaluation as a social practice (Saunders, 2012), starting from the distinction between the concept of use – organizational capacity to use evaluation outputs; and usability – evaluative design can facilitate the potential use of findings. Social practice in this context, argues Saunders, is considered as the kind of routine rule-governed behavior evaluators engage in as they carry out their work. Ledermann (2011) argues in this respect for the multifaceted nature of evaluation, especially when applied to complex situations and objects. There are of course many different uses of evaluation whilst knowledge generated by evaluation is connected with practice only in subtle and often indirect ways. Patton (1997), in turn, highlights the broad scope of decisions derived from evaluative knowledge, stressing that evaluation use depends on how people apply knowledge to their specific circumstances based on their particular experiences.
The strong link between modes of use and the flow of experience in which people (and researchers) are involved, highlights the need not only to stress the different types of use of evaluation outputs (instrumental, conceptual, enlightenment, process use, justificatory, see Saunders, 2012), but also to identify the practices that can be changed and the evaluation design that can support such changes.
Conceiving of the use and usability of evaluation as practices entails an understanding of their situated and contextual features. This includes discovering how people make sense of an evaluation process and findings, and how evaluation outcomes become knowledge resources for change and action.
Our research seeks to highlight how an evaluation can stimulate and facilitate processes of action and organizational change. This starts from the assumption of a close connection between the practice of evaluation and knowledge repertoires, embodied in organizations. The evaluative practices with which we are concerned mobilize evaluation knowledge and evaluation criteria relating to values, symbols and cultures, implicitly and explicitly assumed and socially distributed within organizations.
Evaluation knowledge represents a symbolic structure of signs and meanings, and representations of value dimensions of meaning that people, in their daily lives and in their specific organizational existence, transpose and create, by participating in the flow of experience (Yanow and Tsoukas, 2009). Evaluation may in fact be conceived of as knowledge deployed when valuing an object (Beauvois, 1990), mobilizing many possible meanings that affect key capabilities such as judging, criticizing, measuring, controlling and interpreting. Each refers to situations that are widespread and lived on a daily basis, where people are constantly confronted with the need to attribute meaning to events, situations, actions and behavior. They have to resort to readings that express and implement a social intelligence, learned and used in real-life contexts and characterized by two aspects (Le Poultier, 1990, 1993). One is linked to the possible modes of information and knowledge management, with all its potential for errors and distortions; the other is linked to the idea of social action as a set of actions that arise from specific situational and organizational contexts. In this sense, organizational practices, understood as activity systems in which subjects are involved, become pivotal aspect of an evaluation.
Saunders (2012) identifies four domains of practices that give rise to typical situations for evaluation use:
the first is the national/international context, with an ‘evaluative emphasis’ on the issues of accountability and the use of evaluation aimed at a more equitable distribution of resources and improving overall performance;
the second is programmatic, linked to specific interventions and outcomes of learning that they generate, identifying the use of evaluation to support decisions for the improvement of policies, projects and initiatives;
the third is the institutional context, with a focus on evaluation within organizations that aims to analyze and improve processes, using the knowledge gained to change structures, reconfigure business processes and modify activities;
the fourth context is self-oriented, activated and undertaken by practitioners for reasons of understanding and development, sharing with others the existing problems and the identification of common arrangements for dealing with them.
This article presents an evaluation practice that is on the border between programmatic and institutional use, highlighting recognized modes of organizational practice considered by practitioners as being important and emblematic. People have articulated and plural representations of the problems they face in daily work. They cope with high levels of polarization between a plurality of positions, interpretations and interests, including different and at times contradictory representations of the relationship between costs and benefits that their actions may incur (Contandriopoulos and Brousselle, 2012).
Social actors’ commitment in this environment vary on a continuum from low to high involvement (Saunders, 2012) leading to different levels of evaluation outputs and different ways of using resources to change existing professional practices.
The need to manage different levels of involvement in putting into practice evaluative knowledge, is informed by Activity Theory (Engeström and Sannino, 2010), a theoretical perspective which adopts the metaphor of boundary crossing to describe learning processes that involve organizational actors within their spheres of activity.
Considering the usability of evaluation knowledge through an Activity Theory lens entails dealing with the routines and inertia of organizational life (Ling, 2012), identifying and coping with uncertainties, in order to understand and change the system of activities in which people operate. From this perspective, evaluation is a social and inductive practice that makes it possible to analyze the behavior, professional practices, and relationships of the actors involved in action.
This article describes an example of an evaluation in an Italian Regional government department, highlighting the ways in which methodology and theory interact within contradictory and mutable contexts. Before describing the case we first outline the methods of data gathering and collection and the process for supporting reflexive learning.
Evaluation as a process of applied social research
Our evaluative approach is a process of applied social research. The word ‘research’ refers to the production of knowledge by means of an open process of discussion, active listening, and the involvement of the practitioners working within the organization. We call this an ‘expert system’ because organizational personnel possess expert evaluative knowledge enabling them to deploy criteria, tacit beliefs and assumptions to make judgments. The word ‘social’ refers to the need for constant transaction and negotiation with organizational actors (each with their own distinct representations and interpretations) in order to agree on the aims and methods of the evaluation. The word ‘applied’ relates to the use made of the evaluative procedure and its importance for users, making evaluation an opportunity to support the growth, development and learning of organizational members.
This decentralized and situated stance necessarily requires professionals to take responsibility for their actions within the organization. They are, in fact, the creators of local knowledge, capable of learning from each other and finding the best responses to constantly changing local problems. The connection between evaluation aims and in-depth descriptions of organizational and work-place practices relies on the dynamic of particular settings and contexts to define what is conceived as relevant.
The evaluation approach presented above requires methodology and techniques able to exploit the uniqueness of specific cases and facilitate the involvement of organizational actors, who contribute to the analysis of what is to be evaluated (Ryan and DeStefano, 2000; Shaw, 1999; Simons, 1996; Stake, 1995).
The aim is to promote and support the detection of professional and occupational practices, in order to identify possible pathways of change and transformation. The methodological approach is located in specific settings, thus promoting the involvement of organizational members, the development of evaluative evidence and activation of learning as part of the evaluation. This also implies the need to support self-evaluation among professionals through the joint identification of key issues; facilitating communication processes between the different levels of the organization; and promoting critical reflection in order to find appropriate indicators and evaluation criteria.
Creating a setting that fosters connections and synergy between various stakeholders and levels of the organization allows participants to reflect on their experience, shared information and relational processes (Brisolara, 1998; Burke, 1998; Cousins and Whitmore, 1998; Garaway, 1995; Levin-Rozalis and Rosenstein, 2005; Springett, 2001).
In this sense, the evaluation is designed as a pathway to dialogue, cooperation and negotiation that can inspire and support organizational learning, identifying practices on which the participants can address sustainable change processes, since such practices and processes are local, contingent and contextual.
Research design
The case presented in the article uses techniques for the elicitation of evidence and evaluation criteria, which enhance the contribution of staff, through knowledge of their histories and professional practices. In particular, the evaluation process is agreed and co-constructed with the various stakeholders involved. The basic hypothesis is that, as argued by Saunders (2012), this allows organizational actors to share and negotiate. Such negotiation covers the reasons and motivations for the evaluation; establishing operational and strategic objectives; promoting mutual involvement; specifying methodological and work settings and the type and level of involvement for different actors and stakeholders; and agreeing on evaluation outputs.
This is achieved through the implementation of three parallel and complementary evaluation pathways. First, the evaluation of professional practices involving organizational personnel, who have expertise in analyzing organizational practices, and interpreting and representing organizational functioning. Second, the evaluation of internal functions, involving both operators and managers of the organization concerned with the integration of different contributions from a standpoint of change and transformation. Third, those engaged in the creation of an evaluation culture involving organizational actors in the reflexive process as much as in actual assessment tasks.
Cutting across these pathways, the methodological approach followed three main phases:
knowledge production – recognition of working practices by reconstructing practitioners’ professional lives;
knowledge sharing and learning – discussion and reflection on critical issues and good practices with the aim of identifying significant evaluation evidence and criteria;
planning and development – identification of actions for organizational change.
The use of ‘instructions to the double’ technique to enhance professional practice
The key tool adopted in our case-study is the ‘instructions to the double’ technique. This technique was originally developed in the 1970s within the FIAT (Italian automotive) company. The method has as its objectives: to help stakeholders make explicit their expertise and tacit knowledge; allowing them to formalize informal experience so as to make it communicable; and uncovering the assessment by professionals of their specific contexts (Oddone et al., 1977). From an operational point of view, the ‘instructions’ are based on the request to the practitioners to imagine having a double who will take their place at work the next day and provide him or her with all the information and instructions necessary to know what to do and what not to do, so that no one will be aware of personnel change. In this way, respondents are prompted to review their routines by describing their everyday actions not only in technical terms, but also in terms of attitudes, emotions and intentions.
The technique was used to support the evaluation process through a detailed analysis and representation of practices. It activates processes of reflection on these practices, in order to produce evaluative evidence, co-build the evaluation process with professionals and define possible routes of transformation, emphasizing what individuals say they think and criteria they use.
This technique is particularly useful in an evaluation process because it surfaces knowledge about what is right or wrong from the perspective of organizational actors. It creates opportunities to find evidence about technical and operational knowledge, strategic knowledge, summative knowledge that enables the subject to takes decisions with ethical weight; and rules that determine inclusion (or exclusion) in social realities (Gherardi, 1990, 1995, 2003). Nicolini (2009b) emphasizes that the discursive production that is generated by the ‘instructions’ allows the conventional moral pillars of organizational practices (what the author defines as the local ‘goods’) to be elicited, as well as the ways in which individuals negotiate and express these in everyday activity.
Finally, the representation and sharing of professional practices highlighted thanks to the instructions to the double technique allow the sharing of relevant outputs of the evaluation.
The case: A request for the evaluation of a service organization
The focus is an evaluation carried out over more than a year (between the end of 2010 and the first months of 2012) in the Social Policies Department of the Valle D’Aosta Regional administration in northern Italy. The Department coordinates and controls activities designed to ensure effective social policy is delivered by public and private bodies.
Its organizational structure is divided into four services that group together resources and social workers: two that deal with different forms of physical and mental disability, a third is a Families and Youth Service, and a fourth a Central Service combining a training office, an early childhood office, an office for the elderly, and an office that deals with those who wish to substitute civic service for national military service. The services and activities are therefore extremely diverse not only in terms of the large number of offices, but also in terms of the nature of their work, which includes dealing with heterogeneous problems that often require practitioners to take decisions on critical or urgent cases for which there are no standard guidelines.
Given recent unfavorable economic and social circumstances at national and European levels, and the increasingly complex needs of the Aosta Region, the administration has come to realize that it must develop and enhance the use of resources at its disposal. When this evaluation began there were difficulties as a result of the Department’s operational structure. Practitioners lacked knowledge of each other’s professional roles, and the functioning of the various services and offices. The Department therefore decided to initiate an evaluation that would directly involve the practitioners working in the organization. This was intended to support them identify problems and possible solutions.
Whilst acknowledging the operational importance of the issue to hand, the agreement reached between the researchers/evaluators and the Department’s Director was to develop a process of participatory research and reflection. A process of organizational evaluation was proposed, started from the analysis of the functioning of the organization and the quality of the services provided. This was intended amongst other things to enable staff to identify possible ways of improving service quality.
The evaluation procedure was implemented in order to help practitioners:
understand and monitor their functions;
control and better manage their resources;
foster a shared evaluation culture;
encourage a positive orientation to change with regard to issues identified.
The assumption (and therefore the challenge) was that better understanding of resources, problems, and related solutions, would trigger organizational innovation.
The setting of the evaluation
Three different working settings were designed:
All three settings were set up as transactional spaces between the social and individual in which people would be able to:
constitute working groups to share critical concerns of their profession and organization
learn to reposition themselves in their work by evaluating their professional experience
begin individual and collective sense-making and sense-giving processes about work practices
engage in action in support of transformation and change.
All of these aspects were a means to identify emerging uncertainties and connect practice based knowledge with an evaluative process.
Evaluation steps
Step 1 – Knowledge production
In the knowledge production phase, a first set of professional practices was identified using the instructions to the double technique. This were applied in three principal steps:
‘Imagine you have to write all the instructions necessary for a person who, tomorrow, will replace you in your role, without your colleagues noticing; the instructions are not “cold” but must contain relational and social elements. The double will learn how to work at your job as if he/she were you, not just a list of tasks to be performed …’
The objective of this phrasing was to make explicit everyday micro-actions and micro-decisions implemented by the professionals and to promote reflective thinking.
Step 2 – Knowledge sharing and learning
The shift from knowledge acquisition to learning was made possible by using representations of work practices and critical incidents. The content of the instructions to the double was analyzed both by the researchers and the practitioners and was depicted through concept maps. These maps were presented by the researchers to organizational actors and fed into subsequent discussions. From an operational point of view, the work on the maps included: a phase of individual reflection in which participants checked overall consistency and located their own point of view within a broader landscape; and a phase of discussion in plenary during which each professional made a first selection of problematic aspects to be targeted for possible change.
Step 3 – Planning and development
This step was mainly oriented to promote reflexive learning and encourage practitioners to identify solutions for criticalities/organizational problems and possible actions for innovation and change. For this reason, a ‘cross-sectional instrument’ (Gorli et al., 2012, 2015) was identified as a suitable tool, able to give an overview of the reported difficulties.
Problems were divided into four quadrants depending on their type, two of which focused on collective and two on individual issues encountered. In the first quadrant were technical and structural problems that staff faced e.g. insufficient equipment; differential access to resources or unhelpful procedures. Difficulties in the second quadrant are also collective but focusing on relationships and the social system. The third quadrant once again relates to processes, but the emphasis was on how individuals interpret their professional and organizational positions: e.g. recurrent issues concerning roles and the construction of working and personal identity. The fourth quadrant also covers individual problems in relationships to the organization’s structural features e.g. difficulties in understanding, adapting or using particular technologies. These often required individuals to develop technical and organizational skills.
On this basis staff proposed a series of actions aimed at reorganizing working processes and improving the organization as a whole (See Table 1).
Examples of ‘instructions to the double’.
Findings and results
The first phase of knowledge production, was the collection of a large number of stories /instructions that displayed a rich repertoire of technical, relational and procedural professional practices. Included below are several extracts from the data which illustrate the scope of these practices.
The most important findings were related to:
Critical issues that characterized professional activities
Lack of internal resources (human, financial, physical space)
Processes related to the daily management of work (difficulty in defining priorities; difficulty in organizing work and emergency management)
Gaps in internal communication (between operators in the same service and between different services) with consequent repercussions in the relationships with users
‘I never know what I need to do in the morning: during the evening I plan to do some activities that, the day after, I cannot do because of various problems that occur, the emergencies, the number of calls to the public that I get, the requests from the manager etc …’ ; ‘… Know that you will get many phone calls that have nothing to do with what you do. This is because of colleagues in other offices telling the user to contact you when it is really the responsibility of another operator …’
Intrinsic characteristics of the profession/constraints
Heterogeneity of the profession in terms of the number of internal and external stakeholders, diversity of issues and activities
Characteristics of the user that are configured to be particularly difficult to manage
Emergencies and urgent tasks that are a specific characteristic of the profession and its daily activities
‘… so I cannot really tell you what you have to do tomorrow because everything is urgent! …’;
‘… I make phone calls and talk to the psychologist, the psychiatrist, Caritas, the Municipality, the Department …’ ; ‘… By the way, procurement is to require the CUP [Unique medicine center for reservation] for the ministry, they are to publicize the calls and then the results in the Official Gazette. Then there is a relationship with the office of the Region of contracts …’; ‘… something unexpected may happen, such as when the user arrives suddenly and is particularly angry …’
In the second phase, of knowledge sharing and learning, the analysis of the instruction to the double content highlighted sets of professional practices that were subsequently represented by the researchers / evaluators in concept maps. These included:
the organization’s structural components;
use of formal and informal artefacts;
relations with colleagues and users;
ethics and values;
strategies used to manage and organize work;
work situations requiring ‘flexible thought’ and rapid action.
The synthesis represented in the maps helped the operators to: voice questions and doubts; indicate and share existing good practice; and identify possible proposals and solutions.
Voice questions and doubts ‘But what is meant by an “emergency”? When the user doesn’t know where to go and sleep at night and we have to act quickly, or something else? … No, because emergencies differ greatly from service to service and from office to office, they aren’t all the same.’
Indicate and share existing good practice ‘… in our office we’ve introduced the use of text messaging to give users information and updates. It’s very useful because the messages arrive directly on the users’ cell phones, so that you’re sure they’ll read them, because they don’t have to go and look for them but we send them directly.’
Identify possible proposals and solutions ‘… one thing that could be done to solve the problem of complex and rigid bureaucratic constraints would be to give greater autonomy to the heads of services so that they could decide by themselves whether or not to keep certain procedures.’
In the last phase of the evaluation, emerging practices were then discussed using the cross section framework that distinguishes between individual and collective needs and technical/procedural and the processual. The goal was to organize and prioritize: ‘naming’ aspects considered to be most relevant and then identifying priorities for action.
This step identified proposals for management and resolved critical issues such as:
The setting of deadlines for projects, calls for tender, competition announcements, and initiatives in general, with the aim of making the information available and the main source for the management of everyday work (short term goals / activity to be implemented in all the services)
The creation of a mapping of skills, roles and activities of staff (medium-term objective: 6 months)
The establishment of a secretariat unique for all services, whose structural features were determined by the operators themselves through the creation of the mapping described above (long-term goal: 1 year).
It is worth emphasizing how the three steps above matter in order to test one of the assumption underpinning our research questions: that the involvement of people as ‘system experts’ in the investigation of their activities enhances the possibility of identifying critical issues and uncertainties. This also clarifies the meaning of a real time evaluation, connected to crucial issues and problematic features. These features, that unfold and stem from specific and situated contexts, improve the potential relevance and meaningfulness of the knowledge in planned and facilitative ways. The use of specific instruments like the ‘cross sectional tool’ help people to position themselves and their practices within the complexity of their organizational life. This instrument also makes it possible to review and transform traditional practices in use, thus enhancing the reflexive learning potential of the evaluation.
Discussion of the case
The evaluative procedure within the Social Policies Department, departed markedly from a logic that conceives of evaluation as a standardized program intended to yield value judgments endowed with scientific validity. On the contrary, it was practice-based (the knowledge was jointly produced with the organization’s practitioners), negotiated (agreements were reached on the aspects to be evaluated), and generative (the conditions were created for critical reflections on professional practices and possible changes).
The evaluation action described in the case is characterized by the presence of a plurality of stakeholders, goals and different interests, requiring careful attention to a number of factors that affect the use and usability of the knowledge that evaluation produces.
First, the evaluation process involved researchers as external evaluators in the construction of the process itself and negotiation of agreements and decisions with agency personnel. The use of qualitative methods oriented to the production of knowledge helped elaborate organizational practices, based on situated knowledge, close to the real-world of participants. It was therefore useful for promoting organizational change and development and evaluation outputs that were practical and actionable.
Second, a culture of evaluation that led to the decision to invest in a collaborative process with professional staff produced a transparent analysis of the organization’s work that made public not only its outputs and outcomes, but also its problems and difficulties. This made it essential for front-line professionals to form alliances and agreements with institutional stakeholders such as middle management.
Third, institutional pressure, generated by the economic situation, and the consequent need to produce data on its accountable activities increased the organizational members’ involvement in the evaluation process and ultimately the use and usability of the evaluation. On the other hand there will always be those who are indifferent and disengaged which reduces the potential use of evaluative knowledge.
Fourth, the strategies and means of communication used by the evaluators, as well as by institutional stakeholders, were important. For example the coordination group, in which managers, facilitators and researchers/evaluators worked in an integrated manner performed a vital role bringing together different parts of the research.
The implementation of these four aspects required constant positioning by researchers/evaluators in the face of a complex and challenging context.
Conclusions
The link between the practice of evaluation and evaluation of practices lies in the use of potential knowledge outputs for change, improvement and adaptation. Evaluation, from this perspective, requires that the researcher/evaluator should be immersed in the context of a specific organization, facilitating dialogue and adapting methodologies and operational logic to make it possible. It is an evolving enterprise within a stream of experience which is both articulated and complex, calling for interpersonal relations and different dynamic processes, introducing not just positive opportunities but also elements of entropy and chaos.
Finally to return to the research questions posed at the beginning of this article:
‘How does an evaluation become a resource for action and transformative practice?’
The ability to recognize repertoires of practice in use through qualitative methods that involve organizational actors, facilitates the use of the knowledge produced for transformation and change. In particular, it emphasizes the alliance of individuals and employing institutions, encouraging investment in the evaluation and increasing the potential use of evaluation outputs.
Which characteristics of domains and contexts provide conditions for using evaluative knowledge for new practices?
We conceptualized the context in terms of artefacts, settings and actions that offered potential resources for new practices. Using the perspective of boundary crossing (Engeström, 2004; Saunders, 2012), the internal facilitators can be conceived as ‘boundary spanners’ or ‘brokers’ (Allen, 1977; Carlile, 2002; Cook and Brown, 1999). Their role was not just facilitating interaction between different actors, but also the promotion of positive investment and engagement in the functional use of knowledge produced through evaluation. The practitioners groups are in quintessential ‘boundary’ roles, able to intercept and circulate knowledge related to different operating environments. The use of qualitative approaches to the recognition of situated knowledge facilitated the sharing of knowledge and the legitimation of each other’s competences and roles.
What is the role of researcher or evaluator in triggering formative and reflexive opportunities?
The role of the researcher/evaluator is engaged rather than neutral within the approach we adopted. Her/his role is that of a conveyor of conversations and exchanges among individuals with divergent opinions, interests and preferences. Her/his effort is to facilitate the involvement of users as co-producers, by jointly structuring with them the knowledge that is considered relevant or indeed expendable.
Under these conditions an evaluative process assumes a formative role. Evaluating services from within an organization means working within the organizational practices as they are used as part of everyday professional life.
The methodology described in this article is useful for generating questions, mobilizing resources (time, space, budget, people), encouraging different parties to invest in the process, initiating a system of activities involving multiple stakeholders (executives, managers, employees, trainers) and facilitating innovative transformation (production of internal data, training activities, moments of shared reflection). It provides knowledge resources for change and development.
The evaluation of complex activity systems requires specific ways of conceiving and implementing data collection and analysis, in order to improve reflexivity in an uncertain and often problematic context. This entails three specific points of focus that can serve to summarize some of the key findings of the evaluative experience this article has addressed.
The first is an epistemological point, relating to the connection between organizational practices and evaluation as a collective and negotiated process that people shape and share in organizational contexts and then use for appraisal. This is embedded and embodied knowledge that is socially defined and constantly refined through practice.
The second point is methodological and relies on how a practice-based perspective can help to focus and identify key uncertainties and criticalities, in order to provide sustainable and realistic knowledge for both learning and accountability. Evaluating a complex system in real-time means dealing with situated, dynamic and socially constructed knowledge that emerges through on-going interaction. The main challenge for the researcher is that researching from within (Shotter, 2008, 2010), is understanding, jointly with organizational actors (front-line staff, practitioners and managers), what is considered as a relevant problem (uncertainties, criticalities, questions to be solved) in the lived experience of organizational life.
The third point refers to the organizational and institutional dimension: the adoption of an ad hoc methodology, close to organizational settings, requires an explicit alliance with board and managerial stakeholders in order to facilitate an evaluation. A willingness to deal with the production of local and situated knowledge, via field notes, narratives and accounts cannot to be taken for granted. It requires political legitimation, institutional support and attention to different epistemic and professional cultures within the organization.
The availability and sustainability of the three dimensions described above are essential conditions to conceive, launch and realize an evaluation that aims to have a formative role, supporting reflexive learning and improving activities adapted and transformed by participants.
In a famous painting (The Human Condition, Washington, National Gallery) Magritte places an easel on which a landscape is painted in front of a window that overlooks the very same landscape. It is unclear whether the landscape we see is part of the painting on the easel or is of the actual landscape outside the window. Evaluation as we have approached it, like Magritte’s ‘painting within a painting’, allows us to represent an organizational landscape in ways that reveal and assess many details of organizational reality. Of course some ambiguity always remains: unavoidable yet essential in evaluation as in other spheres of endeavor.