Hitting the bullseye: Learning to become a reflexive monitor in New Zealand

Abstract

Reflexive monitors (RMs) have been found to be vital to the success of co-innovation projects. While the practices utilized by RMs have been examined in various contexts, we examine the roles they have played in a new cultural context in New Zealand (NZ) and how it has been possible to embed these roles in a diverse range of innovation projects in the primary sector. This article will address this gap in terms of explaining the case-specific behaviours that have been utilized in six different co-innovation projects in the NZ agricultural innovation system. Qualitative data from interviews with five RMs will be used to argue that RMs are a key component in the co-innovation process and are required to play diverse roles depending on project circumstances to enhance system innovation – for example, devil’s advocate, project supporter, consensus seeker, conflict mediator, critical enquirer or encourager. The findings have implications for the characteristics that make a good RM in terms of openness to new ideas, facilitation and critical thinking skills and how they report on the practice of monitoring a project reflexively utilizing monitoring and evaluation techniques.

Keywords

Co-innovation agricultural innovation systems innovation projects

Introduction

The concept of co-innovation is a mechanism to link collaboration and innovation in order to solve complex problems (Lee et al., 2012). Co-innovation involves understanding that each actor has a role in designing their future, as opposed to the technology transfer approach, where science determines innovations that are then passed onto end users with little or no feedback (Mylan et al., 2015). In fostering co-innovation, the fulfilment of facilitator roles is seen as important (Kristiansen, 2014; Nettle et al., 2013). In this article, we focus on a form of facilitation called reflexive monitoring. In a seminal resource on applying reflexive monitoring, van Mierlo et al. (2010b: 11) provide the following definition regarding the position of a reflexive monitor (RM) in a project:

[A RM] encourages participants to keep reflecting on the relationships between the key items: the ambitions of the project, usual practices and the way they are embedded in the institutions, plus the developments in the system that offer opportunities for realising the ambitions of systems innovation.

Originally developed in the Netherlands (van Mierlo et al., 2010b), the RM role is not widespread in New Zealand (NZ) where formal RM use is only beginning to emerge (Rijswijk et al., 2015). Earlier work has shown that implementing new roles oriented at enabling co-innovation in agricultural innovation systems (AISs) requires considerable role changes of actors (Minh et al., 2014; Nettle et al., 2013) and bears risks of failure when not properly adjusted to the context (Klerkx et al., 2017; Reid and Brazendale, 2014), especially when a co-innovation approach is enacted in an AIS with a predominantly linear technology transfer orientation such as the NZ AIS (Turner et al., 2016). This article explains how forms of the RM role have developed in a different context than where it was developed, through an analysis of RM practice in the Primary Innovation programme in NZ. This programme aimed to shift thinking on research and innovation in the NZ AIS towards transdisciplinarity and co-innovation (Botha et al., 2014). Application outside of the context in which it was developed has not been examined in great detail before, and thus, can provide important considerations for those interested in incorporating a RM role into future co-innovation programmes. In the following section the theoretical framework the article utilizes is presented. The research method and results are then presented. Finally, the discussion and conclusions section links back to literature to reflect on the findings and show the broader implications of the study in terms of the enactment of RM roles and their embedding in the AIS.

Theoretical framework

Different forms of a RM role utilizing action learning

Reflexive monitoring in action (RMA) is theorized to support system innovation, and Arkesteijn et al. (2015) report that there are various forms a RM position can take at certain points in time. These include observer, facilitator or even criticizer that works to link ambitions, practices and subsequent project developments (Arkesteijn et al., 2015). Importantly, in regard to the workload of a RM, it is also a requirement that they do not fulfil too many other tasks within the project so they can maintain focus on broader systemic change (van Mierlo et al., 2010b). Figure 1 shows a continuum from ‘appreciative inquiry’ through to ‘critical analysis’ to highlight the extremes of where a RM can act, depending on project circumstances. It should be noted, however, that RMs can sit anywhere along the continuum and their positions might be altered by changes over the project lifetime. Appreciative enquiry involves encouraging the project team to build momentum. On the other hand, critical enquiry involves questioning the project team and the barriers to project outcomes. Both are examples of facilitation techniques (Kristiansen, 2014). The subsequent findings utilize RM perceptions to indicate the diversity of action on the spectrum (Figure 1).

Figure 1.

Spectrum of reflexive monitoring attitudes and differences according to approach (adapted from van Mierlo et al., 2010b).

At each stage of the RM cycle, an action learning process can be followed to determine how a RM might act

Observe: The process of observation draws on multiple forms of evidence from body language, facial expressions, tone of voice, interpersonal communication, language used, content of the conversations, short interviews, conversations, structured participant reflections and secondary data sources (Dick, 1991; Forester, 1999; Kitchin and Tate, 2000). van Mierlo (2013, personal communication) found that successful RMs were typically experienced facilitators. As a consequence, they are familiar with structuring small group processes of dialogue and decision-making.

Analyse and evaluate: All the data collected during the previous stage can undergo thematic analysis (Flick, 2009). The depth of analysis depends on the speed at which the cycle is moving; the faster the cycle, the quicker the thematic analysis. The key questions during analysis are:

Are these behaviours and actions consistent with the co-innovation principles? (i.e. will it assist the project overcome/change any potential barriers to success within the system?)

What will the likely impact of the observed behaviours, actions or practice be on the ambition for change if no intervention occurs?

What is driving the observed behaviour, practices and action?

van Mierlo et al. (2010b) and Nederlof et al. (2011) provide insights into what behaviours and system characteristics are desirable and which may hinder systemic change. This literature and the RMs previous facilitation experience provide a reference point for RMs against which to evaluate behaviours and activities within the project.

3. Reflect: Once the data have been analysed, reflection can occur on how behaviours, practice or activities could be altered (or current practice strengthened) to enhance the ambition for change or generate systemic change. Each option should be carefully evaluated based on the benefits and costs of its application. Who is involved in the reflection will depend on the speed at which the cycle is moving; the faster the cycle is moving, the fewer people involved. If the cycle is occurring rapidly, the RM may be the sole reflector. Reflection should be structured to assist the project team to achieve their ambition for change by mitigating systemic failures (Nederlof et al., 2011; van Mierlo et al., 2010a).

4. Act: All actions and interventions should be undertaken by the most suitable person and will depend on the nature of the issue. For example, it may be the RM in a meeting setting or the project manager in consultation with other project members. How these actions occur will need to be negotiated with the project team at an early stage of the project. There is a wealth of literature and practice which may inform RMs’ choice of action and the benefits and trade-offs associated with each alternative (Chambers, 2002; Chevalier and Buckles, 2013; Dick, 1991).

Methodology

Primary data were gathered using semi-structured interviews with the five RMs in each of the six co-innovation projects in the Primary Innovation programme (one interviewee acts as RM for two projects). Data were collected in late 2015, after each RM had been in their role for approximately 3 years. The co-innovation projects included a project examining heifer rearing in the dairy industry, an integrated forestry sector project, a nutrient management project involving the dairy industry, an irrigation scheme project, a project aiming to reduce a pest in tomato and potato crops and a project looking at broader systemic change within the NZ AIS. Interview questions were developed to gather perceptions on the RM role each interviewee had played, key moments in their RM practice and key challenges involved with the role. Also included were context-specific questions regarding the project the RM was involved in. The questions asked were modified to suit the RM being interviewed, allowing them to talk more specifically about significant issues relating to each innovation project in a more reflexive manner (Beers and Bots, 2009).

Results

The role of RMs in NZ is analysed here through the lens of RM practitioners. Early within the initial analysis phase of the co-innovation programme (Primary Innovation) which included six co-innovation projects in which the RM role was applied, a gap was identified between RM theory (van Mierlo et al., 2010a, 2010b) and practical applications of the RM role in NZ. In particular, the experience and tools being used by RMs, and complexity in regard to the expectations of the RM role within each project, presented challenges (Rijswijk et al., 2015). The following seven steps provide a timeline for the events that indicate how the RM role has developed in the Primary Innovation programme. Of primary importance was the iterative process that has been utilized in order to enhance learning and increase the alignment of the project tasks with the meeting of project objectives. Simultaneously, significant effort has gone into the utilization of more developed theoretical understandings of the RM role in the Dutch context, utilizing expertise from Wageningen University and Research Centre (WUR) and subsequently applying that knowledge to the NZ AIS.

Research programme begins with six projects requiring RMs (October 2012): initial confusion, questioning of the RM role.

Regular monthly meeting of NZ RMs begins (April 2013).

van Mierlo (primary author of RMA methodology document) visits NZ (August 2013) providing a key question for RM practice: ‘what is the ambition for change in the project?’

RM trip to WUR (April 2014), including RM workshop with van Mierlo, highlights diversity in RM role and similar questions from others working in the co-innovation space.

Projects changed names from ‘case studies’ to ‘innovation projects’ (July 2014): key shift in thinking for the research team, reflected in terminology.

NZ RM workshop (July 2014): bullseye diagram developed, and ‘RM guide’ started.

Reflection from RMs via interviews (late 2014/early 2015).

Below we summarize the experiences and lessons as to how the RMs coped with these challenges.

What does a RM in NZ do?

Based on the experiences of those operating as RMs in the six cases, it was clear that there was no ‘one size fits all’ definition or approach to reflexive monitoring, as there were examples across the entire spectrum identified by van Mierlo et al. (2010b) (Figure 1). All those interviewed agreed that the role is about supporting the project manager and team to achieve the project goals; ‘a supporting role but a critical supporting role’ and was ‘a role that doesn’t get much recognition’. Other aspects of the role are identified in Table 1. As one RM noted:

Table 1.

Definition of a RM by current RMs.

Tasks	Description
Supporting role (supporting project manager and wider project team)	‘Support work done by project team’ ‘Find ways to get the group to agree, not everyone will agree but everyone has to be able to live with it’ ‘There to help take temperature gauge, let the project leader know how the process is going’ ‘Picking up different things from what a project manager would pick up’
Get the project team where it needs to go	‘It doesn’t have to be a straight line, it can be a bit wobbly, because it will be, constantly assessing against what you’ve said you want to achieve, and how are we going towards it’ ‘Always asking why’ ‘Asking the question “is the project on track?”’ ‘If the direction is changing, do they realize?’ ‘Keeping them on track towards the goal’
Identifying conflict	‘Don’t get involved in the conflict between members…highlight conflict to project manager’ ‘Mediate conflict…if that’s what project manager wants from you’
Data collector/evaluator	‘Making sure the project is tracking along’ ‘Figuring out what is causing blockages’
Facilitator	‘Facilitate project meetings’
Providing feedback	‘Two different parts, devil’s advocate and pushing hard and looking for positives and building support’ ‘Offer opinions throw things back at them to think about’
Identifying the right stakeholders to be involved and valuing their knowledge	‘Making sure everyone’s knowledge is continually included’ ‘Making sure the right people are involved at the right time’ ‘Having everyone’s knowledge heard’ ‘If someone is missing ask why’

RM: reflexive monitor.

you adapt your skills to the role, and RMs require certain personality traits and mindsets rather than particular skills…[they must be] open to other viewpoints; [have a] strong team mentality and want to see collaboration and co-learning outcomes.

Over the life of the research programme, there have been shifts in thinking about the RM role. At the beginning of the project, there was confusion and questioning, as to whether the RM was essential and how it was different to being a good facilitator. As the programme began and there was interaction across projects and with other RMs (notably workshops with van Mierlo), there was acceptance of the RM role, albeit with some confusion about how exactly it was to be undertaken in specific projects. van Mierlo (2013, personal communication) provided an initial question to help guide RMs ‘what is the ambition for change?’

Applying RM principles to inform decision-making in co-innovation projects

RMs can analyse and reflect on co-innovation projects based on the steps of the action learning cycle (van Mierlo et al., 2010b) (Figure 2, inset). Reflection and action should be structured to assist the project team and achieve their ambition for change by mitigating systemic failures (Nederlof et al., 2011; van Mierlo et al., 2010a). It was also found that the RM cycle in Figure 2 is useful for deciding when a RM should intervene in a project to uphold co-innovation principles.

Figure 2.

Action learning cycle that could be implemented by a RM derived from research findings. RM: reflexive monitor.

RM influence and challenges

During the most recent workshop involving NZ RMs (step 6 in the timeline), it was found that not all characteristics identified in Table 1 have the same level of influence on the role of the RM as others. A diagram was subsequently developed through discussion of the question: ‘what influences the way we work and our ability to have impact as reflexive monitors?’ Each RM contributed their personal list of items. The workshop was used to identify similar issues and highlight which of these were under the control of the RM. The resulting ‘bullseye’ diagram is shown in Figure 3. In the bullseye are aspects that the RM has the most control over, as the circles expand the RM has decreasing influence on these aspects of the role and may find it can be unproductive to attempt to address these concerns (Figure 3). The central controllable aspects of a co-innovation project are of primary importance in regard to choosing individuals to take on a RM position. Follow-up interviews (after the most recent workshop) revealed that RMs believed the most important requirements for the role were personal skills, a good relationship with the project manager, having a support network, having a clear job description and having freedom to experiment. Similarly, although the RM has less of an influence over the outer rings (Figure 3), these aspects of the project can still present concerns. For example, the challenges a RM will face in the role will be influenced strongly by:

Figure 3.

The bullseye highlighting things a RM can control with decreasing influence.

The project they are involved with:

The project might go against the RMs own principles (e.g. personal concerns about genetically modified organisms);

The project leader and their expectations of the role:

How the project leader defines the role;

What the project leader expects of the RM (e.g. interventionist role, sit back and observe or somewhere in between?)

The RM’s personality:

May not be comfortable taking on an interventionist role or being passive/reserved;

The RM having another role in the project (e.g. they also provide technical expertise or are conducting social research):

This may cause tension between how the RM is seen by the project manager, project team and how the RM sees the role;

The RM working in the same organization as the project leader:

May make it harder to be objective or critical;

The RM working in a different organization to the project leader:

May not be aware of the political climate the project leader is operating in;

May not understand how the project leader’s organization works;

May be working with different company structures and hierarchies;

The physical proximity to the project team:

It may be expensive and/or time-consuming to attend all meetings;

It may take longer to build up trust;

There can be a lack of opportunities for informal interaction.

Although some of these challenges may be out of the control of the RM, those that cannot be controlled need to be managed in a way which assists the project team and its partners to achieve their ambition for change. RMs indicated that it is extremely important as a RM to reflect on all these factors in order to inform practice and identify potential future risks to the project. The primary focus of the RM should be on what they can do to encourage and support the application of co-innovation principles within their projects. These principles were built from previous work on co-innovation and RM theory and are based on taking time to understand the problem, being inclusive, valuing various sources of knowledge, being open and honest, sharing a vision, sticking with the process, being flexible and being aware of the wider context (Nederlof et al., 2011; van Mierlo et al., 2010b). The interview findings suggest that RMs collect data, provide feedback, support the project and are critical when required. The 13 lessons learnt (Table 2) also make this point quite clear. At all times, RMs are required to foster relationships with various actors using components of a broad toolkit which can now be found online (AgResearch Limited, 2016).

Table 2.

Lessons learnt from the experiences of RMs.

Lesson	Examples
Build a relationship with project manager	‘Need honest conversations around expectations’ ‘Regular communication’ ‘Need their buy-in’ ‘Help facilitate you into the group’ ‘Establish a relationship of trust and rapport with the project leader tough and very direct discussions will come up’ ‘It’s a hard road to get the project leader to realize things need to be done differently – there is a fine line between being seen as helpful and being seen to be interfering’
Define the role	‘Work with project manager to define the role and what their expectations were’ ‘Important to ask this as you are there to help them’ ‘Don’t go in and say what you think, project manager has to have buy-in’ ‘Need a clear definition of the role at the start’ ‘You must remain disconnected from the project – it is not your project, you need to remain apart from it in order to see it clearly’ ‘Must have the skills to “speak the truth kindly” and remain dispassionate when those who are personally involved get defensive when you touch a nerve’
Use accessible terminology	‘Jargon doesn’t work…is a barrier…use laymen terms’
Be flexible in your approach	‘Be willing to try any approach – think creatively about methodologies’ ‘It takes a lot of time – more than you think’
Have open communication channels	‘Always be willing to see another point of view, and encourage others to see other points of view also’ ‘Things won’t happen the first time you bring it up – keep telling the same consistent message until they are heard’ ‘Give consistent messages’
Have a support network	‘To talk to and off-load’ ‘Don’t necessarily need solutions from them’
Monitor and evaluate	‘Part of your role’ ‘Helps you understand/track what is going on’
Provide feedback	‘Can only identify change, you cannot make change happen’ ‘You point out the behaviours needing change and actions that must be taken, but cannot make them change, only support them to change’ ‘If change isn’t occurring, or they disagree, then you need to be able to self-evaluate and accept that you might be wrong on this one’
Specific training is required	‘Facilitation training’ ‘Conflict resolution’
Build trust	‘With the project manager’ ‘With project team members’
Use different strategies according to participants	‘Interview team members individually, as this allows them to get across the real institutions and attitudes that are driving the team culture, as well as highlighting what they believe the key problem or ambition for change is’
No right way to do the role	‘Best advice I got was from another RM – just make a start, just do something…it is very difficult to know what to do as a RM, so it is literally taking a step out and hoping a stepping stone presents itself so you can go forward’ ‘Context specific – approach role differently based on a number of factors’
Have a buddy	‘Someone to learn from’ ‘Talk things through with…doesn’t mean giving you answers’

RM: reflexive monitor.

Discussion and concluding comments

This article analysed the role of RMs in regard to six NZ innovation projects applying a co-innovation approach, and it was found that a RM is required to be prepared for various situations, depending on the individual project, the stakeholders involved and the broader context in which the project sits, as they will influence the actions that can be taken. The RM in each co-innovation project allowed for the actors involved to create space where they could enact their own collective transitions towards project outcomes (in line with Audet (2014)). Our study highlights the important role of a RM in regard to encouraging innovation and shifting the mindsets of project stakeholders. As Cohen and Ilieva (2015: 201) explain, the ‘complexity and uncertainty of a transition make it difficult, if not impossible, to deliberately engineer’. The changes in project team understanding and thinking brought about by RMs playing the variety of roles mentioned earlier, for example, devil’s advocate, project supporter, consensus seeker, conflict mediator, critical enquirer or encourager (Figure 1), create the potential to significantly transform the direction of the project and associated outcomes. The practical lessons learnt from RMs in the co-innovation projects in NZ provide evidence of the changing nature of approaches taken by RMs to tackle complex problems by reflectively considering the actions of those involved throughout an innovation project (Table 2). In line with van Mierlo et al. (2010b), there is no ‘right way’ to be a RM, you need to be flexible, strategic and work on relationships. Simultaneously, a RM should primarily focus on the aspects of a project they can influence (the middle of the bullseye in Figure 3) as that will be most productive, hence most likely to alter the innovation project and/or AIS they are embedded in. By further developing the RM continuum (Figure 1), to highlight how and when it might be best to intervene (Figure 3), this work has also furthered understanding around pragmatism in RM practice. The NZ context within which the innovation projects were conducted provided a novel example of utilizing RM techniques through a process of action learning.

In terms of the embedding of RM roles within an AIS in which it has not been previously applied, the article provides some points for reflection. Specifically, that there are resources available for RMs to consult, in order to maximize the impact of their contribution to the innovation project/s within which they act (AgResearch Limited, 2016; Rijswijk et al., 2015; van Mierlo et al., 2010b). This finding is supported by work elsewhere on embedding co-innovation support roles (such as a RM) in AIS (Klerkx and Nettle 2013; Klerkx et al., 2017; Minh et al., 2014; Nettle et al., 2013), which show that new roles such as a RM need to be institutionally supported and properly resourced. Furthermore, in understanding that the roles played by a RM can be diverse and context specific, it is also important to be realistic about what a RM can achieve within an innovation project team. In fact, while the results explained the perceived roles in regard to the RM experience in NZ, the most important contribution of this article is that, as with any relatively new role description, being open and honest about expectations from the start of a project is critical. Importantly, the RM should be explicit about what role they will play within the innovation project. The RM can then allocate appropriate time and energy to fulfilling these duties and not be overburdened with other aspects of project management and delivery. In this manner, RMs can focus on the process of creating the space for reflection within the project team in a more specific manner, as opposed to fulfilling multiple duties that spread them too thin to properly fulfil the RM role.

Footnotes

Acknowledgements

The authors would like to thank Neels Botha, Lan Chen, Laurens Klerkx, Penny Payne and three anonymous reviewers for their feedback on this manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The authors acknowledge funding for Primary Innovation by the Ministry of Business, Innovation and Employment grant (CONT-30071-BITR-AGR) and co-funding received by New Zealand dairy farmers through DairyNZ (RD1429).

References

AgResearch Limited (2016) Reflexive monitoring. Available at: http://www.beyondresults.co.nz/PrimaryInnovation/Reflexive-Monitoring/Pages/default.aspx (accessed 4 May 2016).

Arkesteijn

van Mierlo

Leeuwis

(2015) The need for reflexive evaluation approaches in development cooperation. Evaluation 21(1):99–115.

Audet

(2014) The double hermeneutic of sustainability transitions. Environmental Innovation and Societal Transitions 11:46–49.

Beers

Bots

PWG

(2009) Eliciting conceptual models to support interdisciplinary research. Journal of Information Science 35(3):259–278.

Botha

Klerkx

Small

. (2014) Lessons on transdisciplinary research in a co-innovation programme in the New Zealand agricultural sector. Outlook on Agriculture 43(3):219–223.

Chambers

(2002) Participatory Workshops: A Sourcebook of 21 Sets of Ideas and Activities. London: Earthscan.

Chevalier

Buckles

(2013) Handbook for Participatory Action Research, Planning and Evaluation. Ottawa: SAS2 Dialogue.

Cohen

Ilieva

(2015) Transitioning the food system: a strategic practice management approach for cities. Environmental Innovation and Societal Transitions 17:199–217.

Dick

(1991) Helping Groups to be Effective: Skills, Processes & Concepts for Group Facilitation. Chapel Hill: Interchange.

10.

Flick

(2009) An Introduction to Qualitative Research. London: Sage.

11.

Forester

(1999) The Deliberative Practitioner: Encouraging Participatory Planning Processes. Cambridge: MIT Press.

12.

Kitchin

Tate

(2000) Conducting Research into Human Geography: Theory, Methodology and Practice Harlow. Harlow: Prentice Hall.

13.

Klerkx

Nettle

(2013) Achievements and challenges of innovation co-production support initiatives in the Australian and Dutch dairy sectors: a comparative study. Food Policy 40(0):74–89.

14.

Klerkx

Seuneke

de Wolf

. (2017) Replication and translation of co-innovation: the influence of institutional context in large international participatory research projects. Land Use Policy 61:276–292.

15.

Kristiansen

(2014) Facilitating innovation in networks composed of non-mandated relations. International Journal of Action Research 10(1):34–53.

16.

Lee

Olson

Trimi

(2012) Co-innovation: convergenomics, collaboration, and co-creation for organizational values. Management Decision 50(5):817–831.

17.

Minh

Friederichsen

Neef

. (2014) Niche action and system harmonization for institutional change: prospects for demand-driven agricultural extension in Vietnam. Journal of Rural Studies 36:273–284.

18.

Mylan

Geels

Gee

. (2015) Eco-innovation and retailers in milk, beef and bread chains: enriching environmental supply chain management with insights from innovation studies. Journal of Cleaner Production 107(1):20–30.

19.

Nederlof

Wongtschowski

van der Lee

(2011) Putting Heads Together: Agricultural Innovation Platforms in Practice. The Netherlands: KIT publishers.

20.

Nettle

Brightling

Hope

(2013) How programme teams progress agricultural innovation in the Australian dairy industry. The Journal of Agricultural Education and Extension 19(3):271–290.

21.

Reid

Brazendale

(2014) Insights from the New Zealand experience of farmer first research. Outlook on Agriculture 43(3):213–217.

22.

Rijswijk

Bewsell

Small

. (2015) Reflexive monitoring in New Zealand: evaluation lessons in supporting transformative change. Evaluation Journal of Australasia 15(4):38–43.

23.

Turner

Klerkx

Rijswijk

. (2016) Systemic problems affecting co-innovation in the New Zealand agricultural innovation system: identification of blocking mechanisms and underlying institutional logics. NJAS - Wageningen Journal of Life Sciences 76:99–112.

24.

van Mierlo

Arkesteijn

Leeuwis

(2010a) Enhancing the reflexivity of system innovation projects with system analyses. American Journal of Evaluation 31(2):143–161.

25.

van Mierlo

Regeer

van Amstel

. (2010b) Reflexive Monitoring in Action. Wageningen/Amsterdam: Communication and Innovation Studies, WUR.