Procedural and participatory ethics: community-based evaluation in practice

The original contribution the article makes to practice:

• A project steering group that provided ongoing ethical review is described, including examples of ethical issues it considered, and discusses this in relation to literature about ethics in practice.

Background

Institutional Review Boards (IRBs) assess the ethical merit of evaluation proposals and retain the right to approve, require amendments or disapprove them in advance, especially when the evaluation includes working with vulnerable communities and/or the evaluation findings could have significant implications for policy, programmes or services (Ethics Guidelines Working Group, 2020). That is an important and necessary step as it helps ensure that evaluators have considered the risks and benefits of their projects before they get underway. In this article, I discuss the evaluation of an organisation called SafeMan SafeFamily to illustrate how ethic matters are emergent, cannot be predicted in advance and require ongoing attention. The evaluation was commissioned by the New Zealand government’s Ministry of Social Development. Our project steering group provided a useful forum where the project’s key stakeholders – the funder, organisation being evaluated and a community leader – could, alongside the evaluation team, deliberate on the matter and decide the best way forward. The steering group served to build trust between the project stakeholders, help transfer findings into Ministry of Social Development policy and decision-making, deepen evaluation insights and resolve any emerging issues and associated ethical considerations.

SafeMan SafeFamily is a registered charity that aims to end family violence in Aotearoa New Zealand. SafeMan SafeFamily takes a “peer-led, professionally supported” approach to helping men who have used violence but are now committed to becoming violence-free make that change. Rather than delivering a ‘course’ or ‘curriculum’, SafeMan SafeFamily focuses on holistically supporting men through a process summed-up as ‘Uncover, Discover, Recover’. Working through this process involves men uncovering what has happened to them and what they have done, discovering what they can do to address unresolved trauma and the drivers of their violence, and then put those learnings into practice. SafeMan SafeFamily works with adult men of all ethnicities, including Māori, Pasifika, Pākehā/New Zealand European, and other ethnicities. They come from a wide range of socio-economic backgrounds, including no educational qualifications, higher-degrees, unemployment and professional careers. Some have spent years in prison. What they all have in common is a history of using violence and a strong desire to become violence free – their participation in SafeMan SafeFamily is voluntary.

SafeMan SafeFamily is developing a reputation in the family violence sector for getting good results with men and families who typically feel excluded from mainstream social services – including those who have gang affiliations and criminal records. SafeMan SafeFamily founder Vic Tamati, a Samoan man, grew up in a violent home in New Zealand, started using violence himself and eventually became a gang leader. But after yet again perpetrating violence on his wife and children, he realised he needed to stop using violence but didn’t know how. Things weren’t changing even after attending numerous stopping violence programmes. Vic started talking with other men who, like him, wanted to become violence-free but couldn’t break the cycle about what they needed to change. Out of these conversations emerged the SafeMan SafeFamily approach (as briefly described in the paragraph above) which has helped Vic stay violence-free since 2008.

Given the organisation’s unique approach and growing reputation, the Ministry of Social Development determined to fund a 2-year process and outcome evaluation to systematically assess how SafeMan SafeFamily helps men desist from violence and the difference this makes for men and their families. The author – who identifies as Pākehā/New Zealand European – collaborated with a Māori evaluation and social research consultancy services provider, Hector Kaiwai (www.awaassociates.co.nz) to deliver the project. The focus of this article is on the approach we took to ensuring the evaluation was undertaken ethically.

Procedural ethics and community-based practice

The relational nature of community-based research and the unpredictability of how things unfold over the course of a project can sometimes sit uneasily with ethics committees’ desire for surety. Committees look for detail about who, what, where, when, how, and risks and protections. These are all good and important questions, but in changeable environments, and when community leaders want, expect and have the right to say how things will proceed, it can be very hard if not impossible to provide IRBs with certainty. It is easier to anticipate how a project will proceed in a context where it is expert-driven and the project parameters are stable or can be controlled, as in the positivist research tradition, which is where the ethics review process comes from (Ljungberg, 2021).

Ethics committees emerged from the medical and behavioural sciences community’s response to ensuring unethical research never happened again, such as the Nazi WW2 experiments, Thalidomide trials, the Tuskegee Syphilis study and New Zealand’s own “Unfortunate Experiment” (UNLV Division of Research, 2022; Coney & Bunkle, 1987). In contrast to work done in the positivist tradition where research parameters are controlled, it is hard to predict how community-based work will unfold over time because what happens in and around the project setting is not controlled. Given this, procedural ethics committees’ expectations can sit uncomfortably with community-based work, but the participatory research literature “abounds with examples of ethical issues in the field and practical strategies to resolve these, such as ensuring co-researchers are involved in each project phase” (Lenette, 2022, p. 80).

The procedural review process has an important role to play to ensure research and evaluation is ethical from the outset, but IRBs are grounded in the positivist tradition. However, evaluating a community-based programme can be unpredictable, which means carefully considered plans can quickly become unsuitable given emergent needs, preferences and interests, which is what happened during the SafeMan SafeFamily evaluation. To help ensure the evaluation remained on track and was conducted ethically from start to finish a project steering group was formed involving community-leaders and representatives from the funding agency (the Ministry of Social Development).

Providing communities with an opportunity to meaningfully participate in determining what ethical practice looks like is discussed in the participatory research literature (Bussu, 2021; Lenette, 2022) and the culturally responsive evaluation literature (Chouinard & Cram, 2020; Hood et al., 2015; Newcomer et al., 2015). It is also consistent with the New Zealand Productivity Commission’s call for ‘a fair chance for all’, which includes ‘… an opportunity for whānau and those experiencing persistent disadvantage to have agency as commissioners and leaders of evaluation’ (New Zealand Productivity Commission, 2022).

Ethical oversight in practice

Taking a participatory approach to evaluation does not necessarily require that a project’s stakeholders form a steering group. But it does require that relevant stakeholders are meaningfully involved in determining due process (Kemmis et al., 2014; Sette, 2021). Stakeholder involvement needs to have a clear purpose in order for the project, and its stakeholders, to benefit from their involvement. Their involvement does not need to be formalised, but in the case of SafeMan SafeFamily, having project stakeholders providing project governance, including ethical oversight, delivered great benefit, especially given the depth of SafeMan SafeFamily’s leaders’ knowledge about their peer-community, the strong relationships they have with the men and many of their families, and their intimate knowledge of the SafeMan SafeFamily approach. Examples of the benefits of this approach to the project are discussed below, with particular attention to ethical practice.

The Ministry of Social Development’s Ethics Committee acknowledges that research and evaluation plans are liable to change, but they nonetheless and justifiably want to know that there is a plan, that it has been carefully thought through, and that it is ethically sound. As stated by the Ministry of Social Development:

Ethical assessment of research and evaluation requires addressing and balancing a number of potentially competing ethical principles, perspectives and interests. The balance may not be easy to identify at the outset of the study, and it may change over the course of the study as more information comes becomes available, the context for the study changes, or changes are required to the study design (Ethics Guidelines Working Group, 2020, p. 6).

The Ministry of Social Development’s Ethics Committee approved the formation of a steering group that would meet monthly to provide oversight of the SafeMan SafeFamily evaluation. As per the Committee’s standard practice, the Committee would make itself available should it be needed for subsequent review or advice. The steering group included two representatives from the Ministry of Social Development, including the evaluation contract manager and a subject matter expert (family violence and violence cessation), and three SafeMan SafeFamily board members, including founder Vic Tamati, a registered psychologist, and a psychiatrist. The steering group’s responsibility was ensuring the evaluation was achieving the project’s goals. While this included ensuring that what SafeMan SafeFamily does and how is portrayed accurately, including in project reports, it did not extend to authoring those reports – that was the evaluation team’s responsibility.

The evaluation team (including this article’s author and Hector Kaiwai) reported monthly to the steering group about what had happened, what was coming up, any issues that had arisen and what was learnt. The emerging findings were used by the steering group members on an ongoing basis, to inform family violence sector policy being by the Ministry of Social Development and by SafeMan SafeFamily in funding applications. The steering group helped the evaluation team resolve issues and considered and approved any significant changes proposed to the evaluation plan (two significant changes are discussed below). The regular, transparent reporting and rich discussions that took place in the steering group meetings built shared understanding between the evaluators and steering group members and trust in the evaluation process.

There was the option to return to the Ministry of Social Development’s Ethics Committee if anyone in the steering group or evaluation team felt unsure about how to address an ethical issue, but no one ever felt the need to. The initial evaluation plan, as approved by the Ministry of Social Development’s Ethics Committee, was drafted in good-faith but once the evaluation team got underway the plan soon changed. For instance, to recruit interviewees the plan involved SafeMan SafeFamily screening out men and women who were acutely traumatised or unwell, briefing those who were well about the evaluation, and passing on the names of those who were interested in being interviewed to the evaluation team. However, Vic Tamati told the evaluation team that a better way to recruit men for interview would be for the evaluators (the author and Hector Kaiwai) to join the men’s discussion groups so they could meet and get to know each other in a safe environment. The men could then decide for themselves if they wanted to be interviewed. This suggested approach was discussed and endorsed by the steering group.

For months, no one volunteered to be interviewed and the evaluation timeline was blown. The steering group was able to approve the extended timeline for approving the interviews and that the evaluators kept going to the group meetings. Trust and rapport developed as the men became more comfortable with the evaluators and the project. As trust and rapport developed, the men started to come forward for interview. Given the evaluators had met the men many times (in group meetings over several months), there was a level of comfort and familiarity between them that would not have existed otherwise which meant the interviews provided rich information. Information sheets and consent forms as approved by the Ministry of Social Development’s Ethics Committee were shared with the men (discussed further below).

Other changes needed to occur to respond to the growing interest of families in the evaluation process. The plan approved by the Ministry of Social Development’s Ethics Committee stated that SafeMan SafeFamily’s community worker would inform the men’s partners, wives and (adult) sons and daughters who were well about the evaluation and pass the details of those who were interested in being interviewed to the evaluation team. It also stated that families would be interviewed by a woman. Yet, because the men on the evaluation team had established trust and rapport with the SafeMan SafeFamily community, the family members said they would prefer to be interviewed by them. When the evaluation team reported to the steering group that families were coming forward and expressing a preference to be interviewed by the men, this change was endorsed.

As noted, no one individual decided on the suitability of changes to the evaluation plan; this was the role of the project steering group. To us as evaluators, this collaborative approach is consistent with Aotearoa New Zealand Evaluation Association Standards, which states that evaluative practice should be ‘methodologically responsive, appropriate, trustworthy [and] undertaken with care’ (ANZEA & SUPERU, 2015, p. 17). To put this into practice in our case meant a commitment to shared decision-making with the evaluation commissioner – the Ministry of Social Development – and representatives of the community of interest – SafeMan SafeFamily – when determining what ethical practice would look like throughout the project. The evaluators committed from the outset to not impose or require a certain way of working on SafeMan SafeFamily. How we did things would be – and was – decided in partnership at the level of the steering group. This helped the evaluation team ensure we were well informed, carefully considered, and responsive to the culture of the programme we were evaluating and the needs and interests of the men and their families. In our view, SafeMan SafeFamily’s membership on the steering group helped address an equity issue in evaluation – the marginalisation of community voices from research and evaluation design (Hood et al., 2015). But as Lenette (2022) notes, participatory ethics is not inherently ethical, especially if evaluators ignore the intricacies of ethics in practice and fail to be reflexive in their approach.

The research and evaluation ethics literature includes two approaches to addressing moral dilemmas: the first is impersonal and rests on information, logic and law; the second is personal and based on communication and relationships (Bussu, 2021). The second approach is consistent with the SafeMan SafeFamily project, whereby the evaluation team worked with the funder and SafeMan SafeFamily leaders on an on-going basis to ensure the evaluation proceeded ethically. It was helpful for the SafeMan SafeFamily project team to set out an initial plan and have it reviewed by an ethics committee, and it was beneficial to know there could be follow-up with a committee manager if we needed it. But what mattered most was ongoing reflexive practice supported by frequent participation in the men’s discussion groups and open and trusting relationships between the evaluation team and steering group. In our monthly meetings, the evaluation team and steering group members deliberated on any ethical matters that had arisen during the month and considered how to best address any concerns that had arisen or were arising to ensure the SafeMan SafeFamily’s community members (men and families) were kept safe. This is consistent with the notion that ethical practice, especially in longer-term projects, requires building and maintaining trust and rapport between researchers, project stakeholders and community members (O’Reilly, 2009).

The steering group members, especially on the SafeMan SafeFamily side, had intimate knowledge of the community being evaluated which enabled a richly informed consideration of matters that had arisen (preference for the evaluators to join the meetings, families presenting themselves for interview without being invited to, a man interviewing women). SafeMan SafeFamily’s steering group membership gave the organisation an on-going and meaningful voice and share of decision-making power about how the evaluators should respond to any matters arising. Ministry of Social Development’s membership ensured the funder’s view was considered and the psychiatrist and psychologist provided points of view informed by long-standing connections to SafeMan SafeFamily alongside other professional affiliations and responsibilities. As noted, the steering group and evaluation team never had substantive disagreement or felt the need to defer back to the Ministry of Social Development’s Ethics Committee for a second-opinion or advice.

Even though the sometimes-messy nature of community-based and participatory research tends to disrupt carefully considered plans, which need reconsidering given local happenings and circumstances, a plan does need to be in place. Stakeholders and participants need to be clear about the purpose, risks, benefits and protections, but it may be best that the details about process (who, what, where and when) allow for flexibility, because while not every evaluation project may be as unpredictable as the SafeMan SafeFamily project, plans are liable to change when working in community settings (Patton, 2011).

While detailed information sheets may give an IRB comfort that the plan they approved is being communicated to potential project participants, in this project the information sheet was often updated so it was consistent with changes to the evaluation plan, as approved by the steering group As best we could tell, few interviewees thoroughly read the information before providing their consent. Given that 40% of adults (in New Zealand) have limited reading ability (Radio New Zealand, 2018), project documentation (consent forms and participant information sheets) should be appropriate to the audience, kept as short as possible, highlighting the key points, and written in plain language so it is easy for project participants to understand. Most SafeMan SafeFamily interviewees preferred to hear from us what the project was about and how their information would be used and gave their consent verbally (audio-recorded), which emphasises the importance of evaluators making themselves personally available to talk to those who want to understand a project so they can be fully informed.

Conclusion

Community-based evaluators must contend with complexity as a matter of course (Patton, 2011) and they also need to be responsive to those communities’ cultural mores (Chouinard & Cram, 2020; Hood et al., 2015), which is why it is argued here that it would be helpful for IRBs to reconsider their role and the requirements of evaluators working in community settings. It has been shown here that while ethics review by an IRB was helpful for getting the SafeMan SafeFamily evaluation underway with a thoroughly considered evaluation plan, the plan soon became unsuitable given the dynamism of the project’s community setting. It was therefore better suited to include ethical discussion and oversight at ongoing, monthly steering group meetings involving community leaders. Keys to the steering group’s success were:

• The community leaders’ strong relationships with the men and families the evaluators were working with

Evaluators working in community settings need to be responsive to emerging local needs, interests and preferences, many of which cannot be predicted in advance. Given this, if a project is set up in such a way as to provide for local community oversight throughout the project lifecycle, I contend it is reasonable for IRBs to devolve responsibility to sufficiently resourced and supported community leaders who have intimate knowledge of the people and what is happening in the community through the project lifecycle, while remaining available for institutional review should disagreement, intractable or serious ethical concerns arise while the project is underway. This is a shift away from the positivist approach to research and evaluation from which IRBs emerged, but is consistent with ensuring that communities in which evaluation is taking place are given meaningful say and opportunities to participate in determining what ethical practice looks like. This is consistent with participatory (Bussu, 2021; Lenette, 2022) and culturally responsive (Chouinard & Cram, 2020; Hood et al., 2015; Newcomer et al., 2015) approaches to evaluation, both of which highlight the importance of attending to the relationships in, and the dynamics of the context in which evaluation is taking place.