Assessing the quality of a deliberative democracy mini-public event about advanced biofuel production and development in Canada

Abstract

The importance of evaluating deliberative public engagement events is well recognized, but such activities are rarely conducted for a variety of theoretical, political and practical reasons. In this article, we provide an assessment of the criteria presented in the 2008 National Research Council report on Public Participation in Environmental Assessment and Decision Making (NRC report) as explicit indicators of quality for the 2012 ‘Advanced Biofuels’ deliberative democracy event. The National Research Council’s criteria were selected to evaluate this event because they are decision oriented, are the products of an exhaustive review of similar past events, are intended specifically for environmental processes and encompass many of the criteria presented in other evaluation frameworks. It is our hope that the results of our study may encourage others to employ and assess the National Research Council’s criteria as a generalizable benchmark that may justifiably be used in forthcoming deliberative events exploring different topics with different audiences.

Keywords

discourses of science energy policy public consultations public participation

1. Introduction

Evaluation is a vital step towards improving practice in the field of deliberative democracy as outlined in the Nine Principles of Effective Deliberative Public Engagement produced by Involve, the National Consumer Council (2008) and Diane Warburton. Yet despite this broad recognition, evaluations are rarely conducted for a variety of reasons. Rowe et al. (2005) identify a range of theoretical, political and practical issues that pose difficulties for those who wish to evaluate public engagement events and highlight the process of ‘selecting evaluation criteria’ as a core challenge. In this article, we provide an assessment of the criteria presented in the 2008 Nation Research Council report on Public Participation in Environmental Assessment and Decision Making (NRC report) as explicit indicators of quality for the ‘Advanced Biofuels’ deliberative democracy in-person event. The results presented here contribute to the deliberative democracy literature produced by Fishkin et al. (2000), Kahane et al. (2010), Rowe et al. (2005), Einsiedel et al. (2001) and others, and should be of interest to those who work in public policy and citizen engagement both within and outside the environmental domain. It is our hope that our assessment may encourage others to examine the NRC criteria as a generalizable benchmark that may justifiably be used in forthcoming deliberative events.

The 4-day ‘Advanced Biofuels’ engagement took place over two non-consecutive weekends in fall 2012 at Concordia University in Montréal, Québec¹ and was divided into three phases (see Section A1 and Table A1 in Online Appendix). This event was the ninth in a series of public engagement efforts that have been conducted using this approach since 2007 on a range of topics including salmon genomics and biobanking (Burgess, 2014). Participants completed a pre- and post-test measures online survey a few weeks prior to and immediately following the event. The objective of this deliberation was to obtain the views of 26 citizens on the need for advanced lignocellulosic biofuels in Canada and, if a need existed, what would constitute a more socially acceptable approach to its development and production. Advanced lignocellulosic biofuels was defined as new biofuels made from plant cell walls (lingocellulose) obtained from feedstocks such as agricultural straws and forestry residue, as opposed to municipal solid waste and algae or first-generation biofuels² made from food crops (World Bank, 2010). In the following sections, we outline the methods employed during this assessment followed by results and conclusions. In order to remain succinct, we focus primarily on the quantitative results of our assessment process supplemented with some qualitative analysis of participant-observations where appropriate.

2. Overview of the National Research Council evaluation criteria

The 2008 NRC report offers guidance in the form of ‘best processes’ by clarifying evaluative criteria for public engagement events conducted within the environmental domain. The indicators of event quality presented in the NRC report are shown in Table 1 and analysed in detail in the results section of this article (see NRC, 2008: 71). Certain illustrative criteria listed in the NRC report require long term follow-up of participants or scientists and are therefore beyond the scope of the grant funded Advanced Biofuels event (as indicated * in Table 1). As this study was not conducted by a government office and had no official mandate, our team was unable to assess how the results influenced the technical analyses of experts or the perceptions of nonparticipants. These criteria will therefore not be addressed in this article. However, we plan to address these remaining criteria in future engagement efforts.

Table 1.

Three types of results of environmental decision processes with illustrative indicators.

1. Quality of assessments or decisions

(a) Concerns expressed by publics were addressed in analysis
(b) Information was added; more information was considered in the process
(c) Technical analyses were improved*
(d) Outputs reflected a broad view of the situation that addressed all issues considered important by participants
(e) Conclusions were based on and consistent with the best available evidence
(f) Innovative ideas were generated for solving problems

2. Legitimacy of process and decisions

Pre-existing conflict was reduced or dissent clearly acknowledged and dealt with:
(a) Mistrust among participants, including government agencies, was reduced
(b) Participants accepted the assessment or decision process as having conformed to standards of sound analysis and decision making, even if they did not agree with the final assessment or recommendation for action
(c) The assessment or decision was widely accepted, even among nonparticipants*
(d) Participants went outside the process to overturn its results, for example, with legal challenges or attempts to influence legislation (a negative indicator)*

3. Capacity for future decisions

(a) Public participants became better informed about relevant environmental, scientific, social and other issues
(b) Participants and public officials gained a better understanding of each other*
(c) Public officials gained skill in organizing decision processes*
(d) Participants gained skill in participatory decision making
(e) Scientists gained understanding of public concerns*
(f) Scientists developed, or committed to develop, new data or methods*

The evaluation results presented in this article contribute to the body of work currently underway within the field of deliberative public engagement (for a review of debates on evaluation, see Abelson et al., 2013 and Supplementary Appendix Section A.2). In general, these evaluation criteria seek to advance the important goal of improving how public engagement events are translated into effective policy, how they can improve their representativeness and transparency and ultimately help elicit social learning. The NRC criteria were selected to evaluate the Advanced Biofuels event because they are decision oriented, are the products of an exhaustive review of similar past events, are intended specifically for environmental processes and encompass many of the criteria presented in other evaluation frameworks.

3. Methods

Participant sample and pre–post measure online surveys

Demographic filters for the participant sample included gender, educational level, age, religious affiliation and identification as a member of a visible minority group. We were seeking to assemble a mini-public (Goodin and Dryzek, 2006) comprised of 26 citizens who did not have a vested or special interest in biofuel production but who would instead represent a wide range of views and perspectives relevant to biofuel production (Longstaff and Burgess, 2010). The sample was intended to resemble City of Montréal demographics as detailed in the Census Canada 2006 Community Profiles data. Similar to Rask et al. (2012), we employed a pre- and post-test survey to evaluate the ‘Advanced Biofuels’ event. All participants, with one exception,³ completed the entire event and all associated tasks including the pre- and post-measure online surveys. The pre- and post-test measure online survey questions were drafted to address the NRC evaluation criteria as well as the abbreviated version of Dunlap and Van Liere’s New Environmental Paradigm (NEP) scale, and learning and opinion changes over time (see also Supplementary Appendix Section A.3). A link to the pre-event online survey (along with a confidential ID code) was emailed to participants prior to the event and completion of the survey was mandatory in order to participate in the in-person portions of the deliberation.

4. Evaluation results

Quality of assessments or decisions (NRC indicator 1)

By the end of the first weekend, 21 of the 26 participants (81%) voted that they were supportive of advanced lignocellulosic biofuels production in Canada given certain considerations and recommendations. Both event organizers and participants were careful not to exclude the concerns expressed by participants (see indicator 1a in Table 1), especially those of the five participants who remained opposed to the development of advanced biofuels in Canada. In fact, a large group deliberation session about alternatives to biofuels was added to the second weekend of the event in order to capture these views. In addition, during each small group session on the first weekend of the event, small group facilitators invited participants to brainstorm lists of concerns, followed by benefits associated with advanced biofuel production. The various trade-offs and consequences associated with each of these concerns and benefits were then discussed by participants in both small and large group settings. In this way, a rich discussion of the wide array of issues associated with advanced biofuel production would provide a context for subsequent deliberations about public policy recommendations for Canadian policy makers.

In total, the participants spent over 18 hours in small and large group sessions deliberating about advanced biofuels. This culminated in a set of final ratified recommendations made by participants on the fourth day of the event that were highly complex, containing three main recommendation areas with a total of 13 bullet points and 38 sub-bullet points (see Figure 1). The full recommendations are not presented here, but in brief, addressed both the views of those who supported and opposed advanced biofuel production by including recommendations such as, ‘Fostering a public desire to reduce overall fuel consumption and make more efficient use of fuel, so that the development of advanced lignocellulosic biofuel does not simply promote increased fuel usage’ and ‘The establishment of an independent group to consider how best to measure the risks and benefits of advanced lignocellulosic biofuels in Canada and from a global perspective’.

Figure 1.

Summary of the three participant recommendations to ensure the social acceptability of advanced lignocellulosic biofuel development in Canada.

In the post-event survey, all but two participants agreed strongly (n = 18) or somewhat (n = 5) that the group’s final recommendations reflected a broad view of the situation that addressed all issues considered important by participants (indicator 1d). Although not specifically defined in the NRC report, we argue that participants also generated innovative ideas for solving problems (indictor 1f) through final recommendations that sought to balance key debates in the literature with original solutions (Figure 1; cf. Ribeiro, 2013). For example, suggestions included combining an original and relatable Canadian icon (e.g. David Suzuki Foundation) or mascot (e.g. BioBeaver) with an independent assessment group that would be established to foster best practices in developing advanced biofuels while promoting awareness of biofuels from a critical perspective.

The conclusions reached by participants were sensitive to the best available evidence (indicator 1e) on advanced biofuels. After completing the mandatory pre-event survey, all participants received an in-depth 20-page background information booklet that covered a wide range of topics relevant to advanced biofuels including the science and technology of biofuels, historical and contemporary challenges and a range of other topics. The booklet was written by the organizing team of social scientists based on an extensive literature review and feedback from relevant experts who hold divergent points of view in order to present the widest possible range of opinion to participants (for more information about the booklet, please see Supplementary Appendix Section A.4).

Information was also added and considered during the process (indicator 1b) through the presentations of expert speakers and an experiment conducted with participants. The experts presented information to participants on issues such as biofuel policy and economic and industry perspectives on the first day of the first and second weekend (see Supplementary Appendix Section A.4). In the post-event survey, 88% of participants rated the expert speakers as good (n = 11) or excellent (n = 11). The experiment, which demonstrated how enzymes decomposed plant material, was intended to provide participants with a hands-on learning experience relevant to biofuel production and was led by two scientists who work with enzymes for biofuel production. Participants’ opinions about this experiment were not quite as favourable as other aspects of the event with one participant rating this process as very poor, eight rating it as neutral, nine as good and seven as excellent.

Legitimacy of process and decisions (NRC indicator 2)

We addressed the following criteria: pre-existing conflict was reduced or dissent clearly acknowledged and dealt with in a number of ways during the Advanced Biofuels event. Although most (81%) participants were supportive of advanced lignocellulosic biofuel production in Canada given certain considerations and recommendations by the end of the first weekend, 5 of the 26 (19%) participants were not. As discussed earlier, participants wanted to ensure that the views of those opposed were also carefully documented during the deliberative process. After a group discussion, it was suggested that we include a separate deliberative session on the final day of the event in order to capture the views of those opposed to biofuel production in Canada (see Supplementary Appendix A.5). The concerns of these participants were also captured in the detailed criteria listed under each of the group’s final recommendations.

In addition to asking participants to ratify their final group recommendations through an in-person voting exercise, we also invited participants to rate the event during the post-event online survey at the conclusion of the event the following day. We were interested in whether mistrust among participants and the process had been reduced (indicator 2a) and if participants accepted the assessment or decision process as having conformed to standards of sound analysis and decision-making, even if they did not agree with the final assessment or recommendation for action (indicator 2b). The evaluation portion of the online post-event survey began by asking participants to rate the entire 4-day deliberative event overall. A total of 25 participants completed each of these questions. Most (80%) reported that the event was a little (n = 6) or not at all (n = 14) stressful. All agreed that the event was somewhat (n = 5) or very (n = 20) enjoyable; somewhat (n = 1) or very (n = 24) worthwhile; and somewhat (n = 2) or very (n = 23) interesting overall. However, when we asked how difficult the event was for them personally, responses were mixed with about one-third stating that it was somewhat (n = 7) or very (n = 1) difficult.

Responses were also positive when asked to rate the processes employed during the deliberative event and the final group recommendations (see Figure 2). For example, when participants were asked if the processes that led to the group’s final recommendations were fair, 23 reported that they were very fair while only 2 stated that they were somewhat fair (n = 1) or not at all (n = 1) fair. Most also agreed that the processes were very (n = 23) or somewhat (n = 1) trustworthy with one exception. However, when we asked participants if they were willing to abide by the group’s final positions even if they personally had a different view, responses were not quite as favourable. For this question, 17 (68%) stated that they were very willing to abide by these positions and 6 stated that they were somewhat willing, while 2 reported that they were only a little or not at all willing to abide by these positions. However, in this case, somewhat willing could be interpreted to mean that these participants have reservations about or perhaps do not fully agree with all of the recommendations, but because they view the process as legitimate are nonetheless willing to support (perhaps begrudgingly) the full positions of the group.

Figure 2.

Rating the process employed during the event.

Capacity for future decisions (NRC indicator 3)

Evidence that participants became better informed about relevant issues (indicator 3a) can be found in the group’s final recommendations for Canadian policy makers and self-reported confidence levels in their knowledge of biofuels. It is interesting to note that despite the fact that most were frequent news consumers, participants had little awareness of biofuels overall. In the pre-event online survey, most or 81% of participants rated their level of knowledge of biofuels as low. However, by the end of the event, these participants now mostly rated their knowledge levels as moderate (n = 13) or high (n = 10) and were able to develop and ratify highly complex recommendations that covered a range of environmental, scientific, social, economic, regulatory and other issues (see Figure 1). The three key issues addressed by these recommendations were selected by the group from a combined master list of 71 concerns and 51 benefits as part of the agenda setting exercise at the end of the first weekend. In the post-event survey, 88% of participants viewed this agenda setting exercise favourably rating it either as good (n = 10) or excellent (n = 12).

The recommendations were ratified in a final session of the event to assess the collective agreement of the participants on each point. The post-event survey and transcripts reveal that participants were able to develop these complex group recommendations because of the skills that they had gained in participatory decision-making (indicator 3d) throughout the event such as being able to listen and consider other points of view even if different from their own. In the post-event survey, participants were asked a series of questions that addressed specific aspects of the deliberative event that foster participatory decision-making (see Figure 3). All participants reported that the event objectives were either somewhat (n = 7) or very (n = 18) clear; and all stated that they felt somewhat (n = 2) or very (n = 23) included in and somewhat (n = 2) or very (n = 23) respected during the deliberation. The majority of participants also rated their interactions with others favourably with 92% rating their discussions with other participants as good (n = 8) or excellent (n = 15).

Figure 3.

Rating specific aspects of the deliberation.

Furthermore, almost all participants stated that they felt somewhat (n = 2) or very (n = 22) heard or listened to during the deliberation with the exception of one participant. In fact, participants strongly encouraged each other to share their views (including persistent disagreements) and make their voices heard throughout the event. As stated by one participant,

… and I hope people are going to put these pointed and very sharp concerns to them [referring to the expert panel] and hear what they have to say. And if they come up with something that you don’t believe in, you gotta let them know that too.

The positive nature of these responses corresponds well with the fact that when asked to rate the deliberative event overall, all participants rated it as either excellent (n = 22) or somewhat good (n = 3) and all stated that they would be somewhat (n = 5) or very likely (n = 20) to attend an event like this one again in the future, including those who had been opposed the development of advanced biofuels in Canada. It is important to note that the one participant who found that the processes that led to the groups final recommendations were not at all fair, or trustworthy, and was not at all willing to abide by the group’s final position, nonetheless also reported that they would be very likely to attend an event like this again in the future and rated the event overall as excellent. This finding and subsequent analyses regarding the motivations of dissenters and what deliberative democracy researchers can do to prevent the attrition of these individuals and encourage them to participate fully in all aspects of public engagement events deserves further scrutiny.

The final group recommendations also underscored the importance of participatory decision-making to future citizen engagement events, which included efforts such as investing in education. When asked to provide advice about how best to engage the public in future events, participants rated most methods (e.g. media news articles and town hall meetings) as very or extremely important with the exception of telephone opinion polling surveys. When asked specifically if it was sensible (in their opinion) for researchers to rely on a deliberative event like the one they had just completed when trying to develop Canadian policy: 22 agreed, 2 disagreed and 1 stated that they did not know.

5. Discussion and conclusion

The evaluation results presented here demonstrate that the Advanced Biofuels deliberative democracy event scores high on many of the illustrative indicators listed under the NRC process-oriented criteria, which offer clear and robust measures for assessing quality. However, one of the perceived limitations of our assessment could be that the evaluators were also involved in certain organizational aspects of the deliberative event. It is worth acknowledging that one of the authors was brought in as a consultant specifically due to her experience in evaluation and was not a member of the core team. While we recognize that arm’s length evaluation of public engagement events certainly helps to avoid the perception of bias (although social scientists regularly evaluate their own work when presenting research results), it is also important to engage evaluators who have in-depth knowledge of the methods and goals of deliberative public engagement from the outset, which may make identifying potential evaluators who have expertise in both areas and are truly unknown to the core team difficult. Nevertheless, due to the recognized politics of public engagement events on contested topics like advanced biofuels, where scientific, industrial and political interests intermingle, it is important to consider how to build future indicators that address bias in the assessment process.

It is recognized in the literature that heterogeneity and ambiguity over how to operationalize evaluations does exist (Abelson et al., 2013), and that this is perhaps limiting the attention paid to the development and testing of agreed-upon indicators. We hope that others will be encouraged to build on what was learned in this mini-public deliberative study and experiment with other engagement activities in this and other topic domains. Future research efforts that address the NRC criteria not addressed here would be highly beneficial (see Supplementary Appendix Section A.6). For example, it would be worthwhile to analyse how the results of deliberative events impact nonparticipants, scientists and policy makers, and how participation in these types of events influences the behaviour of participants over time by tracking future engagement in community- or policy-related activities or voting habits (as stated in Powell et al., 2011a). We would also recommend including motivations for participation in future assessments as they are not specifically addressed in the NRC indicators.

Footnotes

Acknowledgements

The ‘Advanced biofuels: A public deliberation’ event was completed by the Genozymes-GE3LS research team. The Genozymes-GE3LS team designs and tests methods to support public involvement in social and environmental issues relating to biotechnology, genomics and bioenergy. Research team members providing essential theoretical and logistical support for the event are Elyse Amend, Gabriela Capurro, Haluk Dag, Patricia Hanney, Ernest Hoffman, Shereen Joseph, Holly Longstaff, Anna Meshcherova, David Secko, Wendy Smith, Geoffrey Tobin and Shirley Tran.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was funded by Genome Canada and Genome Quebec.

Notes

Author biographies

Holly Longstaff is a partner at Engage Associates Consulting Group and the Interim Associate Director for the Office of Research Ethics at Simon Fraser University. She is also a Research Associate with the Department of Journalism at Concordia University and an Ethicist for the BC Cancer Agency Research Ethics Board. Her research examines ethical issues from a social science perspective.

David M. Secko is an Associate Professor in the Department of Journalism at Concordia University. He currently runs the Concordia Science Journalism Project () and his research covers the analysis of new models of science journalism and processes of deliberative public engagement with science and technology issues.

References

Abelson

Blacksher

Boesveld

Goold

(2013) Public Deliberation in Health Policy and Bioethics: Mapping an emerging, interdisciplinary field. Journal of Public Deliberation 9(1): 5.

Burgess

(2014) From ‘trust us’ to participatory governance: Deliberative publics and science policy. Public Understanding of Science 23(1): 48–52.

Einsiedel

Jelsøe

Breck

(2001) Publics at the technology table: The consensus conference in Denmark, Canada, and Australia. Public Understanding of Science 10(1): 83–98.

Fishkin

Luskin

Jowell

(2000) Deliberative polling and public consultation. Parliamentary Affairs 53(4): 657–666.

Goodin

Dryzek

(2006) Deliberative impacts: The macro-political uptake of mini-publics. Politics & Society 34: 219–244.

Kahane

Weinstock

Leydet

Williams

(2010) Deliberative Democracy in Practice. Vancouver, BC, Canada: UBC Press.

Longstaff

Burgess

(2010) Recruiting for representation in public deliberation on the ethics of biobanks. Public Understanding of Science 19(2): 212–224.

National Consumer Council (2008) Deliberative public engagement: Nine principles. Available at: http://www.involve.org.uk/wp-content/uploads/2011/03/Deliberative-public-engagement-nine-principles.pdf (accessed 7 May 2013).

National Research Council of the National Academies (2008) Public Participation in Environmental Assessment and Decision Making (ed Dietz

Stern

). Washington, DC: The National Academies Press. Available at: http://www.nap.edu/catalog.php?record_id=12434

10.

Powell

Colin

Kleinman

Delborne

Anderson

(2011a) Imagining ordinary citizens? Conceptualized and actual participants for deliberations on emerging technologies. Science as Culture 20(1): 37–70.

11.

Powell

Delborne

Colin

(2011b) Beyond engagement exercises: Exploring the U.S. National Citizens’ technology forum from the bottom-up. Journal of Public Deliberation 7(1): 1–47.

12.

Rask

Worthington

Lammi

(eds) (2012) Citizen Participation in Global Environmental Governance. London: Earthscan Publications.

13.

Ribeiro

(2013) Beyond commonplace biofuels: Social aspects of ethanol. Energy Policy 57: 355–362.

14.

Rowe

Horlick-Jones

Walls

Pidgeon

(2005) Difficulties in evaluating public engagement initiatives: Reflections on an evaluation of the UK GM Nation? Public debate about transgenic crops. Public Understanding of Science 14(4): 331–352.

15.

World Bank (2010) Second-generation biofuels: Economics and policies. Available at: https://openknowledge.worldbank.org/bitstream/handle/10986/3891/WPS5406.pdf?sequence=1 (accessed 31 March 2013).

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.07 MB