Abstract
In the research project, Medialog, a mixed participatory method, was applied to improve mutual understanding between consumers and chain actors in the French beef supply chain. This article presents a mixed methods evaluation of this participatory method using a SWOT analysis. The evaluation reveals several strengths of the participatory method such as dialogue and flexibility, as well as opportunities for wider application, for example, growing interest in multidisciplinary research and public involvement. However, identified weaknesses, such as quality of participants and time-consuming nature, and threats, such as lack of interest among policymakers and competing methods, should be monitored. The participatory method is found to be suitable to study various subjects and objects, particularly related to food consumption and supply chains.
Public participation is defined as “the practice of informing, consulting and involving members of the public in the agenda-setting, decision-making and policy-forming activities of organizations or institutions responsible for policy development” (Rowe & Frewer, 2004, p. 512). Over the past 30 years, the public has increasingly been involved in the development of policy and the preparation of policy decisions (Rowe & Frewer, 2005). Public involvement can be organized according to many different procedures. Standardized participatory methods, such as focus group discussions, citizen juries, site visits, public hearings, and so on, have formalized, well-developed procedures and have been widely implemented (Beierle & Konisky, 1999; Rowe & Frewer, 2000). Conventional participatory approaches have been criticized for their “soft” character because of their focus on qualitative data collection; for being biased, impressionistic, and unreliable (Cornwall & Jewkes, 1995); for being used for other purposes than public participation (Chess & Purcell, 1999); for their weak participant-centered approach (Deng & Wu, 2010); and for inadequately addressing public learning processes (Daniels & Walker, 1996). Innovative participatory approaches have emerged aiming to counteract the perceived failures of conventional approaches. The innovation can refer to procedural innovations as well as creativity in combining conventional methods (Beierle & Konisky, 1999). Variants on the standardized procedures might be more efficient and appropriate for the specific objective of the public participation exercise. Using different methods complementary to each other might be better suited to the specific objectives of public participation (Pidgeon et al., 2005). Mixed methods combine both qualitative and quantitative data collection and analyses (Collins & O’Cathain, 2009). The advantages of mixed methods research are well recognized (Vitale, Armenakis, & Feild, 2008). The use of a combination of methods can compensate weaknesses that stem from a singular approach through triangulation of outcomes and other data or accommodate different groups of participants depending on particular requirements at a specific point in time (Pidgeon et al., 2005). Nowadays, both standardized and nonstandardized participatory approaches are applied in a large number of areas, such as public health, industrial relations, science and technology, sustainable development, land-use planning, and service delivery (Bruges & Smith, 2008; Gauvin & Abelson, 2006; Petts & Leach, 2000).
Public participation can have different objectives (Bruges & Smith, 2008; Rowe & Frewer, 2000; Shmueli & Plaut, 2004). One possible objective that can be reached with participatory methods is the enhancement of mutual understanding between different parties involved in a particular subject. This is especially useful in settings with a low reciprocal confidence, where public involvement might help restore confidence and improve awareness about each others’ perceptions and/or concerns. A clear example of such a situation has arisen in the European beef sector. Following several consecutive food safety crises such as the mad cow disease crises and the dioxin scare, public confidence and trust in the beef sector has declined steadily in several European countries. Beef consumption has suffered severely from these incidents, laying a heavy burden on the sector (Verbeke, 2001). Although European beef consumers nowadays express some degree of renewed trust in the beef supply chain, they remain vigilant toward possible beef safety incidents and the application of new processing technologies in beef (de Barcellos et al., 2010; Van Wezemael, Verbeke, Kugler, de Barcellos, & Grunert, 2010). In this respect, the use of a public participatory approach might be an opportunity to improve the mutual understanding between consumers and chain actors, bringing their perceptions and ideas closer together (Bergeaud-Blackler & Paola Ferretti, 2006; Verbeke, Frewer, Scholderer, & De Brabander, 2007).
To ensure the successful further use and dissemination of public participatory methods, currently applied methods should be subject to evaluation. Evaluation can provide valuable insights into the drawbacks, pitfalls, and points of attention that should be taken into account when applying participatory methods. For this reason, evaluation studies of participatory methods are highly valuable when existing methods are planned for use in another setting.
Statement of Purpose
This article reports on a mixed methods evaluation study of a mixed participatory method that has been applied in the project called Medialog. First, the Medialog project and its participatory method are briefly presented. Second, objectives and research questions of the evaluation study are provided. Third, materials and methods for the evaluation study are explained. Procedures of conceptualization, data collection, and data analysis of the evaluation study are explicated, and the role of the researchers in the evaluation study is discussed. Fourth, in accordance with the sequential design of the evaluation method, procedures and results of the qualitative research phase are given, followed by procedures and results of the quantitative research phase. Finally, results are presented, followed by conclusions and recommendations for future research.
Medialog Project
The French Agence Nationale de la Recherche (ANR)–funded Medialog project (2007-2009) aimed to establish the preconditions for an improved mutual understanding between different stakeholders in the beef supply chain. As such, this project resorted under the research stream that is defined as action research, aiming to contribute to increased well-being and a more equitable relationship between different actors (Christ, 2010; Reason & Bradbury, 2004). The method that was used in Medialog is mixed methods based on public participatory methods and aimed to organize and foster a dialogue between protagonist groups involved in the French beef supply chain. Hereafter, the participatory method that was used in Medialog (illustrated in Figure 1) is referred to as “the participatory method.”
The participatory method.
Participants consisted of a panel of beef citizen–consumers (CC panel), on the one hand, and a forum of stakeholders involved in the beef supply chain, such as beef chain members, economic actors and institutions, consumer organizations, representatives of public institutions, and researchers who were also involved in the Medialog project, on the other (see Figure 1). To investigate the possibility of creating a dialogue between protagonist groups that could enable comparing and contrasting attitudes, three specific cases were examined: the use of antibiotics in cattle breeding, ground beef contaminated by Escherichia coli O157:H7, and the nutritional impact of meat consumption on humans. These three examples were used as case studies to investigate whether the negative image of meat in the media could be counteracted by an open dialogue between citizen/consumers and other stakeholders in the beef chain.
Repeatedly over a 1-year period, the CC panel formulated their beliefs, convictions, and questions with respect to scientific issues. The reflections of the CC panel were presented and compared with the positions of the stakeholders within the framework of a forum, allowing the dialogue between the CC panel and the stakeholders. Both groups supplied instruments and information to the other group for consideration and feedback. In addition, semiolinguistic analyses of the participants’ discourses and of media coverage and advertising were performed. Findings were consolidated by a quantitative consumer survey investigating dialogic mechanisms. Figure 1 illustrates the research protocol combining different conventional research methods such as focus group discussions for the constitution and semidirective animation of the CC panel, sensory tests and analyses for providing tools to the CC panel reflection, survey-type questionnaires for organizing the confrontations between actors and tracking the opinions of the CC panel, semiological and linguistic analysis for identifying the sociocultural values of the universe, and the differences of opinions between stakeholders.
The results of the Medialog project showed that the occurrence of meat contamination issues in the media did not affect meat quality perception by the CC panel during their confrontation with the meat safety systems and its actors in reality. This indicated that the participatory method can be successful in facilitating an open communication and dialogue. The possibilities of an open and reflexive informative communication were illustrated. Findings from this participatory study and recommendations with respect to transparency and communication have been disseminated to stakeholders and the project funding agency (ANR).
Need for Evaluation
The scope of this article is focused on reporting an evaluation study of the participatory method. The Medialog project was chosen for evaluation of its participatory method for a number of reasons.
First, the participatory method was successful in creating the preconditions for an open dialogue between citizen/consumers and other stakeholders in the beef chain. This successful application makes it worthwhile to investigate possible further applications of the participatory method. Second, this participatory method had been applied also in a previous study on animal welfare (Leusie & Alessandrin, 2005) and has now in the Medialog project been applied in a different research context. This process intended to provide insights into key attention points when applying the method in another context or setting. This knowledge is valuable for researchers who would like to apply the same method in their own research field. Third, owing to its mix of qualitative and quantitative research methods, the participatory method is a suitable example of how different qualitative and quantitative research methods can be successfully combined. For this reason, making the participatory method (and its evaluation as presented in the present article) known to a wider research community can improve mixed methods research practices.
Owing to the mixed methods approach, including a substantial degree of public participation, and because of its successful applications in research on animal welfare and the beef chain, the participatory method was believed to be a promising and usable method for further application. The application of this participatory method in the Medialog project has been a valuable attempt to further use and disseminate this approach. However, simply copying a method to another research context might not always be straightforward. A careful evaluation of the participatory method can provide insights into the drawbacks, pitfalls, and points of attention that should be taken care of when applying this participatory method. This evaluation study will provide insights about the participatory method, which will facilitate its wider application and further dissemination.
The evaluation of public participatory methods is recognized to be a difficult task (Rowe, Horlick-Jones, Walls, & Pidgeon, 2005). Every method has its pros and cons, which have been comprehensively documented in scientific literature with respect to the standardized methods (Abelson et al., 2001; Bruges & Smith, 2008; Gauvin & Abelson, 2006; Petts & Leach, 2000; Rowe & Frewer, 2000), but which are often ambiguous concerning nonstandardized mixed participatory methods. Possible disadvantages of using nonstandardized methods relate to the fact that they are often not well described in literature, making it difficult to repeat, evaluate, and optimize the procedures (Rowe & Frewer, 2000). A proper evaluation of the advantages and disadvantages of a specific participatory method is heavily dependent on the topic, the context, and the objectives of the specific research design. In spite of the great and well-acknowledged importance of evaluation and the fact that participatory and mixed research methods are being used for decennia now, empirical examples of evaluations of innovative and mixed participatory methods have not received equal attention in academic literature (Leech, Dellinger, Brannagan, & Tanaka, 2010; Rowe & Frewer, 2004).
Objectives and Research Questions of the Evaluation Study
The objective of this evaluation study is to evaluate the participatory method and investigate its usability and possibilities of further dissemination. Furthermore, this evaluation study will provide herewith one example of how research methods can be externally evaluated. The first research question concentrates on the identification of the strengths and weaknesses of the participatory method, as well as on the opportunities and threats facing the participatory method. Because of its exploratory nature, this research question will be answered using a qualitative approach. The second research question investigates the possibilities for wider application and further dissemination of this participatory method, necessitating a quantitative approach that integrates the insights obtained from the qualitative evaluation phase. This second research question not only investigates possible contexts in which the participatory method could be used but also points of attention that need to be taken into account when applying this participatory method in different settings or contexts. In this article, a mixed methods approach is applied to evaluate the participatory method, and the experience with the Medialog project is used as a case study. Hence, this article by no means aims at evaluating the success of the Medialog project in achieving its objective as such but at providing an ex post external evaluation of the participatory method.
The Approach of the Evaluation Study
The evaluation of the participatory method is based on a mixed research method including SWOT. SWOT is the acronym for “Strengths, Weaknesses, Opportunities and Threats.” SWOT is a strategic planning tool used to evaluate in a systematic way the external opportunities and threats and the internal strengths and weaknesses of a project or business venture (Sabbe, Verbeke, & Van Damme, 2009). Though originally designed for planning and strategy development, SWOT can be used in ex post evaluations to verify the relevance and coherence of the implemented method, especially if this task was not undertaken during the development of the method itself (EuropeAid, 2005). A comprehensive SWOT analysis, combined with data collection through a strategic orientation exercise, is considered to be a valuable and reliable method to perform an overall exploration of the strengths, weaknesses, opportunities, and threats of the participatory methods (EuropeAid, 2005). During a SWOT analysis, strengths, weaknesses, opportunities, and threats of the participatory method are explored and qualitatively listed. After analyzing these, a second round of data collection is organized, in which the degree is quantified to which opportunities can be taken advantage of and threats can be dealt with. This analysis allows determining possibilities and strategies for further application and dissemination of the participatory method (EuropeAid, 2005; Petts & Leach, 2000). Main points of interest for future strategy development are identified (Sabbe et al., 2009). What SWOT actually does is not only merely evaluates the quality of the participatory method itself but also provides insights into the further possibilities of applying the participatory method. Choosing SWOT as an evaluation method enables the formulation of clear guidelines on how and what to take care of when applying this participatory method in an alternative context. As such, these guidelines and identified attention points when applying this participatory method in another research context can be taken care of in advance, increasing the chances of a successful new application of the method. Using this evaluation method is particularly interesting in the field of nonstandardized mixed methods, as only the wider use of these methods can provide the basis of methodological standardization.
The main advantages of using SWOT analysis relate to its user-friendly application, the simple format, and the adaptability to a variety of situations, people, and events (Nicholls, Thoburn, Crecy, & Smart, 2008). These positive characteristics have led to the use of SWOT for the evaluation of participatory methods also in previous studies, such as the study by Petts and Leach (2000). Hereafter, the SWOT method used in this evaluation study is referred to as “the evaluation method.”
The decision to use SWOT as the basis for the evaluation method for the participatory method is based on our research questions. Although strengths and weaknesses of the participatory method (Research Question 1) could be investigated by the various criteria outlined in literature (see, for instance, Chilvers, 2007), possibilities for wider application and further dissemination of the participatory method (Research Question 2) cannot. A SWOT analysis designed for strategic decision making can thus provide answers to both research questions.
The evaluation method follows a mixed sequential design. Figure 2 illustrates the two sequential phases of conceptualization, data collection and data analysis, consisting of identifying the internal and external factors that might affect the method (Qualitative Phase 1) and specifying the strategy that could facilitate further usability and dissemination of the method (Quantitative Phase 2). During both sequential phases of the evaluation method, primary data are collected and analyzed. Data collected in the two phases are not only complementary, as the results from the qualitative approach are enhanced by the results of the quantitative approach, but also nested, as the qualitative results provide input for the quantitative data collection (Cameron, 2009). The conclusions drawn from the results of the first qualitative strand lead to formulation of questions, data collection, and data analysis for the quantitative strand (Teddlie & Tashakkori, 2006).
Exploratory sequential mixed design of the evaluation study.
SWOT analysis is typically done by so-called prime witnesses, that is, people who are well familiar with a project, situation, or environment, who are consulted at several points in time during a so-called strategic orientation exercise (Chapman, 2010). Using SWOT analysis as an evaluation tool for the participatory method, people who are actively involved in the participatory project and who are therefore most familiar with its participatory method are in the best position to think about the future possibilities of this participatory method.
Procedures of the Evaluation Study
The evaluation study of the participatory method was performed using an exploratory sequential design during the period April to October 2009 in which the researcher started by qualitatively exploring the topic before building to the second quantitative phase (Creswell & Plano Clark, 2011). The first phase consisted of a listing of SWOT components. The second phase consisted of scoring of a SWOT matrix and performing a quantitative analysis. In both phases, data were collected by electronic means among the same sample of individuals actively involved in the Medialog project and familiar with its method (n = 12; identical sampling). These individuals are further referred to as “the respondents.” About half of the respondents were linguists, mainly involved in the semiological analysis components of the Medialog project, whereas the other half were active in socioeconomic research, regarding topics such as the organization of the agro-food industry, market analyses of agricultural products, or consumer research. This diversity in the backgrounds of the respondents ensured variability in the obtained SWOT components. The limited sample size does not endanger the quality of the results of the SWOT analysis. The significance of the results of a SWOT analysis is determined by the high and decision-making position and involvement of the participants rather than by the number of participants. A SWOT analysis can be legitimately performed by a single analyst, a few managerial participants, or by a group of managerial or highly involved respondents (Langer, Alfirevic, & Pavicic, 2005). In this evaluation study, the limited size of the group of individuals who were actively involved in the participatory method led to the limited sample size, though without imperiling the required high position and involvement of the respondents in the project.
Role of the Researchers in the Evaluation Study
The role of the principal researcher consisted in collecting the data (qualitative as well as quantitative), analyzing and interpreting the data, and discussing the findings. In the qualitative phase of the evaluation study, the researcher recruited respondents and administered the open-ended questionnaires. The qualitative data analysis was performed and checked by a senior researcher to assure correct and reliable assignment of the data into the four categories (strengths, weaknesses, opportunities, threats). This allowed the researcher to distant herself from the qualitative data collection process and make relatively objective conclusions about it. Next, the researcher invited the same participants to take part in the second phase of the research. The quantitative results were discussed considering the complete lists of the qualitative data collection by the principal researcher to ensure consistency of the discussion of the qualitative and quantitative results. The researchers align with the pragmatic paradigm, placing the research problem, that is, the evaluation of the participatory method, as central and applying different approaches to understand the problem (MacKenzie & Knipe, 2006). The choice of the evaluation method is inspired by the researchers’ experience with both qualitative and quantitative research methods and the belief that mixed methods research can produce meaningful results thanks to the multiple perspectives involved.
Phase 1: Qualitative Research Phase
In the first phase, an open-ended questionnaire was sent to the respondents in which they were asked to list as many as possible internal strengths and weaknesses and external opportunities and threats of the participatory method. Eleven completed questionnaires yielded extensive enumerations including 44 strengths, 41 weaknesses, 43 opportunities, and 26 threats. These extensive lists were filtered from repeated and overlapping answers, and eventual misclassifications were relocated. The answers in the filtered lists were categorized based on their content and coherence, resulting in a final list of 5 strengths, 5 weaknesses, 3 opportunities, and 4 threats, which was extensively discussed with the respondents in brainstorming sessions (the so-called member check) before finalization. All elements listed below are based on the findings derived from responses of the 12 respondents.
Internal Strengths of the Participatory Method
Strengths are intrinsically linked to the participatory method and represent a possible source of competitive advantage for the participatory method. Based on the open-ended questionnaire, five strengths have been identified. First, the participatory method was considered as enabling an open dialogue between possibly opposite parties. Communication between these parties is not focused on convincing but rather on enhancing the mutual understanding of possibly conflicting points of view. The dialogue is a constructive confrontation of diverging opinions between consumers and chain actors and an enriching experience as both parties become aware of each others’ opinions and concerns. The enabling of a dialogue is a typical characteristic of participatory methods that go further than just informing (“public communication”) or hearing the public (“public consultation”; Rowe & Frewer, 2005), indicating that it is legitimate to call this a true participatory method.
Second, this flexible participatory method was perceived by the 12 respondents as easily adjustable and tailor-made to the objects and subjects studied. This inherent flexibility allows adaptation and anticipation to possible problems during the course and implementation of the research.
Third, the participatory method was believed to be very user-friendly and thus relatively easy to use. It consists of well-defined research stages with clear procedures. This is an important characteristic for the further dissemination and the usability of the participatory method in other settings. Because of its ease of use, the participatory method can be easily adopted by other researchers. Communication about these user-friendly procedures is a crucial step in the dissemination of this participatory method.
Fourth, another perceived strength of the participatory method was that it combines multiple approaches. This combination yields a rich amount of complementary data about the research topic, providing a more complete view on the topic, according to the 12 respondents. Since both qualitative and quantitative methods are included, complementary data with a very different nature are brought together. By the combination of methods into a mixed methods approach, some of the weaknesses of a singular approach can be compensated through triangulation of outcomes and other data (Pidgeon et al., 2005), making the results more comprehensive and reliable. For instance, weaknesses linked to qualitative research might be compensated by quantitative studies, reducing their severity without necessarily completely disappearing. The combination of different approaches and mixed methods provides added value to research projects and the results obtained from the application of such nonstandardized methods. This comprehensive complementarity suggests that the whole might be bigger than the sum of the parts. Being well-described and user-friendly, the further use and dissemination of the participatory method is facilitated.
Fifth, the participatory nature of the method allows consumers and actors to provide input and opinions about policy topics. This creates chances for the participants to contribute actively in the debate rather than just belonging to the passive mass. The opportunity for consumers and citizens to formulate their concerns and receive answers from the beef chain actors allows them to actively contribute to the debate. Therefore, the participatory nature of the method is considered a strength by the 12 respondents.
Internal Weaknesses of the Participatory Method
Weaknesses are intrinsically linked to the participatory method and represent a source of possible competitive disadvantage. Five weaknesses were deducted from the list of responses obtained through the 12 completed open-ended questionnaires. The first weakness that was mentioned by the respondents relates to the quality of the participants. The recruitment of suitable, sufficiently active, open, knowledgeable, and communicative participants is a difficult task. As a result, the formation of the CC panel and the forum of actors, that is, the search, selection, and motivation of the participants over the duration of the project, pass off laboriously. The weakness with respect to the quality of the participants is often linked with qualitative methods such as focus group studies (Abelson et al., 2001; Petts & Leach, 2000).
Second, the participatory method was perceived to be time consuming because of the difficult selection of the participants, the fairly slow data collection, and the time-intensive analysis. Therefore, this participatory method was seen as only suitable to provide results on medium to long term, and therefore only suitable to inform medium- to long-term policy decisions. Although site visits and surveys also need time to be performed, it is especially the analysis and reporting of focus group discussions that is time consuming (Abelson et al., 2001; Petts & Leach, 2000).
Third, the respondents highlighted that a major part of the findings from this participatory method are merely qualitative and thus not quantified. Because of the limited quantification, findings might be perceived by third parties as superficial, inadequate, or not suitable for decision making. This characteristic is inherently connected with the nature of qualitative methods and data. By adding a quantitative method, this weakness can partly be resolved, however without being completely wiped out.
Fourth, the participatory method creates artificial settings, as is often the case in research contexts, for instance, experimental consumer studies. Respondents suggested that panel members might not tell their true concerns and that panel leaders or moderators possibly influence the discussions. The assumption of both citizens/consumers and actors being open-minded and critical at the same time might be false. Also the actors can pretend to be “better” (being more knowledgeable, more involved, more open-minded) than they are in reality, according to the 12 respondents. Furthermore, after a while the dialogue yields “educated” consumers. All those artificial circumstances could endanger the objectivity of the results.
Fifth, the participatory method is not consensus driven. Respondents indicated to regret that the efforts to enhance mutual communication do not consistently result in a general agreement on the discussed topics. However, the fact that the participatory method is not consensus driven might be a bigger problem in policymaking than in this setting, since consensus might not be needed to enhance mutual understanding but is more required for policymaking purposes.
These strengths and weaknesses are internal factors, that is, they are linked to the participatory method itself. Improvements to the participatory method should try to counteract the weaknesses, while retaining the positive aspects of the participatory method. According to the 12 respondents, the participatory method is clearly strong in some aspects, but also inherently weakened by some of its own characteristics. By communicating clearly that this method is time consuming, and that it does not necessarily strive for a consensus, policymakers and funding agencies will not have unrealistic expectations about the outcomes of the participatory exercise, which can help decrease the perceived weaknesses. Furthermore, communication is also important to handle some of the external threats (see following sections), most notably to familiarize researchers and managers with the participatory method. If the participatory method and its procedures become better known, the use and efficiency of the participatory method might increase.
External Opportunities for the Participatory Method
Opportunities are new perspectives and chances for the participatory method that might provide chances for its application and usability. Three opportunities were deducted from the list of responses obtained through the 12 completed open-ended questionnaires. The first opportunity that was mentioned by the respondents is the growing interest in multi- and interdisciplinary research. Combining different scientific disciplines and methods is perceived as a good approach toward problems with a possible societal impact. Multidisciplinary research is gaining momentum and the use of mixed methods has been encouraged (Creswell & Garrett, 2008). This participatory method perfectly goes along with this trend in the current research environment, which provides opportunities for its further application.
Second, growing interest in and demand for public involvement activities are seen as an opportunity for this participatory method. There is a growing interest in involving society, the public, end users, stakeholders, and consumers to build awareness about issues with societal relevance. This participatory method is designed to use this public involvement to enhance the mutual understanding between actors. The participatory method is therefore suited to take advantage of this opportunity according to the 12 respondents.
Third, this participatory method is applicable for emerging issues in the society, which might benefit from public involvement and participation, such as sustainable development, pesticide use, food safety, greenhouse gas emission, fair trade, and ethical issues in agriculture, livestock, and food production. These highly relevant questions of the day offer opportunities for the application of the participatory method in other domains, many of which are situated in or linked with the environment, agriculture, and the food chain.
External Threats for the Participatory Method
Threats are negative external aspects that might cause problems or losses when the participatory method is applied. From the 12 completed open-ended questionnaires, four threats have been identified. The first threat that was mentioned by the respondents is the potential lack of interest in the research results. There might be a low policy, managerial, or scientific interest for partly qualitative results obtained from social sciences research. If the outcomes are not taken into account, the application of the participatory method can become pointless.
Second, a strong competition between different participatory methods and research teams mastering these methods was perceived by the 12 respondents. Competing methods might be better known, more widely used, or more commonly accepted and therefore they might be preferred over participatory exercises.
Third, the agendas of policymakers, opinion makers/leaders, and mass media are often short-term agendas. Their limited time frame does not always correspond with the time frame of independent and externally funded research, which might be especially the case with this time-consuming participatory method. Therefore, the 12 respondents argue that this participatory method is merely suitable to provide answers on the medium or long term, limiting its applicability.
Fourth, research funding policies might be not too keen on financing research that uses qualitative or mixed methods, partly because of the possible perceived inconclusiveness of the results. The resulting lack of funding might be a threat for the use of this participatory method in other research settings.
These opportunities and threats identified based on the responses of the 12 respondents are factors characteristic to the environment of the participatory method. Changes in the participatory method will not change the environment, but it might change the way the participatory method responds to these external factors. Attempts to change the environment are beyond the scope of this research. What really matters is to look for the method that is the most capable to take advantage of the opportunities and counteract the threats.
Phase 2: Quantitative Research Phase
In the second (i.e., the quantitative) phase of the evaluation study, the qualitative findings were used to develop the quantitative instrument. The SWOT components resulting from the first phase open-ended questionnaire were used as elements in a matrix that was used as the data collection instrument in the quantitative research phase. The SWOT components were combined in a matrix where the rows were filled with the internal strengths and weaknesses and the columns with the external opportunities and threats. In this matrix, each of the internal components was confronted with each of the external components, allowing a quantitative approach based on the initial qualitative findings. Next, 12 “prime witnesses” involved researchers and stakeholders (the same respondents as in the qualitative phase) were asked individually to attribute scores to every single cell of the matrix. These scores represented their answers on four questions related to the quadrant encompassing the cell (see Table 1). Scores were attributed according to two guidelines: first, a maximum of 12 points could be attributed to each column; and second, each single cell score had to be within the range of 0 to 3, indicating points of no (0), low (1), medium (2), or high (3) interest.
Meaning of the Quadrants of the SWOT Scoring Matrix.
The aggregated scores per cell indicate the relevance of each cell relative to other cells of the SWOT matrix. Given the way of attributing points, only scores higher than 10 were considered as indicating possible points of interest. Aggregated scores per quadrant reveal the most relevant strategic choice concerning the participatory method. Based on the distribution of the aggregated mean scores over the quadrants, possible strategic choices are illustrated (see Table 2). Standard terminology for strategic choices from strategic management literature and SWOT analyses followed by strategic orientation exercises is used (see, for instance Januszewska, Viaene, Gheysen, & De Steur, 2009). This approach allows prioritizing strategies that best match the internal strengths and weaknesses of the participatory method. By focusing simultaneously on the internal and external environment, the most suitable strategic option for the participatory method will be identified, providing guidance for its further use and dissemination.
Strategic Choices in the Quadrants of the SWOT Scoring Matrix.
The cells with the highest scores in the aggregated scoring matrix (Table 3) are the combinations of internal and external factors that are evaluated as most important by the respondents. The most promising opportunity, that is, the opportunity with the highest aggregated column total (123), is growing interest in public involvement, suggesting that opportunities related to involvement of the public in debates about issues with societal relevance are very appealing for the further use of the participatory method. The most relevant threat, that is, the threat with the highest aggregated column total (130), is the strong competition between and with other participatory methods, which might be better known, more widely used, or more commonly accepted compared with this participatory method.
Aggregated SWOT Scoring Matrix (n = 12).
The analysis of the aggregated cell scores over the four quadrants in the scoring matrix allows identifying the key points of interest or key issues facing the participatory method based on the responses of the 12 respondents. The aggregated cell scores in the first quadrant of Table 3 indicate to what extent a specific strength allows or helps for benefitting from a specific opportunity. Following the scores obtained, dialogue as a strength is important to be able to benefit from the opportunities related to interest in public involvement, allowing taking advantage of the fact that consumers and citizens are involved. The dialogue also plays an important role in benefitting from opportunities related to a growing interest in multidisciplinary research and emerging issues. The strength related to the participatory nature of the method makes it possible to benefit from the opportunities related to interest in public involvement. The strength dealing with combination of the different methods allows benefitting from multidisciplinary research and emerging issues. These results indicate that the opportunities are mostly supported by the strengths of the participatory method. The inherent strengths of the participatory method, as recognized by the 12 respondents, allow benefitting from the opportunities in the environment of the participatory method.
The aggregated cell scores in the second quadrant indicate whether a particular strength enables to cope with a threat according to the 12 respondents. The strengths related to use of multiple approaches of a research topic allows counteracting the fact that other methods of participatory research may be better known and more frequently used. Furthermore, this strength can help cope with the threat of lack of interest from the policymakers and as such facilitate funding for future research using the participatory method. Strengths related to the user-friendliness and flexibility of the participatory method are an asset compared with the possible threat that other participatory methods might be better known and more used. Participation from consumers and stakeholders (strength) can contribute to coping with a possible lack of policy interest for the results. These results indicate that the 12 respondents are confident that the threats for the participatory method can be partly counteracted by the strengths of the participatory method.
The aggregated cell scores in the third quadrant indicate whether a weakness prevents the participatory method from facing the threat, which is typically referred to as a “crisis” situation. The time-consuming participatory method prevents to cope with the short-term agendas of policymakers, opinion makers, and the press. The lack of quantification of the results and the fact that the participatory method is not focused on reaching a consensus hinders the search for funding and can make policymakers little interested. The difficulties to find good participants make other methods more attractive. These results indicate that the 12 respondents acknowledge that the weaknesses of the participatory method make it vulnerable for the external threats. The most “dangerous” weaknesses are the time-consuming nature and the lack of quantification. These weaknesses might bring disadvantages or problems if there is no or a bad reaction to the threats in the environment of the participatory method.
The aggregated cell scores in the fourth quadrant indicate whether a particular weakness prevents grasping a particular opportunity according to the 12 respondents. The difficulty to find “good” participants prevents benefitting from the advantages of public involvement and multidisciplinary research. Furthermore, the participatory method is time consuming and not consensus-driven, resulting in a suboptimal benefitting from the advantages of public involvement. The lack of quantification with this participatory method hinders multidisciplinary research. These results indicate that the 12 respondents believe that multidisciplinary research and especially public involvement (although important reasons to apply this participatory method) are not optimally benefitted from because of the weaknesses of the participatory method.
Based on the analysis of the aggregated quadrant scores (in total and/or average per cell), strategic choices can be recommended (see Table 4). The highest aggregated mean score (17) is seen in the first quadrant, indicating that several strengths of the participatory method allow to take benefit from several opportunities in the environment. From a strategic point of view, this quadrant corresponds with an “attack”—or offensive strategy that is, exploiting or using strengths to take maximum advantage of possible opportunities in the environment. This offensive strategy is dominant over the other strategies. Therefore, it is important to build awareness about the participatory method and its possible applications. An aggregated mean score of 11 is found in the second and third quadrants, which correspond with a “defensive” and a “crisis” strategy, respectively (see Table 2). These scores signal that there is still room and need for improvements and optimization of the participatory method, for instance, by complementing the participatory method with further quantitative research methods and avoiding unrealistic promises with respect to timing for the provision of input for policy decision making, since the participatory method is facing several relevant threats. Finally, there is no apparent need for major changes in the participatory method in terms of “cleaning-up” its weaknesses, as shown by the low aggregated mean score in the so-called clean-up strategy quadrant, but careful recruitment of the participants is needed.
Sum and Mean (Sum/Number of Cells) Scores Per Quadrant.
Discussion
As a response to the perceived decline in consumer confidence in the beef chain, the importance of transparency as well as developing mechanisms to involve lay people early in decision processes have been stressed. Although short-term effects of participatory methods as a trust-building activity are limited, it is assumed that this approach can have a positive effect on consumer confidence in the long term (Verbeke et al., 2007). In the food sector in general, and the meat sector in particular, participatory methods have been used to assess public acceptance of new technologies and products (de Barcellos et al., 2010). In the French ANR-funded Medialog project, a mixed participatory method has been applied to enhance mutual understanding between citizen–consumers and actors in the beef chain, aiming at increased reciprocal confidence between these parties. This nonstandardized participatory method has been tailor-made to the project in order to be geared to the research setting and its environment. In this article, the usability of the participatory method is externally evaluated using the SWOT template and the possibilities for future applications are explored.
The dominant offensive strategic choice that comes forward from the evaluation study indicates that the participatory method is capable of exploiting or using its own strengths to take maximum advantage of possible external opportunities. This clearly suggests that the participatory method is suited to be applied in other fields of research and policy, in particular, in relation to emerging issues, for example, with respect to technology adoption, market acceptance, or issues of public debate such as sustainability. For increased usability, the weaknesses of the participatory method should be monitored and improved. Possible improvements can be effected by communicating clearly on timing and outcomes, creating clear procedures on participant management, or giving publicity to the participatory method.
The strengths, weaknesses, opportunities, and threats provided by the people familiar with this participatory method can be considered as criteria that can be used to evaluate the quality (in terms of usability and further possibilities) of the method. Major similarities are found between the elements found in this evaluation study and criteria assessing the quality of participatory methods that are reported in literature. Rowe and Frewer (2004) reviewed 30 public participation evaluation studies published between 1981 and 2004. Their review indicated that acceptance criteria (including representativeness, independence, early involvement, and transparency) and process criteria (including resource accessibility, task definition, structured decision making, and cost-effectiveness) were the most used criteria to assess the quality of public participation studies. The researchers consider these criteria to be key to successful public participation. More recently, looking at academic studies and policy communities in the United Kingdom, it has been concluded that there are seven effectiveness criteria on which considerable consensus exists (Chilvers, 2007). These criteria are presented in Table 5 (left side) and are clearly overlapping with the criteria of Rowe and Frewer (2004).
Similarity in Criteria Listed in This Evaluation Study (Left), Quality Criteria for Public Participation Research (Middle) and Quality Criteria for Mixed Method Research (Right).
Note. Only similar criteria shown; S = strength; W = weakness; O = opportunity; T = threat.
Overall, Table 5 shows major similarities between the criteria obtained from literature and the elements from this evaluation study. Researchers are often confronted with the difficulty to represent in their participatory studies all those interested and affected by a decision or action (Chilvers, 2007). Also, in this participatory method, the quality of the participants has come to the front as a particular challenge. The participatory nature of the method has been mentioned as a strength. Fair deliberation refers to allowing the participants to enter the discourse and put forward their views in interactive deliberation that develops mutual understanding between participants. This idea is connected in this evaluation study to the strength of creating a dialogue between participants. Access to resources was reflected in the fear of insufficient funding for this type of research. The user-friendly participatory method points toward transparent procedures. However, a possible lack of interest in the results was mentioned as a weakness that undermines accountability. Learning processes were included in the artificial setting, creating “educated consumers.” Time-efficiency was mentioned as an important point of attention in the results of our study. This similarity of the criteria obtained from this evaluation study with criteria from literature shows that the SWOT elements are realistic and supports the reliability of our findings.
Notwithstanding the major similarities between the criteria from literature and the evaluation study, two important differences emerge. First, one criterion did not emerge in the evaluation study. No parallel criteria was found for the assessment whether the participatory method was conducted in an independent and unbiased way. This disparity was possibly caused by the type of respondents that participated in the evaluation study, who were researchers or stakeholders involved in or aware of the Medialog project. As such, they were possibly positively biased toward the objectives of the project. Second, several criteria from the evaluation study are not covered by the criteria obtained from literature. Table 5 shows that only a limited number of the opportunities and threats identified in the evaluation study can be linked to similar criteria from literature, indicating that this evaluation study takes into account more external aspects than the criteria from literature. This distinctive characteristic is especially positive in the context of this evaluation study, where the evaluation of the participatory method was performed to investigate possibilities for further use and dissemination, which incorporates by definition external aspects of the participatory method.
Apart from the criteria from the literature with respect to public participation, criteria regarding mixed methods studies are also useful to consider in this context. O’Cathain (2010) provided an extensive overview of criteria used to assess the quality of mixed methods studies. Table 5 lists mixed methods criteria that can be linked to the results of this evaluation study. High similarity is found with respect to criteria highlighting the planning and undertaking of the study. The user-friendliness and the application of multiple approaches refer to high levels of planning quality and design quality. Sampling technique and sample size, the use of the findings by consumers and policymakers, and the feasibility of the research are important criteria to assess mixed methods research (O’Cathain, 2010), as was found in this evaluation study. The fact that multiple approaches have been considered as a strength in this evaluation study refers to a quality criterion that is very typical for mixed methods, namely, yield (i.e., the whole is more than the sum of the parts). Several criteria outlined by O’Cathain (2010) are related to documentation issues (such as the presence of a literature review), which are relevant in scientific reports of mixed methods studies. However, they appear to be less relevant in this evaluation study, where the research focus is on the usability and possibilities for further dissemination of a given participatory method.
The limitations of this evaluation study need to be acknowledged. The present evaluation exercise was performed with a limited number of participants because of the necessary methodological knowledge and awareness of the participatory method at hand. This general assessment of the participatory method does not go into detail on all aspects of the participatory method and does not take into account the overall effectiveness of the participatory method with respect to the results and consecutive policy decisions. Last but not least, this participatory method provides just one case on which the SWOT-based evaluation has been performed. Similar evaluations of research methods are recommended.
The results reported in this article illustrate that SWOT analysis is a valuable tool and evaluation method for investigating possibilities to use participatory methods in alternative contexts. Whereas the evaluation criteria found in literature focus mainly on internal characteristics of methods, the presented evaluation method pays equal attention to external opportunities and threats occurring in the broader environment where the method is used. These external issues are crucial to decide on what strategy to follow with a given method. Especially in the case of nonstandardized mixed methods, performing an evaluation study using a SWOT analysis before applying such a method in another context provides valuable insights into further possibilities. Attention points when applying a method in another context can be taken care of in advance, increasing the chances of a successful new application of the method. Researchers interested in conducting an evaluation study of their own methods are recommended to perform a SWOT analysis to identify the best strategy for future applications.
The objective of this evaluation study was to evaluate the potential application of the participatory method in the given setting and to explore the possibilities for the application of this participatory method in other domains. Although we opt to strive for an optimization of the participatory method in the existing policy and research environment, it is out of the scope of the study to attempt to change the environment, for instance, increasing the appreciation for qualitative research. Further application of this participatory method should monitor the weaknesses and take as much as possible advantage of the opportunities in their specific environment. There are of course limitations to any participatory method as well as to any related evaluation study. As a result, all those who oppose public involvement or those who disagree with the outcomes of the evaluation are likely to find some ground for disputing the participatory method itself as well as the evaluation of it.
Conclusion
The evaluation of the participatory method has been performed in two sequential phases. First, extensive lists of SWOT components have been elicited from a sample of people who are highly familiar with the methods used and who are familiar with the research environment where similar methods can be used. The results of this qualitative approach were used as input for the next phase. Second, a scoring matrix confronting consensus SWOT components has been completed and quantitatively analyzed. The SWOT analysis informed by strategic orientation and brainstorming as performed in this evaluation study has proven useful as a means to explore the usability and possibilities for further application of this participatory method. The results of this evaluation study show that the participatory method can be recommended for use in other domains, especially where information asymmetry exists between individuals or groups. In particular, food consumption, agro-food chains, cases with uncertainties about technology adoption or market acceptance, as well as other topics or sectors where the degree of alienation between consumers and producers is substantial can be studied using this participatory method. The participatory method can be optimized by careful monitoring of its identified weaknesses (in particular the quality of the participants) and taking as much as possible advantage of the opportunities in the particular environment in which the participatory method is applied.
Footnotes
Acknowledgements
All referees and the journal editor are gratefully acknowledged for their valuable comments and suggestions on earlier drafts of this article.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was carried out with the financial support of the ANR—Agence Nationale de la Recherche—The French National Research Agency under the Programme National de Recherche en Alimentation et Nutrition Humaine, Project ANR-06-PNRA-026, MEDIALOG, in which Ghent University participated as a subcontractor for performing the evaluation study.