Process Evaluation in Health Communication Media Campaigns: A Systematic Review

Abstract

Objective. The objective is to examine the scope of health communication media campaign process evaluation methods, findings, and dissemination practices. Data Source. A systematic review of peer-reviewed literature was conducted using database searches. Study Inclusion and Exclusion Criteria. Published studies on process and implementation evaluation of health campaigns with a media component were included. Exclusion criteria included not health, non-empirical, no media campaign, or a focus on other evaluation types. Data Extraction. Articles were assessed for general campaign information, theory use, and details about process evaluation plan and procedures. Data Synthesis. A coding scheme based on 9 process evaluation best practice elements (e.g., fidelity and context) was applied. Process evaluation methods, measures, and reporting themes were synthesized. Results. Among 691 unique records, 46 articles were included. Process evaluation was the main focus for 71.7% of articles, yet only 39.1% reported how process evaluation informed campaign implementation strategy. Articles reported 4.39 elements on average (SD = 1.99; range 1–9), with reach (87.0%) and recruitment (73.9%) described most frequently, yet reporting was inconsistent. Further, the level of detail in reporting methods, theory, and analysis varied. Conclusions. Process evaluation provides insight about mechanisms and intervening variables that could meaningfully impact interpretations of outcome evaluations; however, process evaluations are less often included in literature. Recommendations for evidence-based process evaluation components to guide evaluation are discussed.

Keywords

process evaluation health communication campaigns social marketing systematic review research on evaluation

Objective

Evaluation is essential for health communication and social marketing campaign interventions (hereafter referred to as health campaigns) to create behavior change and improve health.^1-5 Evaluation research within health communication, education, and promotion has primarily focused on outcomes as a way to assess programmatic success. Process evaluation as a type of campaign or program evaluation is often overlooked or conducted with less care than other evaluation types.^6-8 However, comprehensive evaluation is an iterative process that requires process evaluation to contextualize and explain outcomes.^6,8,9 Understanding a health campaign’s process helps to illuminate specific intervention components and mechanisms that contribute to successful outcomes.^3,9 Conversely, examining the process is key to recognizing which aspects of the campaign are ineffective and require improvement in future iterations. These aspects are likely to go undetected in outcome evaluations.⁵ Process evaluation, which reports on implementation monitoring information focused on logistics and procedures is useful for campaign planning.¹⁰ Simply put, “without process evaluation, one cannot differentiate between a bad campaign and a poorly implemented one.”⁹ Given the importance of conducting process evaluations, the purpose of this review is to determine current use of best practices in published literature related to process evaluation, as well as the advisory use of results from process evaluation on campaign dissemination and implementation.

Defining Process Evaluation

While many definitions of process evaluation exist, for the purposes of this review we developed a definition based on an assessment of common features described in evaluation research literature (see Table 1).^5-7,11-14 We conceptualized and operationalized process evaluation as that which describes the campaign, identifies barriers to successful implementation, and systematically assesses whether or not the campaign was implemented as planned.

Table 1.

Process Evaluation Definitions.

Citation	Definition
Balch and Sutton⁵	Process evaluation measures how the program is working. It is used, presumably, to learn what actually happens during the program and whether the program really delivered what it was supposed to. This may include “monitoring” the program to make sure it is delivered according to plan (p. 62)
Fraze et al.¹⁴	Process evaluation assessed the extent to which the campaign was implemented as planned by examining campaign activities and target audience exposure and receptivity (p. e26)
Glasgow and Linnan¹²	Measures the extent to which an intervention was delivered or implemented as intended (p. 490)
Nutbeam et al.⁷	Provide an assessment of how a program is implemented, what intervention activities are provided under what conditions, by whom, to what audience, and with what level of effort (p. 87)
Saunders et al.⁶	Process evaluation looks inside the black box to see what happened in the program and how that could affect program impacts or outcomes (p. 134); assessment of program implementation (p. 136)
Steckler and Linnan¹³	Helps explain why certain results were achieved and understand the relationships among selected intervention or program components (p. 2)
Valente and Kwan¹¹	Process research (also known as monitoring) consists of those activities conducted to measure the degree of program implementation to determine whether the program was delivered as it was intended (p. 85)

Process evaluation often uses a combination of qualitative and quantitative methods from various sources to continually monitor campaign fidelity.¹¹ Practitioners use process evaluation to assess campaign reach, audience exposure and receptivity, acceptability, quality, and attribute causality to the campaign by contextualizing effects and effectiveness.^6,7,13-15 By examining the conditions under which a campaign is implemented, process evaluation focuses on how and why a campaign worked (or did not work) as opposed to outcome evaluation, which focuses on campaign effects and effectiveness.^10,12,16-19 Information gained from process evaluation is fundamental to improve future implementation.^1,2,11

Process Evaluation Frameworks and Best Practices

Increasingly, there have been calls for establishing standards for evaluation reporting by public health and evaluation researchers.^5,20,21 Frameworks for conducting process evaluation attempt to narrow the scope, define goals and objectives, and standardize methods and strategies.^3,6,8,13,15 While several checklists that outline process evaluation exist,^6,8,17,22 and there is increasing acceptance of a core set of elements,⁸ evaluation researchers do not agree on a singular comprehensive process evaluation framework.

Existing frameworks are similar in that they attempt to describe and define steps to guide evaluation planning or essential process evaluation elements. These elements include campaign context, reach, dose delivered, dose received, fidelity, implementation, recruitment, barriers, and contamination (see Table 2 for conceptual definitions). Although valid indicators of campaign processes, it is unclear if these elements are reported by practitioners and researchers with regularity. Evidence suggests that campaigns grounded in theory can facilitate process evaluation planning,^6,9,23,24 but many campaigns are atheoretical.^24,25 Standardized evidence-based process evaluation practices would help guide process evaluation planning in health campaigns.

Table 2.

Process Evaluation Best Practice Elements and Definitions.

Element	Operational Definition^a	Source
Context	Aspects of social, political, and economic environment that influence implementation	Baranowski and Stables¹⁷; Linnan and Steckler²²
Reach (exposure)	Proportion of intended target audience that participates in a campaign (e.g., exposed to campaign messages or campaign event attendance)	Baranowski and Stables¹⁷
Reach (exposure)		Linnan and Steckler²²
Dose delivered	Number or amount of intended units of each campaign component delivered	Linnan and Steckler²²
Dose received	Extent to which the audience actively engages with, are receptive to, and/or use materials or recommended resources or follow-up actions	Linnan and Steckler²²
Fidelity	Extent to which campaign was implemented as planned and consistent with underlying theory and philosophy	Linnan and Steckler²²
Implementation	Composite score that indicates the extent to which the campaign has been implemented and received by intended campaign audience	Baranowski and Stables¹⁷
Implementation		Linnan and Steckler²²
Recruitment	Methods used to attract campaign audience	Baranowski and Stables¹⁷
Recruitment	Methods used to attract campaign audience	Linnan and Steckler²²
Barriers	Problems encountered in reaching campaign audience	Baranowski and Stables¹⁷
Contamination	Extent to which campaign audience receives interventions or content related to the campaign from sources other than the campaign	Baranowski and Stables¹⁷

^aOperational definitions have been adapted to be campaign-specific rather than focus on interventions in general.

The purpose of the current study was to conduct a comprehensive review of published evaluation literature to identify trends in process evaluation of health campaigns. Specifically, we sought to answer the following 3 research questions:

Are health campaign process evaluation methods and findings published in the evaluation research literature?

How are process evaluation best practices used to improve campaigns and reported in the evaluation research literature?

What reporting gaps exist in process-specific evaluation research?

In answering these questions, we aim to synthesize process evaluation literature and make suggestions for how to conduct and disseminate process evaluation research.

Methods

Data Sources

This systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for reporting systematic reviews.^26,27 The authors designed an iterative, comprehensive search strategy to identify literature published through December 2018 on process and implementation evaluation of health campaigns with a media component (e.g., print, out-of-home such as billboards, and/or digital channels to disseminate campaign messages and materials). Search terms with filters for peer-review and English language were queried on the same day in each of the following databases: PubMed, Embase, PsycINFO, Communication Source, ISI Web of Science, CINAHL, and Scopus. These databases represent some of the largest repositories for public health, health education, health communication, health promotion, and medicine research. Queries using Boolean operators were conducted for a combination of search terms and controlled vocabulary or index terms for each database where applicable (e.g., Medical Subject Headings or MeSH for PubMed and Emtree thesaurus for Embase) specific to process evaluation of health campaigns with a media component: “process evaluation,” “implementation evaluation,” “campaign,” “media,” and “marketing” (see Appendix A for example search strategies for Embase and PubMed which are representative of queries conducted). The search terms and related controlled vocabulary specified above were used to limit results to conduct a review of process evaluation specifically, rather than all evaluation research. Searches were exported and imported into the citation manager EndNote X9 (Clarivate Analytics). Queries within each database were repeated 1 day later for accuracy. Duplicate citations were identified and removed using both the EndNote compare function and a manual review. Additionally, searches for gray literature in OpenGrey (a commonly recommended database for gray literature; www.opengrey.eu) produced no results.

Inclusion and Exclusion Criteria

The following exclusion criteria were used for this review: (1) articles not directly related to health or wellness (e.g., education or politics), or did not describe or explain an evaluation study (i.e., did not assess strengths and weaknesses of campaign effectiveness), (2) articles that described evaluation innovations or practices and/or theoretical frameworks or procedural guides on how to conduct evaluation but did not generate original data, (3) articles of monitoring and evaluation of surveillance systems or articles focused on monitoring patient health (e.g., care processes and patient outcomes) or the hospital system, (4) campaigns administered in a hospital or clinical setting that were delivered at the interpersonal level (i.e., patient-provider; however, evaluations of campaigns disseminated in clinical and health care settings were included), (5) one-on-one or group-level campaigns that sought to change individual-level behaviors but did not incorporate a media campaign component, (6) articles focused on formative, outcome or summative evaluation, and (7) evaluation studies reported in conference abstracts, conference papers or proceedings, dissertations or theses, book chapters and/or books. The decision to exclude campaigns without a media component was based on similar criterion put forth by the Community Preventive Services Task Force (CPSTF) systematic review on health communication campaigns, which focused on campaigns with a mass media channel as a way to narrow the scope of the systematic review.²⁸ Often, process evaluation is a small component of an overall reported study that includes a few brief sentences on process metrics and findings. Published literature rarely reports procedures used to conduct process evaluation. Therefore, the decision was made to also exclude comprehensive evaluation studies because of the potential challenges in synthesizing process evaluation within the context of these larger comprehensive evaluation reports.

Data Extraction

A multi-phased screening process was used to screen and review potential articles for inclusion. In the pilot phase, a calibration process was performed where 2 coders familiarized themselves with the data by independently reviewing citation titles and abstracts to screen citations for relevance using a mutually agreed-upon decision tree based on the pre-determined inclusion and exclusion criteria.²⁹ Citations were sorted into 9 categories based on inclusion and exclusion criteria, and the 2 coders met after conducting a blind screening of citations to review sorted lists for consistency and identify discrepancies, which were resolved following a discussion. The remaining citations were divided and independently screened by the 2 coders, calibrating twice more during the title and abstract screening process, resolving coding disagreements using a consensus-based approach. Full text was located for the potentially relevant articles. The coders then reviewed full texts using the same decision tree and processes, excluding additional articles.

A data extraction matrix was used to extract process evaluation design and strategies from the articles (the complete data extraction matrix is available from the first author). Extracted information included: general campaign information (e.g., country of study, campaign name, health issue, dates, audience, lead agency, and components), use of theory or framework, details about process evaluation plan and procedures (e.g., objectives or questions, methods, measures, sample, analysis, and interpretation), and general implications for process evaluation. Articles were divided equally between 3 coders who used a calibration procedure—whereby meeting for each coder to lead a discussion of 1 article and allowing the other 2 coder’s to raise questions or concerns about the extracted data—to ensure consistency and resolve any discrepancies. Full texts were then examined, and relevant information was extracted by the 3 coders independently. Each coder was also assigned 2 articles to spot check the other coders full text extraction. In addition to the citations identified through the database searches, a snowball approach was used to identify related citations from the reference lists of articles included in the full text review that did not appear in the searches, but met inclusion criteria. Twenty-seven related citations were screened for full text review and included in full text extraction. Among these, 4 duplicates were identified, and full text could not be obtained for 7 articles.

Data Synthesis

A coding scheme was developed based on existing checklists of best practices described in the empirical research by evaluation researchers.^17,22 The data extraction matrix was used to code each article for reporting of 9 best practice elements (context, reach, dose delivered, dose received, fidelity, implementation, recruitment, barriers, and contamination) using either “Yes,” “No,” “Unclear,” or “N/A.” A composite score for reporting of best practice elements was calculated for each article (range 0–9). A narrative synthesis of additional themes related to process evaluation methods, measures, and reporting are summarized in the following section.

Results

Search Results

A total of 1313 citations were identified from database queries, and an additional 27 citations were identified from related citations (i.e., those identified from the reference lists of articles included in the full text review that did not appear in the searches, but met inclusion criteria). Duplicate citations (n = 622) were removed, resulting in 691 (52.6%) potentially relevant citations. All 691 citations were screened and 86% (n = 595) were excluded based on eligibility criteria. Ninety-six (13.9%) articles were then assessed for eligibility and 50 articles were excluded (for reasons for exclusion, see Figure 1).²⁶ After an examination of each article’s full text, 46 articles published between 1987 and 2018 met eligibility criteria and were included in this review. Among these articles, the majority (n = 27; 58.9%) were published between 2008 and 2018,^14,30-55 and discussed campaigns that focused on a variety of health issues, including diet and nutrition (n = 8; 17.4%),^{17,47,52,56-60} cancer prevention (n = 6; 13.0%),^{30,38,42,46,61,62} and sexual health (n = 6; 13.0%).^{14,43-45,55,63} Notably, process evaluation procedures and outcomes were the primary focus for the majority of articles (n = 33; 71.7%; Table 3) included in this review.^{4,17,30,31,33-35,37,39,42-45,47-56,58,59,62,64-70}

Figure 1.

Process evaluation systematic review flow diagram of study selection and exclusion.

Table 3.

Descriptive Statistics about Studies for Included Articles (N = 46).

	n	%
Campaign health issue
Cancer prevention	6	13.0
Cardiovascular disease prevention	5	10.9
Diet and nutrition	8	17.4
Drugs and alcohol	3	6.5
Mental health	3	6.5
Physical activity	5	10.9
Sexual health	6	13.0
Smoking	3	6.5
Other health issue^a	7	15.2
Country of study
Australia	8	17.4
Canada	5	10.9
Fiji	1	2.2
Germany	1	2.2
India	1	2.2
New Zealand	1	2.2
Northern Ireland	1	2.2
Samoa	1	2.2
Sweden	1	2.2
The Netherlands	5	10.9
United Kingdom	1	2.2
United States	19	41.3
United States and Mexico	1	2.2
Process evaluation main focus of article (Yes)	33	71.7
Campaign lead agency reported (Yes)	29	63.0
Theory or framework reported (Yes)	26	56.5
Formative research activities reported
Yes	22	47.8
Yes, reported elsewhere	10	21.7
No	14	30.4
Process evaluation objectives reported (Yes)	40	87.0
Process evaluation findings informed campaign implementation (Yes)	18	39.1

^aOther health issues include antibiotics overuse, asthma, diabetes prevention, injury prevention, oral hygiene, and violence prevention.

Reporting of Best Practice Elements

Reporting of the 9 process evaluation best practice elements varied, as articles reported 4.39 elements on average (SD = 1.99; range 1–9). Only 11 articles (23.9%) reported at least 6 of the 9 coded elements,^{4,17,32,35,36,48,51,53,54,56,61} and unsurprisingly, the only article that included all 9 elements¹⁷ was the one that informed the coding scheme. Process evaluation elements, reach (n = 40; 87.0%), recruitment (n = 34; 73.9%), dose delivered (n = 28; 60.9%), and fidelity (n = 24; 52.2%) were most frequently reported, while barriers (n = 13; 28.3%), implementation (n = 9; 19.6%), and contamination (n = 8; 17.4%) were less frequently reported (Table 4). Aside from the element context, best practice elements were deemed not applicable for one article which focused on local media monitoring.⁶⁴ Language used to describe these elements differed across articles as terms (e.g., “fidelity” and “dose received”) were often not explicitly stated and had to be inferred using contextual clues from descriptions of measures and results. Some articles used “reach” and “exposure” interchangeably while others provided data for reach separate from exposure to campaign messages and materials. Notably, 4 (8.7%) articles cited various process evaluation components and definitions put forth by Linnan and Steckler (which was used to develop our conceptual definition and best practices list).^43,48,51,56

Table 4.

Reporting of Process Evaluation Best Practice Elements of Included Articles (N = 46)^a.

	n	%
Context^{4,17,30,35-37,42,43,47,48,50,51,53,55,56,61,63,64,66-68,70,71}	23	50.0
Reach (exposure)^{4,14,17,30-33,35,36,38-46,48-51,53-63,65,67-71,78}	40	87.0
Dose delivered^{4,14,17,32,33,35,36,40,41,47-49,51,53,54,56-63,65,66,68,69,78}	28	60.9
Dose received^{17,32-36,40-43,45,46,48,49,51,54-56,58,60-62,65}	23	50.0
Fidelity^{4,14,17,32,35-37,47,48,50-54,56-58,61,66,67,69-71,78}	24	52.2
Implementation^{17,32,35,36,48,51,54,56,61}	9	19.6
Recruitment^{4,17,30-33,35,36,39-48,50,51,53-55,57,59-61,63,65,67,69-71,78}	34	73.9
Barriers^{17,32,40,45,48-54,56,60}	13	28.3
Contamination^{4,17,49,56,58,66,67,71}	8	17.4

^aWith the exception of context, elements did not apply for one article (Andersson et al.⁶⁴) and was therefore coded as N/A.

Percentages do not add up to 100% since element categories are not mutually exclusive.

Process Evaluation Procedures and Objectives

A majority of articles in this review (n = 44; 95.7%) described the procedures and methods used to conduct process evaluation activities. Among these, 26 (59.1%)^{4,30,31,33-43,46,49,54-58,61,62,65,68,69} employed quantitative methods (e.g., surveys and questionnaires, web and social media metrics, media monitoring, and campaign implementation metrics), while 17 (38.6%) also incorporated qualitative methods (e.g., interviews, focus groups, and content analyses) qualifying them as mixed methods evaluation activities.^{14,32,44,45,47,48,50-53,59,60,64,66,67,70,71} While some articles explicitly defined process evaluation methodology, many articles did not clearly describe sources of implementation data (e.g., materials distribution, campaign post-event, and media placement reports) or how these data were analyzed. Often, counts of implementation activities were reported in the article implying quantitative methods were used. Additionally, the majority of articles (n = 40; 87.0%) included process evaluation objectives; however, objectives were reported with varying levels of detail. Some objectives were clearly stated and specific to the campaign while others were broad using generic phrases such as “to determine whether initiatives were carried out as planned” (Table 3).

Use of Theory and Frameworks

Reporting use of theory and frameworks and formative research activities varied across articles. A little more than half of articles (n = 26; 56.5%) reported the theoretical approach used to inform campaign development and evaluation (Table 3).^{4,14,30-33,35,36,39-42,45,47,49,51-54,56-60,65,70} Among these, behavior change theories (n = 8; 30.8%)^{4,14,30,31,42,49,59,65} including the health belief model,⁷² theory of planned behavior,⁷³ social cognitive theory,⁷⁴ and transtheoretical model⁷⁵ were most frequently mentioned, closely followed by a combination of behavior change theories and general social marketing frameworks (n = 7; 26.9%).^{32,35,36,39,41,56,60} Five articles (19.2%) described evaluation frameworks, such as the RE-AIM framework for evaluation⁷⁶ and PRECEDE-PROCEED model,⁷⁷ that were used to inform campaign evaluation.^{40,51,52,54,57} Two articles (7.7%) were not clear about which theoretical frameworks or models informed the campaign, though the articles did mention using theory.^45,70 Nearly half of the articles (n = 22; 47.8%)^{14,30,32,33,35,38,40,41,43,47,51,53-57,60,61,63,70,71,78} described the formative research activities used to develop the campaign and another ten (21.7%) stated that formative research was conducted but was reported elsewhere (Table 3).^{31,34,36,42,44-46,48,50,65}

Analysis and Interpretation

Among the articles included in this review, only 18 (39.1%) articles described how process evaluation results informed future campaign implementation strategies and activities (Table 3).^{4,30-33,41,45,47,48,52,53,55-57,59,66,71,78} Similar to process evaluation objectives, how process evaluation informed the campaign was reported with varying levels of detail. Some articles (n = 10; 21.7%) outlined the specific changes made (or would be made in the future) to campaign implementation.^{4,31,33,45,47,48,56,59,71,78} For example, Belanger-Gravel et al.³¹ described using other channels (e.g., a gaming app) to increase reach to priority audiences, while Pedrana et al.⁴⁵ discussed revising and refreshing campaign messages and materials to be more effective. Reininger et al.⁴⁷ adjusted media placements to improve dose delivered and Koelen et al.⁷⁸ attempted to make changes to address issues of limited materials distribution.

Two articles (4.3%) described changes to data collection, explaining how process evaluation findings led to adjustments to research instruments and that responses to survey questions informed subsequent rounds of data collection.^52,55 Van Der Feen De Lille et al.⁵⁹ discussed how process evaluation results contributed to establishing campaign goals for the subsequent year of the campaign. Two other articles (4.3%) indicated that implementation activities would be maintained as a result of findings.^30,32 Both Baumann et al.³⁰ and Berkowitz et al.³² noted that since process evaluation showed promising results, the campaign teams made the decision to maintain certain implementation strategies, such as using process data about events to repeat activities with a better understanding of how to plan and prepare to meet audience needs. Other articles were vague and simply stated that process evaluation results would be used to inform future implementation, speculating about the utility of findings for other phases of the current campaign or future iterations.^41,53 Articles also discussed how process evaluation variables were statistically linked to outcome evaluation. For example, one article⁴⁴ clearly demonstrated the relationship between process variables (e.g., exposure to mass media) and outcome evaluation variables (e.g., knowledge, intentions, beliefs, social norms, and self-efficacy).

Conclusions

Public health practitioners have touted the advantages of using process evaluation for health campaigns for decades. Process evaluation data contextualizes summative and outcome evaluation findings and informs future implementation. Yet this review shows that the reporting of process evaluations varies widely and appears less frequently in scholarly literature. While there has been extensive research on best practices for outcome evaluation,^79-81 the opposite holds true for process evaluation. As demonstrated by this review and others before it,^28,82 best practices in process evaluation are infrequently and inconsistently reported, resulting in a process evaluation “black box.” Process evaluation elucidates whether the campaign was implemented according to plan, or if something occurred outside the campaign team’s control that impacted outcome evaluation methods and findings. Results from this review and others suggest that campaign evaluators either do not follow best practice guidelines when conducting evaluations or omit these details when disseminating findings, both of which are a disservice to the field. Since campaign planners develop process evaluation procedures specific to their campaign program,¹⁰ it may be that core process evaluation elements are disregarded. Placing greater emphasis on reporting process evaluation likely requires clearer expectations from common dissemination channels (e.g., journals, conferences, and priority audiences).

Descriptions of theory also were lacking. Theory-based evaluation enables evaluators to make connections between campaign activities (e.g., implementation) and desired outcomes to explain how the campaign achieved intended effects.^3,9 While another systematic review on the use of theory in process evaluations (not specific to health campaigns) found nearly two-thirds of articles reported a theoretical approach,²³ fewer (56.5%) in this review reported using theoretical approaches to inform campaign design and evaluation. Importantly, there was extensive variation in the terminology and language used in articles included in the current review. For example, “reach” and “exposure” were sometimes used interchangeably, and other times reported as 2 separate measures. These types of discrepancies can cause confusion, further highlighting the lack of standardized terminology for process evaluation definitions.⁸ Evaluations are more meaningful and can inform future health campaign efforts across contexts when campaign evaluators follow a standardized set of best practices, or at a minimum, ensure all terms are clearly defined when disseminating evaluation findings.

Though some have argued that reporting standards are necessary to enhance the professionalization of the field of evaluation,²¹ establishing standards can have broad practical implications. Nearly half of articles in this review failed to report fidelity in the published findings, yet understanding fidelity helps to contextualize other findings, particularly summative and outcome evaluation results. For smaller scale campaigns, the process may determine whether it is successful or not. While existing reporting practices and available data allow for a determination of general campaign effectiveness, it does little to explain why some campaigns are more effective than others.²⁸ Greater emphasis on reporting process evaluation findings would help to provide the context to answer this question, and inform future health campaigns.

Future Directions and Recommendations for Campaign Process Evaluation

Based on this review, several recommendations for evidence-based evaluation planning are proposed. Reporting process evaluation is a necessary component of conducting evidence-based evaluations. In accordance with existing recommendations, authors should be expected to report 1) the theoretical approach (including references for the theory or framework), 2) formative research and evaluation methods, 3) key campaign details (e.g., location, dates, and health issue), 4) the utility of the evaluation, including how the process evaluation provides insight about mechanisms and intervening variables that could meaningfully impact interpretations of an outcome evaluation, and 5) if and/or how changes were made to implementation.^9,17,20,82 Trieu et al.⁵¹ is an exemplar article from this review that reported all the aforementioned criteria. Moreover, the 9 best practice elements should be assessed and discussed in reports of evaluation findings.¹⁷ At a minimum, process evaluations should always report campaign fidelity (i.e., if the campaign was delivered as planned) in sufficient detail.

Beyond demonstrating effects and establishing effectiveness, campaign evaluation can help to guide future implementation. Balch & Sutton⁵ argue evaluations are most useful when integrated iteratively. Creating a feedback loop improves campaign implementation, yet only about 40% of articles in this review explained how process evaluation findings informed future campaign implementation strategies. Like some articles in this review^33,66 campaign evaluators could incorporate a statement on how process evaluation data will be used, even if it has not been incorporated at the time of writing. Doing so would ensure data from process evaluations can be used to improve and inform subsequent campaign intervention phases. Additionally, campaign planners should establish a system for sharing evaluation feedback with appropriate partners throughout the life of the campaign as a way of ensuring relevance and utility.⁹

Since rich descriptions of campaign implementation reveal the most effective strategies and help others implementing similar campaign efforts, the lack of detail about implementation in published campaign evaluations make them of less value.²⁸ No articles in the current review discussed major failures or extensive barriers that impacted campaign implementation and associated process evaluation findings. Including this type of information would be of particular interest to non-academic audiences implementing campaigns to avoid duplicating failed efforts. Conducting process evaluations to describe campaign implementation is important when a campaign is deemed effective but is arguably more essential when a campaign is ineffective or unsuccessful in achieving desired outcomes to ensure others do not waste valuable resources or repeat past mistakes.^12,13,69

More comprehensive and consistent dissemination of process evaluation research methods and results would illuminate replicable campaign implementation strategies. Process evaluation objectives were present in this review, yet the specificity of those objectives varied widely. Clearly stated evaluation goals and aims specific to the campaign make for a more rigorous evaluation. Findings from this review demonstrate that research methods (e.g., how records related to implementation data were analyzed) are not explicitly stated and inferences had to be made based on the type of data that appeared in the article text, tables, and figures. Rather than providing generic descriptions of research methodology (e.g., “campaign event data were collected monthly”), campaign evaluators should provide enough detail to explain the type of data collected, source of data, frequency with which data was collected, data management systems or procedures, and how data were analyzed so that methods can be replicated by other campaign planners. This is necessary to provide guidance for smaller scale campaigns at the local level with fewer resources to conduct evaluations. Often critiqued for a lack of rigor,^3,10 detailed descriptions of process evaluation methods would increase the validity and reliability of methods and create an index of process evaluation data collection and analysis methods that other campaign planners can replicate.²⁰ Long term this would allow for comparisons across evaluation studies.¹⁷

There are aspects of process evaluation that we know too little about that should be examined via other research on evaluation studies such as this one.⁸³ One is to clarify how process evaluation is used within broader evaluation framework plans. Are evaluators using process evaluation to monitor campaign implementation or using it to explain campaign effectiveness? Should one take priority over the other, particularly when resources are limited? Answering these questions would enhance our understanding of the utility of process evaluation data. Another area for additional research is to examine the issue of excessive data. The amount of process evaluation data and number of metrics collected on a daily, weekly, monthly, or quarterly basis has the potential cause confusion, leading to challenges interpreting results. An examination of what the data mean and how we interpret findings within the context of other evaluation studies is needed. An overabundance of data may result in important data being excluded from analyses or collecting metrics that do not inform the evaluation. From a practical standpoint, collecting more data requires additional resources to ensure accurate collection and analysis, which can be burdensome to smaller campaign teams. How to reduce the burden of data collection while still gathering the necessary information to make informed decisions regarding implementation remains a question.

More research is also needed to determine the frequency with which to collect data. From the current review, it is unclear if daily, weekly, monthly, quarterly—or some combination of these—data collection is sufficient to inform outcome or summative evaluations and future campaign efforts. It is difficult to develop a campaign evaluation plan without an existing evidence base regarding data collection frequency. A survey of the current landscape of process evaluation data collection practices would help to inform additional standards for conducting process evaluation research.

Limitations

At the review level, due to variations in how methods and findings are reported, some assumptions were made during data extraction and coding. For example, many best practice elements were not explicitly stated or used language and definitions that were different from evaluation researchers. Therefore, some best practice elements had to be inferred during coding; however any inferences were systematically applied. This review was limited to peer-reviewed, English language, published articles and is therefore subject to publication bias. While attempts were made to implement an inclusive search strategy, articles may have been unintentionally excluded from this review if they were published as white papers or in the gray literature in databases other than OpenGrey. Similarly, agency and organizational reports where process evaluation methods and findings are often disseminated were not included in this review which may have resulted in a smaller sample size of articles. Process data may appear in subsequent articles that focus on campaign outcome evaluation, but since outcome evaluation articles were excluded from this review, that information was not captured. Finally, for the purpose of this review, article quality—the elements attributable to reporting and study design—was assessed, while bias—the degree to which results of the study should be believed⁸⁴—was not taken into account when reporting findings.

While it is important to determine if a campaign achieves desired outcomes, it is equally as important to identify the elements of the campaign that facilitated those outcomes. Process evaluation can be central to making that determination; yet standardized campaign monitoring procedures and reporting are lacking in the published literature. There is value in adhering to process evaluation best practices so that others can learn from and replicate effective methods and strategies while avoiding ineffective ones, capitalizing on existing resources. More consistent and thorough reporting would lead to more efficient implementation and evaluation. It is only upon improvements to process documentation that we will ensure process evaluation is no longer an afterthought.

So What?

What is already known on this topic?

Process evaluation is often omitted from campaign evaluation research. Researchers suggest evaluating best practices during process evaluation, but it is unclear if these are reported in the literature.

What does this article add?

We reviewed articles about health campaigns with a media component to determine if process evaluation methods, findings, and dissemination practices are reported according to best practice recommendations. Best practice elements are inconsistently reported and language used to describe these elements differed across articles. Campaign evaluators appear to either not follow best practice guidelines when conducting evaluations or omit them when disseminating findings.

What are the implications for health promotion practice or research?

Process evaluations have practical implications for health communicators’ understanding of how, why, and whether campaigns are effective. At a minimum, process evaluations should always report intervention fidelity in adequate detail. Comprehensive and consistent reporting on process evaluation methods would improve our ability to replicate best practices and inform implementation.

Footnotes

Appendix

Example Search Strategy: Embase

(1) “process evaluation”/exp

(2) “process evaluation”:ti,ab

(3) “implementation evaluation”:ti,ab

(4) 1 OR 2 OR 3

(5) campaign:ti,ab

(6) “media”/exp

(7) media:ti,ab

(8) “marketing”/exp

(9) marketing:ti,ab

(10) 5 OR 6 OR 7 OR 8 OR 9

(11) 4 AND 10

Example Search Strategy: PubMed

(1) “process evaluation”[tiab]

(2) “implementation evaluation”[tiab]

(3) 1 OR 2

(4) campaign[tiab]

(5) media[tiab]

(6) “marketing”[Mesh]

(7) “marketing”[tiab]

(8) 4 OR 5 OR 6 OR 7

(9) 3 AND 8

Author Contributions

Hannah Getachew-Smith: Conceptualization, methodology, validation, formal analysis, investigation, data curation, writing—original draft preparation, writing—review and editing, supervision, and project administration.Andy J. King: Conceptualization, methodology, validation, formal analysis, investigation, data curation, writing—original draft, and writing—review and editing.Charlotte Marshall-Fricker: Formal analysis, investigation, data curation, and writing—review and editing.Courtney L. Scherr: Conceptualization, methodology, and writing—review and editing.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Hannah Getachew-Smith

References

Bartholomew

Parcel

Kok

Gottlieb

Fernández

. Planning Health Promotion Programs: An Intervention Mapping Approach. San Francisco, CA: Jossey-Bass; 2011.

Valente

. Evaluating Health Promotion Programs. Oxford, UK: Oxford University Press; 2002.

Nutbeam

. Evaluating health promotion - progress, problems and solutions. Health Promot Int. 1998;13(1):27-44.

McGraw

McKinlay

McClements

Lasater

Assaf

Carleton

. Methods in program evaluation: The process evaluation system of the Pawtucket heart health program. Eval Rev. 1989;13(5):459-483.

Balch

Sutton

. Keep me posted: A plea for practical evaluation. In Goldberg

Fishbein

Middlestadt

, eds. Social Marketing: Theoretical and Practical Perspectives. Mahwah, NJ: Lawrence Erlbaum Associates; 1997:61-74.

Saunders

Evans

Joshi

. Developing a process-evaluation plan for assessing health promotion program implementation: A how-to guide. Health Promot Pract. 2005;6(2):134-147.

Nutbeam

Smith

Catford

. Evaluation in health education. A review of progress, possibilities, and problems. J Epidemiol Community Health. 1990;44(2):83-89.

Saunders

. Implementation Monitoring and Process Evaluation. Thousand Oaks, CA: Sage Publications; 2015.

Freimuth

Cole

Kirby

. Issues in evaluating mass media-based health communication campaigns. In Rootman

Goodstadt

Hyndman

, et al., eds. Evaluation in Health Promotion: Principles and Perspectives. Denmark: World Health Organization Regional Publications; 2001:475-492.

10.

Parry-Langdon

Bloor

Audrey

Holliday

. Process evaluation of health promotion interventions. Policy Polit. 2003;31(2):207-216.

11.

Valente

Kwan

. Evaluating communication campaigns. In Rice

Atkin

, eds. Public Communication Campaigns. 4th ed. Thousand Oaks, CA: Sage Publications; 2013:83-97.

12.

Glasgow

Linnan

. Evaluation of theory-based interventions. In Glanz

Rimer

Viswanath

Orleans

, eds. Health Behavior and Health Education: Theory, Research, and Practice. 4th ed. San Francisco, CA: Jossey-Bass; 2008:487-508.

13.

Steckler

Linnan

. Process Evaluation for Public Health Interventions and Research. San Francisco, CA: Jossey-Bass; 2002.

14.

Fraze

Uhrig

Davis

, et al. Applying core principles to the design and evaluation of the ‘Take Charge. Take the Test’ campaign: What worked and lessons learned. Publ Health. 2009;123(suppl. 1):e23-30.

15.

Noar

. An audience–channel–message–evaluation (ACME) framework for health communication campaigns. Health Promot Pract. 2012;13(4):481-488.

16.

Thompson

Kegler

Holtgrave

. Program evaluation. In Crosby

DiClemente

Salazar

, eds. Research Methods in Health Promotion. 1st ed. San Francisco, CA: Jossey-Bass 2006:199-225.

17.

Baranowski

Stables

. Process evaluations of the 5-a-day projects. Health Educ Behav. 2000;27(2):157-166.

18.

Wilson

Griffin

Saunders

Kitzman-Ulrich

Meyers

Mansard

. Using process evaluation for program improvement in dose, fidelity and reach: The ACT trial experience. Int J Behav Nutr Phys Act. 2009;6(1):79.

19.

Curry

. Perspectives on monitoring and evaluation. Am J Eval. 2018;40(1):147-150.

20.

Des Jarlais

Lyles

Crepaz

. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. Am J Public Health. 2004;94(3):361-366.

21.

Montrosse-Moorhead

Griffith

. Toward the development of reporting standards for evaluations. Am J Eval. 2017;38(4):577-602.

22.

Linnan

Steckler

. Process evaluation for public health interventions and research: An overview. In Steckler

Linnan

, eds. Process Evaluation for Public Health Interventions and Research. San Francisco, CA: Jossey-Bass; 2002:1-23.

23.

McIntyre

Francis

Gould

Lorencatto

. The use of theory in process evaluations conducted alongside randomized trials of implementation interventions: A systematic review. Transl Behav Med. 2018;10(1):168-178.

24.

Luca

Suggs

. Theory and model use in social marketing health interventions. J Health Commun. 2013;18(1):20-40.

25.

Anker

Feeley

McCracken

Lagoe

. Measuring the effectiveness of mass-mediated health campaigns through meta-analysis. J Health Commun. 2016;21:439-456.

26.

Moher

Liberati

Tetzlaff

Altman

the

. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Ann Intern Med. 2009;151(4):264-269.

27.

Liberati

Altman

Tetzlaff

, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. J Clin Epidemiol. 2009;62(10):e1-e34.

28.

Robinson

Tansil

Elder

, et al. Mass media health communication campaigns combined with health-related product distribution: A community guide systematic review. Am J Prev Med. 2014;47(3):360-371.

29.

Wilson

. Systematic coding. In Cooper

Hedges

Valentine

, eds. The Handbook of Research Synthesis and Meta-Analysis. 2nd ed. New York, NY: Russell Sage Foundation; 2009:159–176.

30.

Baumann

Koller

Wiltfang

Wenz

Moller

Hertrampf

. Challenges of early detection of oral cancer: Raising awareness as a first step to successful campaigning. Health Educ Res. 2016;31(2):136-145.

31.

Belanger-Gravel

Gauvin

Lagarde

Laferte

. Initial recall and understanding of a multimedia communication campaign to promote physical activity among tweens: A process evaluation study. Prev Med. 2014;69:192-196.

32.

Berkowitz

Huhman

Heitzler

Potter

Nolin

Banspach

. Overview of formative, process, and outcome evaluation methods used in the VERB campaign. Am J Prev Med. 2008;34(6 suppl l):S222-S229.

33.

Bhattacharya

Hopkins

Sallis

Budd

Ashiru-Oredope

. A process evaluation of the UK-wide antibiotic guardian campaign: Developing engagement on antimicrobial resistance. J Public Health. 2017;39(2):e40-e47.

34.

Chung

. A smoking cessation campaign on Twitter: Understanding the use of Twitter and identifying major players in a health campaign. J Health Commun. 2016;21(5):517-526.

35.

Coulon

Wilson

Griffin

, et al. Formative process evaluation for implementing a social marketing intervention to increase walking among African Americans in the positive action for today’s health trial. Am J Public Health. 2012;102(12):2315-2321.

36.

Evers

Jones

Iverson

Caputi

. ‘Get your life back’: Process and impact evaluation of an asthma social marketing campaign targeting older adults. BMC Publ Health. 2013;13(1):759.

37.

Fell

Tanenbaum

Chelluri

. Evaluation of a combination of community initiatives to reduce driving while intoxicated and other alcohol-related harms. Traffic Inj Prev. 2018;19(suppl. 1):S176-S179.

38.

Gough

Hunter

Ajao

, et al. Tweet for behavior change: Using social media for the dissemination of public health messages. JMIR Public Health Surveill. 2017;3(1):e14.

39.

Hong

Johnson

Myers

Boris

Brewer

Webber

. Process evaluation of an in-school anti-tobacco media campaign in Louisiana. Public Health Rep. 2008;123(6):781-789.

40.

Honigh-de Vlaming

Haveman-Nies

Heinrich

van’t Veer

de Groot

. Effect evaluation of a two-year complex intervention to reduce loneliness in non-institutionalised elderly Dutch people. BMC Publ Health. 2013;13:984.

41.

Jones

Andrews

Francis

. Combining social norms and social marketing to address underage drinking: Development and process evaluation of a whole-of-community intervention. PLoS One. 2017;12(1):e0169872.

42.

Katz

Young

Reiter

, et al. Process evaluation of cancer prevention media campaigns in Appalachian Ohio. Health Promot Pract. 2017;18(2):201-210.

43.

Nguyen

Gold

Pedrana

, et al. Sexual health promotion on social networking sites: A process evaluation of the FaceSpace project. J Adolesc Health. 2013;53(1):98-104.

44.

Nguyen

Kim

Nguyen

, et al. Exposure to mass media and interpersonal counseling has additive effects on exclusive breastfeeding and its psychosocial determinants among Vietnamese mothers. Matern Child Nutr. 2016;12(4):713-725.

45.

Pedrana

Hellard

Gold

, et al. Queer as f**k: Reaching and engaging gay men in sexual health promotion through social networking sites. J Med Internet Res. 2013;15(2):e25.

46.

Potente

McIver

Anderson

Coppa

. ‘It’s a beautiful day for cancer’: An innovative communication strategy to engage youth in skin cancer prevention. Soc Mar Q. 2011;17(3):86-105.

47.

Reininger

Barroso

Mitchell-Bennett

, et al. Process evaluation and participatory methods in an obesity-prevention media campaign for Mexican Americans. Health Promot Pract. 2010;11(3):347-357.

48.

Rosecrans

Gittelsohn

Harris

Naqshbandi

Sharma

. Process evaluation of a multi-institutional community-based program for diabetes prevention among First Nations. Health Educ Res. 2008;23(2):272-286.

49.

Sandt

Marshall

Ennett

. Community-based pedestrian and bicycle safety program: Developmental framework and process evaluation. Transp Res Rec. 2015;2519(1):51-60.

50.

Tewari

Kallakuri

Devarapalli

Jha

Patel

Maulik

. Process evaluation of the systematic medical appraisal, referral and treatment (SMART) mental health project in rural India. BMC Psychiatr. 2017;17(1):385.

51.

Trieu

Webster

Jan

, et al. Process evaluation of Samoa’s national salt reduction strategy (MASIMA): What interventions can be successfully replicated in lower-income countries? Implement Sci. 2018;13(1):107.

52.

Van Der Kleij

Crone

Paulussen

Van De Gaar

Reis

. A stitch in time saves nine? A repeated cross-sectional case study on the implementation of the intersectoral community approach youth at a healthy weight health behavior, health promotion and society. BMC Publ Health. 2015;15(1):1032.

53.

Webster

Pillay

Suku

, et al. Process evaluation and costing of a multifaceted population-wide intervention to reduce salt consumption in Fiji. Nutrients. 2018;10(2):155.

54.

Gall Casey

Lebovic

Straus

. Process evaluation of the diabetes Canada guidelines dissemination strategy using the reach effectiveness adoption implementation maintenance (RE-AIM) framework. Can J Diabetes. 2018;43(4):263-270.e269.

55.

Zhang

Huhn

Tan

, et al. “Testing is healthy” TimePlay campaign: Evaluation of sexual health promotion gamification intervention targeting young adults. Can J Public Health. 2017;108(1):e85-e90.

56.

Curran

Gittelsohn

Anliker

, et al. Process evaluation of a store-based environmental obesity intervention on two American Indian reservations. Health Educ Res. 2005;20(6):719-729.

57.

Nicklas

Johnson

Farris

Rice

Lyon

Shi

. Development of a school-based nutrition intervention for high school students: Gimme 5. Am J Health Promot. 1997;11(5):315-322.

58.

Nicklas

O'Neil

. Process of conducting a 5-a-day intervention with high school students: Gimme 5 (Louisiana). Health Educ Behav. 2000;27(2):201-212.

59.

Van Der Feen De Lille

JCJF

Riedstra

Hardeman

, et al. Fat watch: A nationwide campaign in the Netherlands to reduce fat intake-process evaluations. Nutr Health. 1998;12(2):107-117.

60.

Verrall

Napash

Leclerc

Mercure

Gray-Donald

. Community-based communication strategies to promote infant iron nutrition in northern Canada. Int J Circumpolar Health. 2006;65(1):65-78.

61.

Davis

Lee

Jarvis

Zorbas

Redman

. Public health information programmes: Can partnerships reduce costs? Health Educ. 2003;103(6):352-362.

62.

Hammond

Freimuth

Morrison

. The gatekeeping funnel: Tracking a major PSA campaign from distribution through gatekeepers to target audience. Health Educ Q. 1987;14(2):153-166.

63.

Alstead

Campsmith

Halley

Hartfield

Goldbaum

Wood

. Developing, implementing, and evaluating a condom promotion program targeting sexually active adolescents. AIDS Educ Prev. 1999;11(6):497-512.

64.

Andersson

Bjaras

Tillgren

Ostenson

. Local media monitoring in process evaluation: Experiences from the Stockholm diabetes prevention programme. J Health Commun. 2007;12(3):269-283.

65.

Blake

Jeffery

Finnegan

, et al. Process evaluation of a community-based physical activity campaign: The Minnesota Heart Program experience. Health Educ Res. 1987;2(2):115-121.

66.

Corbett

Thompson

White

Taylor

. Process evaluation in the community intervention trial for smoking cessation (commit). Int Q Community Health Educ. 1990;11(3):291-309.

67.

Hausman

Spivak

Prothrow-Stith

Roeber

. Patterns of teen exposure to a community-based violence prevention project. J Adolesc Health. 1992;13(8):668-675.

68.

Jalleh

Donovan

James

Ambridge

. Process evaluation of the act-belong-commit mentally healthy WA campaign: First 12 months data. Health Promot J Austr. 2007;18(3):217-220.

69.

Norman

Greenberg

Marconi

, et al. A process evaluation of a two-year community cardiovascular risk reduction program: What was done and who knew about it? Health Educ Res. 1990;5(1):87-97.

70.

Ronda

Van Assema

Ruland

Steenbakkers

Brug

. The Dutch heart health community intervention ‘Hartslag Limburg’: Design and results of a process study. Health Educ Res. 2004;19(5):596-607.

71.

Casswell

Ransom

Gilmore

. Evaluation of a mass-media campaign for the primary prevention of alcohol-related problems. Health Promot Int. 1990;5(1):9-17.

72.

Rosenstock

. The health belief model and preventive health behavior. Health Educ Monogr. 1974;2(4):354-386.

73.

Ajzen

. From intentions to actions: A theory of planned behavior. In: Kuhl

Beckmann

, eds. Action Control: From Cognition to Behavior. Berlin: Springer; 1985:11-39.

74.

Bandura

. The explanatory and predictive scope of self-efficacy theory. J Soc Clin Psychol. 1986;4(3):359-373.

75.

Prochaska

DiClemente

. Stages and processes of self-change of smoking: Toward an integrative model of change. J Consult Clin Psychol. 1983;51(3):390-395.

76.

Glasgow

Vogt

Boles

. Evaluating the public health impact of health promotion interventions: The RE-AIM framework. Am J Public Health. 1999;89(9):1322-1327.

77.

Green

Kreuter

. The PRECEDE-PROCEED model of health program planning and evaluation. Health Promotion Planning: An Educational and Ecological Approach (4th ed.). NY: McGraw-Hill. 2005.

78.

Koelen

Hielkema-de Meij

van der Sanden-Stoelinga

MSE

. ‘Bottle it up – take a cup!’ The fight against nursing caries in the Netherlands: The campaign and its results. Int J Health Promot Educ. 2000;38(2):47-53.

79.

Friedman

Kachur

Noar

McFarlane

. Health communication and social marketing campaigns for sexually transmitted disease prevention and control: What is the evidence of their effectiveness? Sex Transm Dis. 2016;43(2 suppl 1):S83-S101.

80.

Snyder

Hamilton

Mitchell

Kiwanuka-Tondo

Fleming-Milici

Proctor

. A meta-analysis of the effect of mediated health communication campaigns on behavior change in the United States. J Health Commun. 2004;9(1):71-96.

81.

Noar

. A 10-year retrospective of research in health mass media campaigns: Where do we go from here? J Health Commun. 2006;11(1):21-42.

82.

Community Preventive Services Task Force . Combination of mass media health campaigns and health-related product distribution is recommended to improve healthy behaviors. Am J Prev Med. 2014;47(3):372-374.

83.

Coryn

CLS

Wilson

Westine

, et al. A decade of research on evaluation: A systematic review of research on evaluation published between 2005 and 2014. Am J Eval. 2017;38(3):329-347.

84.

Higgins

JPT

Savović

Page

Elbers

Sterne

. Assessing risk of bias in a randomized trial. In: Higgins

JPT

Thomas

Chandler

Cumpston

Page

Welch

, eds. Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021: 205-228. Available from www.training.cochrane.org/handbook.