Abstract
Family-and-home-based interventions are an important vehicle for preventing childhood obesity. Systematic process evaluations have not been routinely conducted in assessment of these interventions. The purpose of this study was to plan and conduct a process evaluation of the Enabling Mothers to Prevent Pediatric Obesity Through Web-Based Learning and Reciprocal Determinism (EMPOWER) randomized control trial. The trial was composed of two web-based, mother-centered interventions for prevention of obesity in children between 4 and 6 years of age. Process evaluation used the components of program fidelity, dose delivered, dose received, context, reach, and recruitment. Categorical process evaluation data (program fidelity, dose delivered, dose exposure, and context) were assessed using Program Implementation Index (PII) values. Continuous process evaluation variables (dose satisfaction and recruitment) were assessed using ANOVA tests to evaluate mean differences between groups (experimental and control) and sessions (sessions 1 through 5). Process evaluation results found that both groups (experimental and control) were equivalent, and interventions were administered as planned. Analysis of web-based intervention process objectives requires tailoring of process evaluation models for online delivery. Dissemination of process evaluation results can advance best practices for implementing effective online health promotion programs.
Keywords
Introduction
The family and home environment has been suggested as a robust leverage point for the prevention of childhood obesity (Haire-Joshu et al., 2008). Notwithstanding, a number of barriers confront advancement of interventions seeking to capitalize on this milieu. A recent systematic review identified the prominent shortcomings of family-and-home-based childhood obesity interventions (Knowlden & Sharma, 2012a). First, there is a general absence of process evaluation transpiring within this intervention prototype. The analysis found that only two of the interventions that operationalized the home environment incorporated process evaluation. Among these, process evaluation data pertaining to intervention implementation were not reported. Second, researchers testing the efficacy of these interventions have noted higher attrition rates, typically falling between 20% and 50% (Warren et al., 2007). Within the context of the home environment, parents are hypothesized to be the primary mediators of behavior change in children (Golan & Crow, 2004). The time commitment required of parents to participate in traditional, face-to-face educational sessions has, however, remained a salient barrier undermining the utility of the family-and-home-based intervention prototype for addressing childhood obesity.
Background
The Enabling Mothers to Prevent Pediatric Obesity Through Web-Based Learning and Reciprocal Determinism (EMPOWER) randomized control trial was designed to begin addressing these deficiencies. To compensate for the absence of process evaluation, process objectives were crafted during the design phase of the program. Ultimately, six layers of process evaluation were incorporated into the trial to assess quality control. In an attempt to reduce attrition, the trial was delivered online. It was hypothesized that web-based delivery of the interventions would provide greater convenience to parents and, ultimately, offset attrition.
Details concerning the design and impact evaluation of the trial have been reported elsewhere (Knowlden, 2013). Briefly, the 8-week trial compared two web-based, mother-centered interventions for prevention of obesity in children between 4 and 6 years of age. Both interventions were developed to support mothers in assisting their children meet four behaviors known to prevent childhood obesity: (a) 120 minutes of daily structured and unstructured physical activity, (b) consumption of 5 cups of fruits and vegetables each day, (c) replacement of sugar-sweetened beverages with sugar-free beverages, and (d) limitation of screen time (leisure time spent in front of a television, computer screen, or portable electronic device) to no more than 120 minutes each day (Clark et al., 2002; Rao, 2008). Both interventions were composed of five educational sessions, with one session devoted to each child behavior. Between posttest (week 4) and follow-up measurement (week 8), a comprehensive booster session was delivered. Educational modalities for each session included (a) one 10-to-15-minute audiovisual presentation, (b) one interactive worksheet, and (c) one discussion board post.
Outcome measures for the trial included the four childhood obesity behaviors (physical activity, fruit and vegetable consumption, sugar-free beverage consumption, and screen time) as well as five maternal-mediated social-cognitive theory (SCT) constructs (environment, emotional coping, expectations, self-control, and self-efficacy) theorized to influence these behaviors (Knowlden & Sharma, 2012b). Impact evaluation of the EMPOWER trial found that child physical activity, sugar-free beverage consumption, and screen time improved in both groups, whereas fruit and vegetable behavior improved only in the theory-based arm of the trial. Change score analysis found improvement in the home environment. SCT was significantly associated with the increase in child fruit and vegetable behavior for the theory-based group.
In designing the trial, efforts were undertaken to ensure equivalency of the interventions; specifically, both interventions were delivered using the same online platform, contained five sessions, were delivered simultaneously, included the same educational modalities, and applied the same measurement tool at pretest, posttest, and follow-up. The primary difference between the interventions was the pedagogical foundation. The experimental EMPOWER intervention was framed according to SCT (Bandura, 2004), whereas content related to the active control, Healthy Lifestyles intervention, was designed to improve general knowledge regarding childhood obesity. SCT was chosen as the theoretical framework for this study for three primary reasons. First, it was the theory applied most frequently (n = 2) in a systematic review of family-and-home-based childhood obesity interventions (Knowlden & Sharma, 2012a). Second, SCT allows for reification of the environment, which is beneficial for targeting the home milieu. Finally, SCT is rooted in vicarious learning. Evidence suggests that children develop food acceptance and preferences, consumption patterns, and proclivity to try new foods primarily through observation of parental role models (Patrick & Nicklas, 2005).
The process evaluation methodology used in this study was based on the framework organized by Saunders, Evans, and Joshi (2005). The Saunders et al. model was developed for use with targeted health promotion programs and is composed of six components: program fidelity, dose delivered, dose received, context, reach, and recruitment. Because the components of the model are generic in nature, it was determined that the elements of the framework could be operationally defined for conducting web-based process evaluation. The purpose of the present study was to report process evaluation findings from the study, identify output inefficiencies, and provide practical techniques for replicating similar types of web-based programs.
Methods
University Institutional Review Board (IRB) approval to conduct the study was requested and granted prior to initiating the trial. Process evaluation data were collected after completion of each educational session using scripted telephone counseling and online surveys. Results were recorded on an aggregate data form, which was assessed by a panel of four experts prior to conducting the trial. The role of the panel was to determine the capacity of the data form to accurately represent the six components of the Saunders et al. (2005) model. Process evaluation data were managed using spreadsheets created in Microsoft Excel 2010. All data were collected and analyzed by the principal investigator.
Program Fidelity
Program fidelity measured the extent to which the interventions were implemented as planned. Both interventions were composed of five sessions, with each session including one video presentation, one discussion board thread, and one worksheet. These three measures were individually scored as being either implemented (scored as 1) or not implemented (scored as 0), for a total possible score range of 0 to 3 for each session. The total possible score of 3 per session was divided against the actual total score received per session to calculate the rate of program fidelity for each session.
Dose Delivered
Dose delivered assessed the number of intended sessions delivered to eligible participants. To assess dose delivery, participants were sent an e-mail message, alerting them when each educational session was available. If a failure of delivery notice was received, telephone contact was made with the participant to ensure delivery. Dose delivered was checked against a data-mining metric that provided the date and time at which each participant accessed the web platform. Dose delivery of each of the five sessions was scored as being either delivered (scored as 1) or not delivered (scored as 0) to eligible participants of each session. The total possible score of 1 for each eligible participant was divided against the actual total score received by each eligible participant to calculate the rate of dose delivered for each session.
Dose Received
Dose received was composed of two components: dose exposure and dose satisfaction. Dose exposure evaluated the extent to which participants actively engaged, interacted, and applied intervention materials. Both interventions were composed of five sessions, with each session including one video presentation, one discussion board thread, and one worksheet. These three measures were individually scored as being either completed (scored as 1) or not completed (scored as 0) by eligible participants, for a total possible score range of 0 to 3 for each eligible participants in each session. The total possible score of 3 for each eligible participant was divided against the actual total score received by each eligible participant to calculate the rate of dose exposure for each session. Dose satisfaction was collected through participant self-report of the perceived benefit of each modality and the overall benefit of the program to the health of the participant’s child. Dose satisfaction was composed of four 5-point Likert-type scales with endpoints ranging from strongly disagree to strongly agree, for a possible score range of 4 to 20 for the eligible participants of each session.
Context
Context was evaluated to assess potential cross-contamination between the groups (i.e., the extent to which the control group was exposed to the experimental program and vice versa). Context data were collected through participant self-report by querying participants’ transmission of program information to any individuals other than immediate family. Context was considered as either not occurring (scored as 1) or occurring (scored as 0), for a possible score range of 0 to 1 for eligible participants of each session. The total possible score of 1 for each eligible participant was divided against the actual total score received by each eligible participant to calculate the rate of context for each session.
Reach
Reach assessed attendance of the intervention audience. Reach data were collected through participant self-report of navigability of the website and ability to find the video presentation, discussion board thread, and worksheet modalities. This measure was also assessed by the principal investigator’s ability to collect process evaluation data from each participant. These four measures were individually scored as being either attained (scored as 1) or not attained (scored as 0) by eligible participants, for a total possible score range of 0 to 4 for each eligible participants in each session. The total possible score of 4 for each eligible participant was divided against the actual total score received by each eligible participant to calculate the rate of reach for each session.
Recruitment
Recruitment was applied to gauge program feasibility and practicality. Data were collected through participant self-report of the perception of effort (feasibility) to complete each session and the time spent on each module (practicality). Recruitment feasibility was composed of one 5-point Likert-type scale with endpoints ranging from very unreasonable to very reasonable. For recruitment practicality, self-reported time spent on each session was requested in total minutes. The a priori dose of both interventions was 2.5 hours (five total sessions with a completion time of approximately 30 minutes per session). Attrition was considered part of recruitment. Three attempts were made to contact participants who failed to complete intervention modalities. For those who did not respond after three attempts, it was assumed the participant had dropped out of the program.
Analyses
Categorical process evaluation data (program fidelity, dose delivered, dose exposure, and context) were assessed using Program Implementation Index (PII) values. PII values allow for comparison between actual implementation and an a priori performance standard. PII was calculated in Microsoft Excel 2010, using the formula provided by Windsor et al. (2000). The first step was to calculate a rate for each categorical measure of process evaluation. Rates were calculated by dividing total possible scores by actual total scores received. Next, the calculated rates were divided by the a priori performance standard of 95% to ascertain session-based implementation index values for each categorical measure. Finally, a composite value of all session-based implementation indexes was calculated to provide an overall performance PII value for each categorical process evaluation measure. Composite PII values were calculated for both the experimental and control groups, and compared to determine if the programs were implemented as planned. Implementation was considered equivalent between groups if composite PII values were greater than or equal to 0.90 (Windsor, Clark, Boyd, & Goodman, 2004). Continuous process evaluation variables (dose satisfaction and recruitment) were assessed using ANOVA tests to evaluate mean differences between groups (experimental and control) and sessions (sessions 1 through 5). ANOVAs were calculated using Predictive Analytics Software (PAWS) Statistics Grad Pack 18.0. Significance levels for continuous variables were set a priori at a p-value less than 0.05.
Results
PIIs indicated that program fidelity, dose delivered, dose received (exposure), reach, and context exceeded the 0.90 level of implementation excellence for both groups. Table 1 provides a summary of results for these process evaluation measures. Inspection of results found that worksheets for sessions 2 and 4 were not completed by two separate individuals in the experimental group. Subsequently, the dose exposure rate for these two sessions fell to slightly lower than 100%. Process evaluation of the experimental intervention found that one participant failed to complete the session 1 review, two participants failed to complete the session 2 review, and two participants failed to complete the session 4 review. These results affected the rate of reach. The two participants who failed to complete the reviews for session 2 eventually withdrew from the program, so it is unlikely that this deficiency led to a significant group imbalance at the time of posttest impact evaluation. Furthermore, completion of reviews was not directly linked to intervention prescription. Subsequently, although these individuals failed to complete the session reviews, they did complete all other session activities during the time they were enrolled in the trial. Therefore, it is unlikely the reduced rate of reach affected the final analysis of the study variables.
Summary of Process Evaluation Results for Program Fidelity, Dose Delivered, Dose Exposure, Context, and Reach
NOTE: EMPOWER group data provided on the left-hand side of the colon; Healthy Lifestyles group data provided on the right-hand side of the colon. Eligibility refers to total participants in each group enrolled in the trial at the time of data collection; eligibility status was removed once dropout was confirmed (through either direct contact or three failed attempts at contact).
Results indicated an absence of statistically significant differences between the two groups for dose satisfaction and recruitment feasibility. Table 2 provides a summary of results for dose satisfaction. Table 3 provides a summary of results for recruitment feasibility. The EMPOWER trial was spread among five sessions during a 2-month timeframe. Concerning recruitment practicality, the average time spent completing the entire EMPOWER (experimental) intervention was 149.21 minutes. The average time spent completing the entire Healthy Lifestyles (control) intervention was 127.50 minutes. These durations were near the estimated dose of 2.5 hours set for the program. A total of seven participants were lost to follow-up during the course of the trial. Reasons for participant dropout are summarized in Table 4.
Summary of Process Evaluation Results for Dose Satisfaction
NOTE: EMPOWER group data provided on the left-hand side of the colon; Healthy Lifestyles group data provided on the right-hand side of the colon.
p-value > 0.050.
Summary of Process Evaluation Results for Recruitment Feasibility
NOTE: EMPOWER group data provided on the left-hand side of the colon; Healthy Lifestyles group data provided on the right-hand side of the colon.
p-value > 0.050.
Participant Reasons for Dropout
Lost at posttest.
Lost at follow-up.
Discussion
Program Fidelity and Dose Delivered
Program fidelity and dose delivered concerned the delivery accuracy of the prescribed protocol. In the case of the EMPOWER trial, fidelity and dose delivered were primarily internal quality control measures. Each of the five educational sessions was developed and uploaded to the platform prior to launching the trial. Program fidelity encompassed reviewing the components of each module against a checklist to ensure all components were present and accessible to participants. Dose delivered entailed assurance that each participant was aware of when each session became available. Inclusion criteria for the program required participants to provide a current e-mail and phone number. Subsequently, dose delivered was primarily assessed by successful delivery of e-mail messages. If a failure of delivery notice was not received, it was assumed that dose delivery was successful.
Dose Received
Dose received comprised two elements: dose exposure and dose satisfaction. Dose satisfaction surveyed participant satisfaction with programmatic modalities. Dose satisfaction results suggested that participants were satisfied with the intervention modalities and perceived the overall program as beneficial for improving the health of their children. Dose exposure evaluated the extent to which participants actively engaged, interacted, and applied intervention materials. As reported elsewhere (Knowlden, 2013), three determinants of childhood obesity improved in both groups: child physical activity, sugar-sweetened beverage consumption, and screen time. Improvement of child fruit and vegetable consumption was exclusive to the theory-based group. Improvement of these behaviors lends credence to the self-reported dose satisfaction and exposure data provided by participants.
Context
Impact evaluation found the EMPOWER intervention was efficacious in improving child fruit and vegetable consumption relative to the Healthy Lifestyles intervention (Knowlden, 2013). As part of the trial inclusion criteria, participants could not be enrolled in external weight management programs. Context results suggested minimal cross-contamination between the experimental and active control arms of the trial, which supports internal validity of the trial findings.
Reach
The learning curve associated with use of a website is a crucial factor that interventionists must consider. During the program, a number of participants commented that they had a difficult time maneuvering the web platform. The principal investigator worked diligently with these individuals to assist them with their technical difficulties. This additional effort on the part of the participants, however, could have reduced overall motivation for the program.
Recruitment
Given that attrition was less than 20% in both groups (13.8% in the experimental group and 10.7% in the control group) and was distributed comparatively evenly between the groups, it was assumed any potential bias introduced through attrition would be minimal and controlled through intention-to-treat (ITT) analysis (Fergusson, Aaron, Guyatt, & Hebert, 2002). A number of prospective personal factors appeared to have led to attrition, including practical reasons (inadequate time), necessities (unexpected illness and computer issues), and/or disillusionment (lack of interest and sensitivity of the topic). Overall, attrition for the EMPOWER trial fell to lower than the norm for this intervention prototype, which promotes the efficaciousness of web-based delivery of family-and-home-based childhood obesity interventions.
Conclusions
The purpose of this study was to report process evaluation of two online family-and-home-based childhood obesity interventions for mothers of children between 4 and 6 years of age. A systematic review of previously conducted family-and-home-based childhood obesity interventions found few with collected process evaluation data (Knowlden & Sharma, 2012a). Among those that did (n = 2), process analysis was limited to participant satisfaction. The EMPOWER trial expanded on previous work by incorporating a complete process evaluation model and by collecting data pertaining specifically to intervention implementation.
Results suggested that both programs were equivalent and administered as planned. The inclusion of process evaluation aided in assessing the true efficacy of the intervention’s impact by reducing the risk of committing type 3 errors.
Recommendations
Program fidelity could be improved by having two independent observers assess the fidelity accuracy of the prescribed protocol. Level of agreement between the two observers could be calculated and reported. A popular measure of interrater agreement is Cohen’s kappa. Cohen’s kappa protects against agreement that might occur through chance alone (i.e., random error). The EMPOWER trial was a pilot efficacy trial, and although interrater agreement would have strengthened evaluation findings, it was not deemed necessary during the planning phase. For larger scale interventions, independent agreement of program fidelity should be conducted and reported.
Dose delivery accuracy for future interventions could be improved by providing additional inclusion criteria to account for e-mail recovery frequency. From a practical perspective, participants may not log in to their e-mail accounts on a regular basis. Because e-mail was the primary mechanism for communicating program announcements, failure of participants to check their e-mail account on a regular basis could influence both retention and prescription adherence. Advanced data-mining metrics should be incorporated into future programs to objectively evaluate if participants were fully compliant with the complete intervention protocol. Data mining for the EMPOWER trials was minimal, because the platform used in this study was not equipped with these features. Examples of beneficial data mining could include monitoring the duration of time spent on each portion of a given web page as well as ensuring that each page was accessed.
For the EMPOWER trial, it was assumed that participation in secondary weight management programs was controlled through inclusion criteria. It is possible, however, that participants were enrolled in health promotion or health education programs that, although not directly targeting childhood obesity, targeted behaviors that inadvertently affected determinants of childhood obesity. Future studies should consider querying participants about any health-related activities that may be occurring in formal child care settings, such as day care. In this way, factors external to the intervention that may indirectly affect the targeted variables could be statistically controlled.
Future programs should focus on reducing the learning curve associated with website navigation. Website design must be user-friendly and should be tested by a small sample of the target population for navigability prior to beginning the intervention. This solution could be further bolstered by hosting a minimum of one on-site introductory module prior to the online intervention. A face-to-face session could provide participant introductions, a course overview, and training for accessing the online course content. In addition, an in-person introductory session could foster relationships between participants and program administrators, which, in turn, could increase motivation for program engagement.
Reach could be further strengthened be easing program accessibility. Mobile phone applications are becoming an increasingly popular method for accessing the Internet. Time constraints are a known barrier for family-and-home-based intervention participant retention. During the program, a number of participants inquired about mobile phone access to the intervention; however, a cost was associated with the mobile phone application, and lack of funding did not permit purchasing licenses for participant use. It is recommended that future programs consider funding the development of mobile phone applications.
A measure of recruitment that was not collected was time spent improving each behavior. For example, participants in the EMPOWER intervention may have spent an average of 30 minutes completing intervention activities for session 3 related to physical activity; however, this quantity does not account for the time that the participants dedicated to improving physical activity in their children. For instance, it was requested that parents increase their child’s physical activity to 120 minutes per day. If, for example, a participant administered only 30 minutes of physical activity per day at pretest, administering an additional 90 minutes of physical activity as prescribed by the intervention should be factored into recruitment time to fully gauge program feasibility. These data could have been collected through daily physical activity and nutrition logs, and subsequently factored into recruitment time. Although these data would have been beneficial from a research perspective, it was determined that requesting daily measurement would be too burdensome on participants. Future programs should consider collecting behavioral data through noninvasive objective measures, such as pedometers or accelerometers, to fully operationalize program feasibility.
Child custody information was not requested from separated or divorced mothers during recruitment. One participant in the experimental arm of the trial did not have custody of her child during session 2 of the study. A makeup session was provided to this participant to aid in retention. Recruitment could be improved by providing additional inclusion criteria to account for child custody. Formative evaluation should occur prior to implementation to account for any additional unforeseen circumstances. It should be noted that this recommended inclusion criteria would primarily benefit programs with time constraint boundaries similar to those required of the current study. Interventions that use ongoing recruitment or that have more flexible time constraints may, however, be able to accommodate periods in which separated or divorced participants would not have direct access to their children.
Implications for Practice
When evaluating health promotion interventions, the ability to gauge accuracy of program delivery is critical for objective assessment of program outcomes. Ideally, process objectives are developed during the planning stages of an intervention and are evaluated throughout a program life cycle. In the absence of process evaluation, there is an increased risk of committing type 3 errors, which, in turn, can lead to spurious impact and outcome evaluation (Basch, Sliepcevich, Gold, Duncan, & Kolbe, 1985). From an evaluation perspective, type 3 errors occur when weak or null program outcomes result, not because of the intervention prescription, but because of inconsistent or inadequate intervention delivery. When contrasting the efficacy of two or more interventions for research purposes, type 3 errors can confound comparability by artificially inflating the perceived utility of one of the intervention arms.
During the development of web-based interventions, it is important to consider that the implementation of online programs differs from traditional intervention delivery and requires tailoring of the process evaluation methodology. For the EMPOWER trial, process evaluation data were collected through telephone counseling, online surveys, and data mining. These methods of data collection were efficient and provided quality feedback of program implementation. From a program management perspective, process evaluation data can aid in identifying delivery inefficiencies. Specifically, by isolating output inefficiencies, program planners can improve outcomes for future delivery of an intervention. In the current study, the incorporation of process evaluation elucidated improvements for future replications of the interventions tested in this trial. Furthermore, many of the improvements appeared applicable to general web-based interventions.
In considering the findings of this report, it is important to note that the overall study of web-based interventions is in its infancy. Much work remains to fully determine the efficacy and utility of this intervention delivery vehicle. This is even moreso the case with web-based health promotion process evaluation. It is therefore recommended that practitioners and researchers who are implementing either partial or full web-based programs incorporate measures of process evalaution and disseminate their findings through professional channels to advance the practice of health promotion.
Footnotes
This study was supported by a University of Cincinnati Graduate School Distinguished Dissertation Completion Fellowship.