Process Evaluation of an Intervention to Increase Provision of Adolescent Vaccines at School Health Centers

Participants

Researchers with Cervical Cancer-Free North Carolina (CCFNC), a statewide collaboration to foster cervical cancer prevention practices and research, partnered with an organization that managed SHCs in all four traditional high schools in one district in central North Carolina. The district student population was 63% White, 20% African American, 11% Hispanic, and 6% Other race/ethnicity, and 59% of students were eligible for free and reduced lunch (Rockingham County Schools, n.d.). Study participants were parents of 2,975 students enrolled in the four SHCs in September 2011.

Procedure

We created a vaccine information packet that included a cover letter signed by the SHC’s director, a promotional flyer, a consent form, two vaccine information sheets, and a preaddressed, stamped return envelope. Materials were two-sided in color, with Spanish translations on the back. The cover letter emphasized the convenience of the SHC, the need for and safety of adolescent vaccines, and the importance of returning a completed consent form. The promotional flyer emphasized similar points but also added quotes and images of parents and teens. We designed the consent form to limit the amount of information parents needed to provide, removing requests for information already available in SHC files, and focusing on allergies and previous vaccination background. In the consent area, we included a check box allowing parents to globally consent to all recommended vaccines, or an option to check individual vaccinations. One information sheet explained Tdap, meningitis, HPV, and seasonal flu vaccines, and the second information sheet explained vaccines recommended earlier in childhood.

SHC staff implemented several procedures to encourage consent form return. Staff mailed the packets to parents to ensure they received it. Both before and after packet mailing, staff used the school’s automated phone messaging system to call all parents with reminders about the consent forms. They sent undelivered packets to parents in students’ backpacks. Students who returned forms by a specified date had a chance to win four movie tickets. The intervention as originally designed appears in the top half of Figure 1.

OPEN IN VIEWER

Figure 1.

Intervention implementation as planned and modified.

Instruments

Data Analysis

We analyzed quantitative data from the materials tracking report, the parent interviews, and the resource reports using SPSS Version 18 (IBM SPSS, Armonk, NY). Using qualitative data from parent interviews and staff communications, we identified themes related to our four process evaluation questions. For each data source, one CCFNC team member (SDG) first read through the open-ended material to identify topical codes and create a codebook. Two CCFNC team members (SDG, KEM) then independently used the codebook to identify themes that emerged from the data, resolving any discrepancies by consensus.

Intervention Modifications

During implementation, we worked with SHC staff to make two modifications to the original intervention design. SHC staff had received questions from parents about the cost of needed vaccinations and whether consented male adolescents would receive the HPV vaccine. Both modifications consisted of additional communications with parents, as shown in the bottom half of Figure 1. The first modification was a letter and follow-up phone calls to 162 consenting parents of children with private health insurance that described the full price of each vaccine, and explained that parents would be responsible for paying any vaccine cost not covered by their insurance. The SHC was concerned that the increased demand for vaccine put them at risk for problems being reimbursed for vaccine costs and collecting administration fees. The second modification consisted of a letter and follow-up calls to the 79 parents who had consented to have their sons receive all recommended vaccinations (using the “check all” box), in order to confirm that parents knew this global consent included HPV vaccine. These communications informed parents that the CDC recommends that boys receive HPV vaccine. For both modifications, SHC staff sought confirmation of the parent’s original consent through parent signature or verbal authorization.

Parents’ Perceptions of Intervention Materials

Nearly three quarters of the 47 parents interviewed remembered the mailed packet somewhat (51%) or very (23%) well, though less than half (43%) recalled receiving an automated phone call about the mailing. Almost all parents told interviewers they read the packet materials somewhat (43%) or very (49%) closely.

Parents generally found the intervention materials to be informative and persuasive. Most parents (70%) felt the materials increased their interest in vaccinating their children, though others noted the materials had no effect (28%), and one parent felt the materials lowered her vaccination interests (2%). Parents indicated that the intervention materials provided important details about which vaccines are recommended for adolescents, and several parents said they had previously been unaware of the HPV vaccination recommendation for boys. Parents also stated that the materials served as a reminder to determine whether their children were up to date on the vaccines and informed them that vaccines were available through the SHC.

Although most parents we interviewed found the materials clear and comprehensive, some parents recommended additions. Consistent with questions staff received, several parents indicated that vaccination materials could have better explained out-of-pocket costs, and how these compared with the amount they would pay for the same vaccines administered elsewhere. In addition, several parents wanted more information about the vaccination process at the SHC, including how staff would determine which vaccinations their children needed, when staff would give vaccines, and details about other SHC medical services.

Implementation Resources

Distributing the intervention materials, organizing and conducting automated and targeted telephone calls, facilitating communications associated with intervention modifications, fielding parent questions, and additional logistics consumed an average of 30 person-hours per SHC (range across SHCs: 18-52 person-hours). About half of this time (14 hours) was at the start of the intervention, as staff addressed and mailed packets. In addition, CCFNC research team members spent 78 hours assembling around 3,200 packets that we delivered to the SHC staff. Altogether, CCFNC and SHC staff spent an average of 4 minutes per packet. Presuming an hourly wage of $15 for packet assembly and $30 for other tasks, we estimate a total cost of $3.92 per prepared packet, comprising $2.45 for program materials (color printing, initial and return postage, and participation incentives) and $1.47 for labor.

Vaccination Consent

SHC staff mailed packets to 2,975 parents of students enrolled to receive care in their clinics (Table 1). Of these, 433 packets (15%) came back as undeliverable by mail, which SHC staff then sent home with students. Table 1 describes the final consent statuses, based on number of consent forms staff initially distributed and accounting for the consent confirmations the two intervention modifications required. About 10% (n = 311) of the consent forms came back with parent consent for at least one vaccine, and 84% (n = 2,495) were not returned. The remaining 6% (n = 169) of forms either came back without a valid consent (e.g., unsigned, no vaccines identified) or came with a consent that a parent later revoked. Of the parents who consented to vaccination, most (79%, 246/311) checked the box to consent for “all recommended” vaccines for teens, rather than indicating individual vaccines.

OPEN IN VIEWER

Table 1.

Vaccination Consent Form Return and Consent.

School	Packets distributed (total)	Form unreturned	% of total	Form returned unsigned/without valid consent a	% of total	Form returned with consent for all vaccines	% of total	Form returned with consent for specific vaccines	% of total
1	660	541	82	46	7	51	8	22	3
2	739	641	87	25	3	59	8	14	2
3	840	704	84	51	6	66	8	19	2
4	736	609	83	47	6	70	10	10	1
Total	2,975	2,495	84	169	6	246	8	65	2

a

Includes forms that came back unsigned, or came back without any indicated vaccines, or for which parents revoked vaccination consent following intervention modifications.

Each modification to the intervention resulted in the loss of initial consents, primarily because of lack of direct communication with targeted parents, rather than active refusal (Table 2). Of the 162 parents of privately insured children, 23% confirmed their consent after receiving the modification, 15% actively withdrew their consent, and 62% did not respond to the cost letter and were not reached by phone. Of the 76 parents of boys, 30% confirmed consent for HPV vaccine, 11% actively withdrew consent, and 59% did not respond to the letter and were not reached by phone. Success contacting parents to confirm or revoke consent varied across sites, and only one reached the majority of targeted parents. SHC staff noted that time presented the biggest barrier to contacting parents.

OPEN IN VIEWER

Table 2.

Intervention Modifications and Loss of Consent.

				Consent lost
		Consent confirmed		Active refusal		Parent not reached
	Consent form confirmation required a	n	%	n	%	n	%
Clarify vaccine costs (private insurance only)	163	37	23	24	15	102	62
HPV vaccine recommended for boys	76	23	30	8	11	45	59

Note. HPV = human papillomavirus.

a

For cost clarification, the modification required confirmation of consent if parents had previously consented to any vaccination and were privately insured. For male HPV vaccination, the modification required consent confirmation if parents had originally consented to all necessary vaccines for a male student eligible for HPV vaccine.

Information About Vaccination Cost and HPV for Boys

We identified two topics that likely require more explanation in vaccination program materials: vaccination cost and HPV vaccine for boys. Over one third of parents whom we successfully recontacted about vaccination cost, and one quarter of parents whom we successfully recontacted about male HPV vaccination, actively withdrew their consent, suggesting that parents needed more details about each to provide fully informed consent.

In contexts where vaccination costs depend on insurance status, providing cost information is important to both parents and SHCs. Previous research identifies cost as a key barrier for parent consent (Brewer et al., 2011; Brewer & Fazekas, 2007; Zimet et al., 2000), and difficulty navigating financial and insurance structures likely prevents some SHCs from offering comprehensive vaccination programs (Daley et al., 2009; Lindley, Shen, Orenstein, Rodewald, & Birkhead, 2009). According to parent interviews and staff feedback, parents want to understand the amount they will be required to pay out of pocket. However, communicating accurate cost information is difficult, as these amounts differ by vaccine and health insurance plan, with each insurance company offering hundreds of plans with different benefits. Even if materials listed the full price and covered amount of different vaccines for the most prevalent health insurance plans, parents’ actual costs also depend on deductible levels and use. Unless recommendations to standardize vaccination costs and to expand the Vaccines for Children program (VFC) to cover administration fees for all eligible children and adolescents are implemented (Kaplan, 2010; National Vaccine Advisory Committee, 2009), SHCs will need to instruct parents to contact their insurance providers to determine out-of-pocket costs. Out-of-pocket costs should be less of an issue under the Patient Protection and Affordable Care Act (2010), as it requires private insurance policies to provide first-dollar coverage for vaccines.

SHC vaccination programs also need to disseminate clear information about HPV vaccine recommendations for boys. The Advisory Committee on Immunization Practices (ACIP) did not universally recommend HPV vaccine for boys until October 2011 (CDC, 2011). The vaccine materials we distributed in September 2011 used CDC language that described boys’ HPV vaccination as an option rather than a recommendation. Parents may have had difficulty determining whether a global consent meant that boys would receive HPV vaccine, and indeed staff fielded several calls from concerned parents. Previous studies found that only 20% of parents were aware that guidelines recommended the vaccine for boys (Reiter, McRee, Kadis, & Brewer, 2011), and a review of literature found widely varying rates of acceptability for HPV vaccine among mothers of sons (Liddon, Hood, Wynn, & Markowitz, 2010). Thus, even under the new ACIP guidelines, parents may need more education about this topic.

Partnerships Between SHCs and Researchers

Our process evaluation also provides insight on the utility of partnerships to enhance SHC vaccination processes. SHC staff and the CCFNC research team shared the ultimate goal of improving the health of adolescents through increased vaccination. Secondary priorities, however, likely differed between the two groups (Cooper-Robbins et al., 2011). The research team wanted to test intervention materials and processes, in part to determine the feasibility and value of disseminating the intervention to other SHCs. SHC staff needed to maintain the trust they had established with adolescents and parents, in order to ensure their ability to continue providing quality care. While these priorities usually aligned, they occasionally diverged, especially when intervention modifications in response to parent concerns compromised intervention fidelity. Modifications that required confirmation of previous parental consent taxed staff resources and resulted in significant loss of consent for vaccination.

In some cases, clear communication between partners about their priorities might avert implementation challenges. For example, during the materials development stage of our project, SHC staff had identified costs as a potential parental concern, and suggested including the list price of each vaccine in the mailed materials. The research team worried such information would be misleading, since list prices rarely reflect out-of-pocket costs, and opted to include a generic phrase about responsibility for noncovered costs. When parents contacted the SHC staff with questions about vaccination cost, however, the SHC staff felt they could not proceed with vaccination without providing the list price and confirming consent, fearing damage to their established relationship with parents.

Fortunately, when we needed to make modifications, SHC staff and the CCFNC research team worked together to quickly draft and distribute information. Our ability to do so was facilitated by the collaboration we had established during intervention planning. In retrospect, however, our planning process would have benefited from more communication. In an attempt to minimize SHC staff burden, the research team tried to limit the requests we placed on staff. Had we spent more time working with SHC staff in the planning stages to identify mutually satisfactory cost language, we may have avoided one intervention modification.

Of course, even the most thorough planning efforts will not prevent all intervention modifications. External events, such as new vaccination recommendations, can affect program processes or parent beliefs. Future SHC staff and research teams may want to establish a protocol to accommodate potential intervention modifications to minimize response time and program impact. Budgeting time to sufficiently implement unanticipated modifications will also be important.

Study Limitations

We did not track differences in adherence to implementation guidelines across clinics. As a result, we could not assess whether such differences existed and affected study outcomes. However, it is likely that SHCs will adapt any intervention to fit their workflow and staff capacity, and thus interventions must allow for some flexibility. The percentage of unreturned forms and overall consent rates were similar across the four sites, indicating that implementation variation did not result in substantial differences in participation.

More than one third of parents who returned consent forms did not sign them, identify which vaccinations their child should receive, or confirm consent following subsequent communications. In addition, most parents did not return the consent form. Prior vaccination is an important reason parents do not consent to school-located vaccination (Cooper-Robbins et al., 2011). This may be particularly relevant for the high school students in our study, as most of the vaccines on the form were recommended for younger adolescents. If parents of children who were or would be vaccinated elsewhere comprise the bulk of nonrespondents and refusals, our intervention may still have succeeded in reaching those parents who could most benefit from a school-located vaccination provider. Our methods did not allow us to determine the motivations of parents who did not return vaccination forms, primarily because patient privacy protections prevented us from contacting parents who did not return forms. In addition, our form allowed parents to mark consent for all recommended vaccines, or for specific vaccines, but did not include a third option that would allow parents to actively refuse consent. We were therefore unable to distinguish parent intent from parent error when blank forms were returned.

Implications for Practice

SHCs have emerged as a potentially important avenue for increasing vaccination rates among adolescents. Our process evaluation of a low-resource, multivaccine intervention illustrates the need for materials that contain information that is timely (e.g., HPV vaccination recommendations for boys) and relevant (e.g., out-of-pocket cost) in order to maximize consent. In addition, our results suggest two strategies for streamlining consent. First, providing a checkbox for parents to indicate global consent for all vaccinations for which their teen is eligible is an option that proved popular among parents and may facilitate complete vaccination of teens. Second, a refusal check box, or a separate refusal form, as used in other programs (Boyer-Chuanroong, Woodruff, Unti, & Sumida, 1997) would allow staff to follow-up with parents who did not respond or who returned incomplete forms. Consent forms that let parents indicate reasons for refusal may inform future intervention efforts. In particular, if parents can indicate previous vaccination of their child, staff can target their outreach efforts efficiently. Finally, close collaboration among SHC staff and others involved in vaccination programs is critical for anticipating barriers and ensuring flexibility in implementation.