Abstract
In 2009, the New York City Department of Health and Mental Hygiene operated 58 points of dispensing, or PODs, over 5 weekends to provide New Yorkers with vaccinations against influenza A (H1N1). The department evaluated the success of the operations, including areas that needed improvement for future mass prophylaxis.
In fall 2009, the New York City Department of Health and Mental Hygiene (DOHMH) operated 58 points of dispensing (PODs) over 5 weekends to provide influenza A (H1N1) 2009 monovalent vaccination to New Yorkers. Up to 7 sites were opened each day across the 5 boroughs, with almost 50,000 New Yorkers being vaccinated. The policies and protocols used were based on those developed for New York City's POD Plan, the cornerstone of the city's mass prophylaxis planning. Before the H1N1 experience, NYC had not opened more than 5 PODs simultaneously and had only experienced the higher patient volume seen with the H1N1 PODs on 1 prior occasion. Therefore, DOHMH identified factors that contributed to the success of POD operations, as well as areas for improvement to inform future mass prophylaxis planning and response. Though this was a relatively small-scale, preplanned operation, during which a maximum of 7 PODs were operated on a given day, the findings have implications for larger-scale mass prophylaxis planning for emergencies.
To reduce community spread and prevent school closures, the NYC DOHMH began planning a multifaceted mass vaccination program for New Yorkers in summer 2009. Plans were developed to distribute vaccine through private providers, hospitals, community health centers, DOHMH clinics, pharmacies, schools, and community-based points of dispensing, or PODs. This article reviews the successes and lessons learned from the H1N1 vaccination PODs to inform future mass prophylaxis planning and response efforts.
Background
New York City receives funding through the Public Health Emergency Preparedness cooperative agreement with the Centers for Disease Control and Prevention (CDC) to develop plans for public health emergencies. Additional funds are provided under the Cities Readiness Initiative (CRI) to select jurisdictions to encourage the development of plans to provide prophylactic medication to their entire population within 48 hours of the decision to do so. 3 To satisfy its CRI mandate, NYC created the NYC POD Plan, which describes the city's response to a widespread threat agent release or a naturally occurring outbreak of disease. Over time, the city has worked to develop the capacity to open up to 200 neighborhood-based POD sites across the city on the same day to dispense medication. Each POD requires a staff of approximately 100 per shift, 85% of whom are nonmedical personnel. Six pretrained core team members comprise the primary leadership for each POD site; they are responsible for POD set-up, as well as training and assignment of all other (general) POD staff. All POD teams report to a centralized command and control (C2) structure based at DOHMH.
DOHMH built its plan to operate H1N1 PODs on the policies and protocols outlined in the NYC POD Plan. DOHMH has previously tested its POD plan through several full-scale exercises and has used PODs to provide anthrax prophylaxis in 2001, to provide annual flu vaccinations to NYC employees and the community since 2004, in response to a hepatitis A exposure in 2008, and to contain a mumps outbreak among the orthodox Jewish community in 2010. However, DOHMH had never operated 7 PODs simultaneously before the H1N1 PODs in fall 2009. Therefore, this real-world extended POD operation was used to test response plans for future large-scale disease outbreaks to determine those factors that contributed to the success of the PODs, as well as to identify areas for improvement.
H1N1 PODs Program Overview
As part of a large vaccination effort, DOHMH decided to conduct a school-located vaccination program that provided H1N1 vaccination in public and participating nonpublic elementary schools. 4 PODs were initially intended to supplement the school-located vaccination program by providing H1N1 vaccine to middle school and high school students, as well as elementary school children who had missed the vaccination day at their school. DOHMH operated at least one POD in each of the 5 NYC boroughs on Saturdays and Sundays for 5 weekends in November and December. As the demand for vaccine was initially unknown, DOHMH operated an additional POD in the more densely populated boroughs of Queens and Brooklyn for the first 2 weekends, for a total of 58 PODs over 10 weekend days. PODs were open for 10 hours on Saturdays and 9 hours on Sundays. Approximately 2,800 staff from 18 different NYC agencies volunteered to staff the PODs, along with more than 200 Medical Reserve Corps (MRC) volunteers. Together with planning staff, they contributed more than 65,000 hours of labor to this operation. 5 DOHMH used a private warehouse vendor to manage supply logistics. Vaccination eligibility criteria were restricted to students for the first weekend and were broadened over time to include eligible individuals from the general population due to lower-than-expected demand. Almost 50,000 people were vaccinated, increasing overall vaccine coverage in the city and decreasing the potential for morbidity and mortality from H1N1 flu (see Table 1).
2009 New York City H1N1 POD Campaign Summary
Evaluation Methods
To capture key lessons learned, DOHMH used a number of methods to collect and analyze data. The department developed and conducted a series of staff debriefings, online surveys, and hotwashes (ie, post-response debriefing sessions in which key players were invited to express their thoughts about what went well and what needed improvement) to collect information. Included in these sessions were staff who planned POD operations, as well as those who worked in PODs or C2 during the H1N1 POD campaign. Exit surveys were conducted for attendees at PODs at select sites.
Weekly POD Team Leader Debriefings
Weekly semistructured POD team leader debriefings were held on Mondays following each POD weekend. The goal of these sessions was to identify key operational problems and proposed improvements that could be implemented rapidly. Participation was voluntary but strongly encouraged, although it was not formally tracked. DOHMH classified responses from POD team leaders into macro-level categories for further analysis, including communication, staffing, external agency interaction, logistics, materials, public information, reporting/data collection, and training/staff resources.
Online Surveys
At the conclusion of the H1N1 POD campaign, DOHMH e-mailed a link to online structured surveys to all staff who had worked in PODs and/or in command and control each weekend. Slight variations in the surveys were made to target specific roles, and individuals were asked to answer between 8 and 14 questions. Surveys also included an opportunity for respondents to provide written comments. The surveys were created to determine which materials provided staff with the most guidance or support during POD operations, to identify the most effective training methods and materials, and to capture general lessons learned from the H1N1 POD campaign. Quantitative responses were tabulated as frequency distributions. Qualitative data were reviewed to identify major themes. Response rates for general POD staff, core team leaders, C2 staff, and interagency liaison staff were 53%, 73%, 61%, and 29%, respectively.
Hotwashes
Under the direction of DOHMH, evaluation staff from the Yale New Haven Center for Emergency Preparedness and Response (YNH-CEPDR) facilitated 4 90-minute hotwashes in January 2010. Each hotwash targeted 1 of 4 different roles DOHMH and NYC employees filled during the response: POD core team leaders (participation rate = 29%); C2 staff (participation rate = 26%); DOHMH staff responsible for POD planning (participation rate = 53%); and leadership from outside agencies that contributed to POD planning and operations, including representatives from the New York Police Department (NYPD), the NYC Office of Emergency Management (OEM), the NYC Department of Education (DOE), and the Regional Emergency Medical Services Council of New York City (REMSCO) (participation rate = 53%). Responses from participants were compiled and reviewed to identify major themes.
POD Exit Surveys
DOHMH conducted exit surveys of POD clients during the last 2 weekends of operations. 6 Each survey consisted of 6 questions: 4 to collect demographic information and 2 to determine how the respondent learned about the clinic and found its location. Surveys were administered orally, with responses recorded by interviewers. Interviews were conducted on December 5-6 at the Manhattan, Bronx, and Brooklyn PODs and on December 12-13 at the Bronx and Manhattan PODs.
Lessons Learned
Planning and Coordination with Partner Agencies
Coordination with municipal labor unions
When municipal workers are used to staff PODs, planners must coordinate with labor relations staff and union administrators early in the planning process to clarify expectations for roles and responsibilities, worker health and safety, and compensation and work hours, to secure the support of the unions. Without union support, it might have been difficult to ensure adequate POD staffing. Weekly conference calls were held with senior-level staff from partner agencies, giving them the opportunity to obtain needed information for their respective employees from DOHMH. Such coordination led to the development of clear overtime policies, payment protocols, and enthusiastic participation by staff across the agencies involved.
POD site operations
To eliminate the need to conduct multiple duplicative visits to the same POD sites and to improve interagency coordination, visits to POD sites should include representatives from health, facility management, and site security partners. Although DOHMH had presurveyed all POD sites, DOE required additional site visits to be conducted prior to each POD weekend to include school leadership and security staff to ensure that security and facility preservation concerns were appropriately addressed.
During these visits, it was sometimes difficult to come to consensus on the route clients should take through the POD, as well as what areas could be used for administrative functions, while balancing the different mission objectives among health, education, and security partners. Therefore, planners needed to adjust the site plans created solely by DOHMH to accommodate the security and facility considerations initially missing from those plans.
Integrated command and control operations
A C2 structure that includes representation from key agencies can facilitate successful operations. Interagency liaisons from OEM, NYPD, DOE, the Fire Department of New York (FDNY), and REMSCO were positioned at C2 to provide logistical support, relay important data to their personnel in the field, and assist in providing emergency medical services support. Survey results indicated that 80% of interagency liaisons found the integration of external agencies into the C2 structure to be effective in managing the event.
Information sharing
Interagency information- and data-sharing expectations and responsibilities should be clarified at the start of an operation. Though several pre-event meetings were held between DOHMH and partner agencies to define when data would be reported and to whom at a given agency it would be given, there was still some confusion experienced in the field between POD team leaders and security staff regarding reporting protocols, leading to significant stress and frustration by both POD team leaders and staff from partner agencies.
Real-World Operations
Real-world operations are invaluable opportunities for training and engaging staff in emergency preparedness and response activities. Staff from across 18 NYC agencies brought a broad spectrum of skills and knowledge to the POD operation. The overwhelming majority of staff had a positive experience. Approximately 96% of general POD staff survey respondents, 89% of core team survey respondents, and 88% of C2 staff survey respondents indicated that they were “likely” or “definitely likely” to volunteer for future DOHMH PODs. Qualitative responses showed that many staff felt proud of their work in the H1N1 PODs, enjoyed working with and respected their colleagues in the PODs, and were appreciative of the positive feedback the public provided to them.
Clear Administrative Protocols
Clear administrative protocols must be communicated to and followed by POD leadership staff. There were several instances where the lack of clear administrative protocols, and/or POD staff's disregard for existing protocols, negatively affected POD operations.
Staff management
It is critical to provide clear protocols for staff assignment and dismissal within the POD and sufficient training to POD leadership to ensure that policies are correctly, fairly, and uniformly implemented across all PODs. When the provided protocols were not followed by POD leadership, the staff assignment process was consistently noted as being the most challenging part of the day, resulting in significant staff frustration and delayed POD openings. When low patient volume necessitated early release of staff, there were scattered reports of POD team leaders dismissing individuals in a manner that was perceived as not being transparent or fair, leading to some staff stating that they would not want to work in PODs again. Policies for staff dismissal should be developed and shared with general POD staff before their shifts.
Data collection and reporting
Clear and consistent data collection instructions, templates, and reporting requirements must be communicated to staff and should be changed only when absolutely necessary once operations begin. If data collection requirements are modified over time, those changes need to be communicated via staff training. During the NYC H1N1 POD campaign, data collection and reporting requirements changed weekly, primarily because of changing target groups for vaccination. This prevented POD and C2 staff from receiving updated templates and instructions with enough time to become comfortable with what would be required for that weekend's reports. The resulting confusion sometimes delayed receipt of needed information from the PODs. Instructions pertaining to reporting periods, data points, and scheduled reporting times to C2 were not clear to all staff responsible for collecting and reporting data, resulting in different types of data being reported from sites on the same day.
Advance Information for Staff
Training, materials, and other information should be made available to staff before their shift. Although 87% of POD staff survey respondents “agreed” or “strongly agreed” that they were provided the necessary training for their roles, many staff requested that more information on POD roles and responsibilities, what to expect when they arrived each morning, and where and when to report be sent to them with their confirmation e-mails and also made available online along with the training video. This was especially emphasized by staff who worked in sites where the just-in-time (JIT) training video was not shown or where incorrect portions of the video were shown (eg, when antibiotic dispensing portions were played for staff in the vaccination PODs).
Essential Information Only
Core team training
Just-in-time training and materials should contain only essential information. Many core team members reported that they felt there was too much information provided in a short time frame during the Friday afternoon refresher training to allow them to assimilate it, even though 82% indicated that they found the Friday afternoon trainings to be useful, and 86% believed that they were provided the necessary guidance and materials to perform their assigned roles.
Core teams found the start-up checklist, which outlined the immediate steps that they needed to take to open the POD each morning, to be the most useful resource provided (70%), followed by the POD organizational chart (52%), set-up job action sheets (52%), and personnel station documents, which provided instructions and a template for assigning staff to roles within the POD (47%).
Command and control training
Although most C2 staff found the materials distributed to them to be useful (85%) and believed that they received the necessary training to fulfill their duties (70%), some found the packet of reference materials they received each day they worked voluminous and not organized so that information could be quickly located when POD sites were calling for assistance. The items most frequently cited as being of greatest utility to C2 staff were those that provided situational awareness: the DEOC seating chart with titles and names for each position (57%), the DEOC phone list (50%), and the C2 organizational chart (41%). Though less frequently cited, C2 job action sheets and protocols, as well as the data collection template and core team contact lists, were also found to be very useful by 25% to 30% of C2 staff that completed the survey.
To better manage the volume of materials, a frequently asked questions (FAQ) document and table of contents should be created to ensure that C2 staff can locate the information they need as quickly as possible and that the answers to common questions are provided uniformly to POD sites.
Staff Pairing
Pairing more experienced staff with less experienced staff is an effective training technique. To build depth for core teams, less experienced senior-level DOHMH staff members were sometimes paired to “shadow” pre-trained core team members. Mentoring by a more experienced POD worker was cited by 41% of survey respondents as being the best way to learn their roles and responsibilities. Qualitative feedback supports this, with less experienced general POD staff stating that they found it helpful to be paired with more experienced staff to learn their role, because the larger-group training environment was often loud and sometimes did not allow for many questions. Shadowing was particularly valuable when staff were unable to view the training video.
Visual Aids
Visual aids improve accuracy of POD set-up. Survey and hotwash feedback indicates that core team leaders found set-up photos depicting how tables, chairs, signs, and individual POD station materials are to be organized in the POD were most helpful to them in efficiently and accurately constructing the POD. Such visual aids should always be provided as part of POD training materials.
Training Videos
A video is an effective and efficient just-in-time training tool. Most general POD staff members had never worked in a POD before, and they received just-in-time video-based training after receiving their position assignments each morning. The majority of staff (83%) found the training and materials provided to them to be useful. Of the training materials, the just-in-time training video was cited as being most useful by survey respondents (58%), versus just-in-time training delivered solely by the POD team leader (42%), training from review of job action sheets alone (42%), or mentoring by a more experienced POD worker (41%).
Summary of Lessons Learned for Future Mass Prophylaxis Operations
Ensure that planning and operations are coordinated with key partner agencies.
Real-world operations are invaluable opportunities for training and engaging staff in emergency preparedness and response activities.
Clear administrative protocols should be communicated to, and followed by, POD leadership staff.
Training, materials, and other information should be made available to staff in advance of their shift.
Just-in-time training and materials should contain only essential information.
Partnering more experienced staff with less experienced staff is an effective training technique.
Visual aids improve accuracy of POD set-up.
A video is an effective and efficient just-in-time training tool.
Incorporating POD “experts” or experienced consultants into staffing models ensures that operations run more smoothly.
Leveraging private warehouse capacity to manage logistics may be a critical strategy for successful POD operations.
Public outreach to publicize PODs must be broad.
Notably, staff members felt unprepared to perform their tasks when the just-in-time video was not shown, or when incorrect or unnecessary parts of the video were shown (eg, set-up modules were shown after the POD was already fully set) because of audiovisual problems. Equipment should be tested before each POD, and POD leadership staff should be provided with a training script to ensure that just-in-time training can always be provided.
POD Experts
Incorporating POD “experts” into staffing models ensures that operations run more smoothly. Qualitative survey results, hotwash feedback, and POD team leader debriefing discussions all indicated that it was invaluable to have POD “consultants” from DOHMH's Office of Emergency Preparedness and Response (the office responsible for POD planning and most familiar with the operation) assist POD core teams with set-up and POD management. Staff from DOHMH's Bureau of Immunization (BOI) who were pretrained in vaccine handling procedures to properly maintain the H1N1 vaccine in the PODs were also valued. This provided POD team leaders with much needed support in translating their training into success in the real-world setting and should be done whenever possible.
Private Warehouse
Leveraging private warehouse capacity to manage logistics is critical to successful POD operations. Use of a private warehouse vendor to manage vaccine and ancillary supply shipments from the CDC provided DOHMH with expert warehouse staff to implement supply kitting, staging, and distribution operations. This not only ensured expert management of vaccine and medical materiel, but also added depth to the limited pool of city logistical staff. Collaborating for the weekend PODs allowed DOHMH and warehouse staff to work closely with each other and exercise protocols and policies that would be expected to be used during emergencies.
Public Outreach
Public outreach to publicize PODs must be broad. There were 530 exit survey respondents: 33% white, 26% Hispanic, 22% black, 13% Asian, and 6% other or no response. A slight majority of respondents were female (53%). As per survey responses, notification through their child's school (18%) and the DOHMH website (16%) were most often cited as the primary source for learning about the city's PODs, followed by newspapers (13%) and television news (12%). Respondents used the Internet most frequently (31%) to locate the actual POD site they visited, followed by the city's 311 information line (13%) and their children's school (12%). 6 Hence, a variety of initiatives must be used to make POD information available to the public.
Discussion
Emergency preparedness requires the development of operational plans based on assumptions that may or may not prove to be true when an incident actually occurs. Regardless of how much training and exercising is done, it is difficult to replicate the conditions under which response plans will need to be enacted. Real-world use of emergency dispensing structures, like the New York City H1N1 influenza vaccination campaign in fall 2009, is the best way to test mass prophylaxis plans. For this reason, all such opportunities to apply and test assumptions of emergency models during planned mass prophylaxis events should be seized.
Evaluation of training sessions, exercises, and responses is a critical component in the development of effective emergency response plans, and resources must be dedicated to planning and implementing evaluation activities related to planned events. NYC's 2009 H1N1 POD operation yielded a variety of invaluable insights that might not have come to light without the experience that 58 PODs afforded to planners.
To ensure that all lessons were captured, analyzed, and could be applied to improve plans, DOHMH developed and implemented an extensive, multifaceted evaluation process that resulted in the creation of a comprehensive after-action report and improvement plan, in accordance with Homeland Security Exercise and Evaluation Program (HSEEP) guidelines. 7 The evaluation process captured quantitative and qualitative feedback from all levels of leadership and operational staff, and offered various opportunities (through weekly debriefings, hotwashes, and online surveys) for participation. The weekly debriefings allowed for critical policy and protocol modifications to be made throughout the course of the campaign. This provided for a well-rounded and (at times) real-time assessment of the operation based on a large number of data points, which NYC is now translating into experience-based updates to its POD plan and training curriculum. Specifically, NYC has revised all data collection and personnel assignment and management protocols and materials as a result of its H1N1 PODs experience, and we are modifying site assessment protocols to ensure that school and security partners are included on future site visits.
By partnering with other city agencies and the labor unions in the planning and execution of this POD vaccination campaign, DOHMH was able to build relationships with other mass prophylaxis stakeholders that may be leveraged for future planning and response efforts. With better understanding of each other's expected missions and operational goals during emergencies, partners may develop more realistic response plans. Preplanning based on accurate assumptions gathered from real-world experience allows partners to think through potential areas where missions and expectations might be disjointed. This could subsequently decrease or eliminate frustration and unnecessary or duplicative work during incidents, as happened with the site assessments and data reporting protocols during this event. Having staff from across city agencies and volunteer groups work together for this operation also provided an opportunity to build relationships and clarify staff expectations that will be critical during any future POD operations.
POD staff must be given the tools they need to successfully carry out their duties. Clearly communicated protocols and procedures are vital to managing expectations related to roles and responsibilities of POD staff at all levels. They must be developed in advance and tested as often as possible in exercises and training sessions before being applied in the real-world setting. This was particularly evident from the challenges experienced with the staff assignment and dismissal protocols, as well as with the audiovisual equipment for just-in-time training during this operation. Contingency plans to respond to unexpected plan deviations (such as an inability to show the training video) must be developed whenever possible. Adequately preparing POD leadership staff decreases their stress level and ensures greater management continuity across POD sites. This, in turn, increases the confidence of POD staff in POD leadership and decreases any perceptions of unfair staff treatment across sites. Ultimately, the result of providing POD staff with adequate training and clear protocols is an effective and efficient system for delivering a critical service and a positive experience to the public.
The H1N1 PODs provided NYC with an invaluable opportunity to train POD staff that would be very difficult to replicate under routine training conditions. More than 2,800 staff from 18 agencies volunteered to work the weekend H1N1 PODs, demonstrating their enthusiasm for supporting this real-world POD operation and learning about what they may be asked to do during future emergencies. Many NYC agency staff and volunteers who are expected to work in PODs have multiple demands on their time and cannot always take time away from their daily responsibilities to attend a full day of training. Approximately 20 full-day training sessions would have had to have been planned, executed, and funded to allow for this many staff to be trained, assuming they were able to be released from their duties for a day. And training sessions, while valuable, are conducted in an artificial setting without the benefit of the realistic pressures of working with actual patients to deliver prophylaxis within a required time frame under the scrutiny of media and city leadership. Ideally, real-world responses should be used to increase the pool of trained POD staff.
Potential POD staff who have positive experiences working voluntarily in PODs during nonemergency operations could be more willing and comfortable with reporting to work when instructed to do so during emergencies. They could gain a better understanding of what it means to work in a POD, feel prepared to perform their expected duties, and appreciate the value of their contribution. The use of “shadowing” to train staff may also contribute to feelings of engagement and usefulness among the more experienced staff who served as mentors and should be used as a training tool whenever possible.
The results of staff surveys from this POD operation reconfirmed prior studies that looked at health professionals' and public health employees' willingness to report to work. They demonstrated that staff are more likely to report if they know what is expected of them and feel educated, comfortable, and valued in their role, particularly through participation in drills and training sessions.8-12
One of the most successful aspects of this POD operation was use of “expert” POD staff to support the core teams. For larger-scale operations, however, this support would likely not be possible. In addition, using very experienced staff during scheduled operations may prevent planners from testing the instructions and materials created for the many inexperienced staff expected to work in PODs during an emergency. Planners should therefore consider if providing such support during smaller-scale events puts POD staff at a disadvantage for larger-scale events, where they would need to be able to operate alone, working only with the materials and protocols provided to them and without the additional expert support.
Lastly, anecdotal feedback, including e-mail sent to the city from the public, was very positive. People said that they were impressed that a government operation could run so smoothly and that staff could be so professional. Survey and hotwash feedback suggests that this made POD staff feel proud about their contribution to the campaign, thereby building morale amongst POD team members. Prior studies suggest that this experience, which served to educate the public on the city's emergency response model, could make citizens more likely to report to PODs during emergencies. 13
Limitations
All lessons learned are in the context of smaller-scale, preplanned POD operations, though many may be leveraged toward improving NYC's mass prophylaxis plans, which have been created to respond to large-scale, worst-case scenarios requiring all NYC residents to receive prophylaxis over a few days' time. This must be kept in mind when these lessons are reviewed, referenced, and considered for incorporation into emergency plans and protocols.
Much of the data collected are qualitative, including not only free text responses to survey questions, but also observational data collected each week by emergency planning staff stationed in the PODs or command and control. While such data can provide more complex understanding of situations and behaviors, there is an inherent potential for bias in the collection and analysis of the data.
Conclusions
Real-world operations are excellent opportunities for testing emergency plans while providing service to the public. Lessons learned must be accurately identified and consistently applied to improve plans. Positive experiences of potential POD staff during nonemergency responses can enhance their willingness to work during emergencies.
Footnotes
Acknowledgments
The authors acknowledge the extensive contributions of their many colleagues, not only from DOHMH, but also from other NYC agencies, who worked diligently to plan for and operate successfully the fall 2009 H1N1 PODs. From DOHMH we would especially like to acknowledge our colleagues from the Office of Emergency Preparedness and Response and the Bureaus of Immunization and Budget and Revenues. Partner agencies included: NYC Department of Education, NYC Police Department, Regional Emergency Medical Services Council, Fire Department of New York, NYC Human Resources Administration, NYC Administration for Children's Services, NYC Department of Design and Construction, NYC Department of Probation, NYC Department of Transportation, NYC Department of Citywide Administration Services, NYC Department of Environmental Protection, NYC Department for the Aging, NYC Department of Homeless Services, NYC Department of Information Technology and Telecommunications, NYC Department of Finance, NYC Department of Housing Preservation and Development, NYC Law Department, NYC Housing Authority, NYC Parks Department, and NYC Office of Emergency Management. We are grateful to all NYC employees, as well as NYC Medical Reserve Corps (MRC) volunteers, who staffed the PODs and command and control center every week.