Preparedness for a Natural Disaster: How Coriell Planned for Hurricane Sandy

Introduction

The need for biobanks (repositories, biorepositories) to prepare for disasters, both natural and manmade, has been discussed. ¹ During a disaster, biobanks have a fundamental objective to protect every single specimen and whole collections under its management including related electronic data, ² and a responsibility to protect the safety of employees from hazards at the organization. Often referred to as “disaster planning” or “preparedness,” methods encompass all efforts to ensure employee safety and reduce negative impacts such as loss of business operations and property damage. ³

A key component for successful preparedness is a commitment to planning from an organization's management team and financial support for the preparedness program. ⁴ Initially, an organization's risks must be assessed to understand the potential and likelihood for loss after a disaster. ⁵ Once hazards are defined, strategies to prevent them or reduce risk must be developed. Implementation of a preparedness plan embodies documentation, management systems, and staff training. Specifically, a plan should include resource management to identify resources needed for disaster response; an emergency response plan to protect people and property that identifies evacuation plans and other disaster responses; a crisis communications plan to communicate with employees and the public; a business continuity plan to reduce loss of business operations; an information technology plan to protect electronic data and provide connectivity; employee assistance to support staff through a disaster; incident management to document responsibilities and help manage disaster planning, response, and recovery; and finally, training to prepare staff for disaster preparedness. ⁵ A preparedness plan should be evaluated regularly to ensure its effectiveness, which may include reviewing documented protocols and testing the understanding of those plans by all staff. Evaluation allows for program improvement should weaknesses in the preparedness be exposed. Furthermore, critique after a disaster can also provide opportunities for plan improvement.

Planning requires a team and commitment to organize, develop, and administer the preparedness plan. Unfortunately, many businesses do not establish robust disaster preparedness plans. ⁶ It has been estimated that nearly 40% of businesses that are affected by disaster do not reopen. ⁷ On the other hand, companies that participate in at least some preparedness planning are more successful in the recovery and the return to normal business operations. ⁸

Although preparedness planning is involved, it is critical to the research community that biobanks prepare for the unexpected to the best of their ability. It is not simply staff and facilities that require protection but the continuation of scientific advancement and promise of tomorrow's discoveries that necessitate an organization's commitment to preparedness.

Coriell's Biobank

Coriell Institute for Medical Research, founded in 1953 and based in Camden, New Jersey, is an independent nonprofit research center dedicated to the study of the human genome. Expert staff and pioneering programs in the fields of personalized medicine, induced pluripotent stem cells, genotyping, cytogenetics, and biobanking drive Coriell's mission to improve human health through scientific research.

Essential to the Institute's support of international scientific research is the Coriell Biobank, which manages and distributes the world's most diverse collection of cell lines, DNA, and other biological resources. Storage at the Coriell Biobank includes nearly 110 liquid nitrogen-filled cryogenic stainless steel tanks, more than 80 ultra-low −80°C freezers, a dozen −20°C freezers and two −20°C walk-ins, upwards of 20 refrigerators (4°C) in addition to eight walk-in refrigerators, and expansive ambient storage areas. Additionally, the Biobank utilizes 70 cell culture incubators and two walk-in incubators.

Since the first repository—a National Institutes of Health collection—was established at Coriell in 1964, hundreds of thousands of cell lines and DNA samples have been distributed to researchers in 65 countries; more than 7000 peer-reviewed articles have been published citing over 12,000 biospecimens from the Coriell Biobank. The Coriell Biobank provided support to the Human Genome Project, a worldwide program to map the entire human genome, and to the International HapMap Project, a project providing an efficient tool to identify disease-causing genes. For more than 60 years, Coriell has set the standard in biobanking services, including the experimental design, collection, processing, distribution, cryogenic preservation, and information management of human biomaterials used in research. ⁹ By developing and maintaining biorepositories as national and international resources for the study of human diseases, aging, and neurological disease, Coriell is committed to providing the scientific community with high quality, well-characterized cell cultures and DNA preparations that are annotated with rich phenotypic data.

Preparedness is Standard

Procedures for disaster preparedness and emergency operations have been documented at Coriell since its inception in 1953; they were formally incorporated into the Institute's ISO 9001 Quality Management program in 2003, and into a written Emergency Operations Plan for Disaster Preparedness and Business Continuity, based on Homeland Security guidelines, ¹⁰ in 2005.

The basic elements in the Institute's disaster preparedness plan comprise routine inspection and documentation of laboratory equipment and building systems essential to operations. Laboratory equipment includes assay instruments, robots, incubators, and freezers. Building systems include water supply, water purification, steam, electrical service, mechanical systems, and alarms. The precise locations of data and valuable materials are documented, as are all electrical outlets providing routine power and those providing back-up electricity supplied by Coriell's on-site generators. The plan also describes Information Technology protection measures to prevent interruption to Coriell's online presence, an effort critical to maintaining the Biobank's interactions with customers, in addition to employee training and communication procedures.

In more recent years, documentation has been—and continues to be—extended to include process-specific instruction for Information Systems, and within Cell Culture, Stem Cell Biology, Cryogenics, and Molecular Biology groups at Coriell. For example, in Molecular Biology, documentation focuses on potential transfer of working-stock biomaterials and reagents from small freezers to comparable cryogenic storage equipment or to units on back-up electrical outlets, managing draft documents in paper form or saved on local laboratory computers but not yet in archived files, and the chain of command specific to decision-making and workflow processes, such as who is responsible for powering down large equipment.

Procedures detailed within Information Systems now include extensive data back-up and recovery procedures. Specifically, all critical servers at Coriell have been virtualized using datastores located on the IBM XIV Storage Array Network (SAN), which affords several key benefits. Primarily, virtual machines allow for efficient back-up and re-instantiation. The implications are such that if a server were to be corrupted, it can quickly and easily be re-created to its last back-up. Further, because virtualized systems are no longer specifically tied to any hardware implementation, new servers can rapidly be introduced to the environment and immediately deployed in a virtualized solution. Finally, the virtual machine back-up tapes themselves are collected weekly from Coriell by a contractor, Docusafe, which houses off-site storage facilities for secure, redundant back-up. The off-site storage facilities are in geographically disparate areas (i.e., greater than 50 physical miles from the Institute), and thus offer reasonable protection against localized natural disasters.

Testing of the Institute's five back-up generators (four natural gas and one diesel fuel) is documented and occurs regularly through Coriell's Core Services department. Specifically, Coriell conducts “full load tests” no less than every 18 months, running the generators for 8 hours under the supervision of an electrical contractor. Meanwhile, a crew of electricians services all switch gear equipment, assuring efficient operation of Coriell's mechanical systems. This preventive maintenance program coupled with generator testing provides needed confidence that Coriell's electrical systems will work during routine business operations, and that the generators will function in the event of an emergency.

Additionally, inclusive of Coriell's approximately 42,000 liters of liquid nitrogen reserves, which is enough to keep the nearly 110 cryogenic steel tanks adequately filled for many weeks, Coriell has also prepared for major disasters by storing failsafe samples at different geographic locations, some nearby and others across the country.

Coriell's plan defines procedures for on-call personnel to respond to triggered alarms and makes available to them, the contact information of laboratory managers. The plan also includes an Institute-wide communications solution, Honeywell Instant Alert^®, which is a web-based emergency communications service allowing customized messages to be sent to multiple recipients through multiple communication devices including phone, e-mail, and pager. First proven successful in protecting more than 1,000,000 students in 2,200 school systems, the service is now used in various business settings, and allows Coriell's management to rapidly and effectively communicate with all staff. As expected, the plan also details general procedures, guiding employees in the event of emergencies such as fire, flood, and/or personal injury.

Hurricane Sandy

During a 7-day period in October 2012, “Sandy” graduated from a tropical wave to a tropical storm, and then to a Category 3 hurricane at its peak intensity. After passing over Cuba and immediately before it struck the East Coast of the United States, it weakened to a Category 1 hurricane. Landfall occurred at Brigantine, NJ, just northeast of Atlantic City and approximately 40 miles from Coriell Institute. Damage along the coast of New Jersey was significant, caused mostly by high winds and flooding. Although inland and more protected, Coriell was among thousands of affected businesses, private residences, hospitals, and state, county, and municipal government offices situated north of the eye of the storm. Considered to be the deadliest and most destructive storm in 2012, the hurricane's footprint was enormous, affecting nearly 24 U.S. states including the entire East Coast, and causing an estimated $50 billion in damage. ¹¹

Coriell's Preparation

Hurricane Sandy had been forecasted by local meteorologists well before landfall, allowing for additional pre-disaster preparations at Coriell. Beyond securing extra supplies to maintain laboratory operations, reserve fuel for the diesel-powered generator was purchased and the four natural gas generators were confirmed to have been recently tested and were operating per performance specifications. Coriell Information Technology personnel worked alongside Core Services to test the emergency generator solely responsible for powering the Coriell Data Center. The generator was confirmed to be in a ready-state and able to provide adequate power to the computing environment for the projected duration of the disaster. This preparation was completed 2 full business days before the storm was in critically close proximity to Coriell to ensure that any last minute adjustments could be made to the computing environment. Coriell Core Services staff also verified the Institute's Instant Alert communication system, designed to notify all Coriell personnel of Institute-related closures and emergency events efficiently.

On Friday, October 26, laboratory staff prepared for the storm which was expected to severely impact the region on Monday. In addition, Information Technology personnel surveyed the Data Center and virtual hosts—confirming that the scheduled back-ups were completed on time and with integrity. The back-ups were then given to Docusafe for off-site storage and potential recovery.

The Molecular Biology Laboratory focused on safely storing samples and large working stocks of reagents, and on attending to major equipment. All high-throughput robotics were monitored for normal start-up operations before being disconnected for electrical surge protection. Alarm performance was tested on all equipment, and major freezers were confirmed to be on back-up electrical outlets. By the end of the day, all computers, robots, liquid handlers, and automated equipment were shutdown and disconnected to protect against surges, and lab-specific phone trees, in addition to the Institute-wide communication system, were utilized among technical and scientific staff for communications.

The Cell Culture Laboratories took similar precautions concerning communications, reagent management, and equipment monitoring. Coriell's incubators being of the highest importance were confirmed to be powered by back-up electrical outlets and guarded by active alarms. The primary and failsafe carbon dioxide delivery systems were validated to be operating at specifications. Proactively forecasting 3 days forward, records of cell lines including specific culturing timelines for medium changes and checking for confluence were reviewed for critical time points, particularly cells scheduled for shipment on Monday following the storm.

Finally, the Stem Cell Laboratory, which had begun its planning nearly a week prior to hurricane landfall, prepared for a worst-case scenario in which the laboratory would be without power for at least 3 days. In addition to discussing the needs of active cell lines, the pre-designated emergency-shift technician and laboratory supervisor reviewed the emergency preparedness plan, ensured all equipment was on back-up electrical outlets, and confirmed proper reagent storage. Handling undifferentiated induced pluripotent stem cell (iPSC) lines requires daily care and manipulation, and with hurricane landfall anticipated for Monday evening, the laboratory manager changed the typical weekend cell maintenance schedule by adding an additional weekend shift to accommodate for the potential of restricted travel on Monday and Tuesday.

Coriell's Response

On Sunday evening, October 28, as the storm approached the coast and its landfall location became more pronounced, Coriell's management discussed the possible closure of the Institute, recognizing that employees could be in danger should they try to report to work. Furthermore, a State of Emergency had been declared by the Governor of New Jersey the previous day, essentially eliminating public transit and restricting automobile traffic. ¹² Recognizing the potential danger and inconveniences to employees, the decision was made to close the Institute; employees were notified of the closure through the Instant Alert system, which sent messages through redundant means including email, text, mobile phone, and land line calls.

While the Institute escaped flooding, and the structure handled winds ranging 40 to 60 miles per hour, Coriell did not escape the loss of electricity affecting more than 1.2 million homes and businesses in the state. ⁷ When the electrical supply to Coriell was interrupted, key alarms triggered, the generators started, and on-call personnel were notified, all as expected.

The dedicated generator adequately provided electricity to Coriell's Data Center, maintaining Coriell's presence on the Internet and permitting email communication among staff. Generators also supported all mission critical −80°C freezers, reagent stock refrigerators, cell culture incubators, equipment alarms, and the fire alarm and building security systems. With respect to on-call emergency personnel, the formal State of Emergency demanded a slightly altered response approach, which was to limit first responders. Key personnel had been identified well in advance of the storm and only those personnel reported to the Institute. The two emergency responders, the director of Core Services and a laboratory supervisor, reported to Coriell; the former provided supervision and constant contact with managers and senior leadership from Monday through Wednesday (October 29–31) while the Institute was closed.

For approximately 55 hours, Coriell operations were solely reliant on pre-planned automated systems to produce temporary power. Upon surveying the laboratories, there were no signs of power disruption to the back-up electrical outlets, meaning all incubators, freezers, etc., were functioning normally. Within the Molecular Biology Laboratory, there were two small, working-stock freezers which were knowingly not connected to back-up outlets. Learning of the lengthy power outage forecast, Core Services used extension cords to power the small freezers for the duration, and well before they lost temperature.

Power to Coriell Institute was restored on Wednesday morning but the Institute remained closed for the day. Despite the formal closure, a number of employees safely travelled to work, including senior management, managers, and technicians within the laboratories, specimen shipping, and customer service groups. Many specimens were ready for distribution, and Federal Express was available to handle shipments, so staff prepared packages for shipping (excepting for shipments to the East Coast). The Institute resumed normal operations on Thursday, November 1, 2012.

Lessons Learned

The success of Coriell's preparedness was dependent on effective long- and short-term planning and communication. The combined preparation both “years in advance” and “immediately prior to the event” were both essential to achieve the positive outcome. Regarding long-term planning, in the days leading up to the storm it would have been nearly impossible to install generators in a timely fashion to withstand the power failures experienced throughout the State of New Jersey. Likewise, it would not have been feasible to build the power redundancy found throughout the facility. The immediate planning associated with laboratory specimen processing was also essential as it identified the required emergency responders, and planned forward into the next week to minimize disruptions to Coriell's business operations. Additionally, the Instant Alert system, which was launched in 2011, allowed management to keep employees easily informed of the emergency closure-status. Other important lessons learned in previous years, which have proven significant since their implementation, include the need for the building's sewage ejection system and secure card-access entryway system be powered on lines with emergency back-up.

After years of planning, thinking through emergency scenarios, and investing considerable resources to mitigate risk, the Institute's response to Sandy was commendable. Nonetheless, Coriell critiqued its response to the storm to identify areas for improvement, spanning operational and physical preparedness adjustments. Operationally, there was discussion of housing emergency response personnel in the building for constant monitoring. Also discussed was sending periodic reminders to staff about storing all important documentation on Coriell's electronic file-sharing portal (data backed-up nightly by Information Systems). Finally, it was suggested to place the few small working-stock freezers not connected to back-up outlets onto emergency power lines ahead of the storm. During Hurricane Sandy, the emergency response personnel ran extension cords to those units which were slightly inconvenient considering elevators were offline in Coriell's 5-story building plus basement. Overall, it was determined that for this specific disaster scenario, Coriell's response was appropriate and successful.

It was recognized that the outcome could have been different had the power outage lasted weeks, rather than days. A resulting fuel shortage in the area could have led to the loss of back-up power from the diesel fuel generator, which is primarily used for powering Coriell's emergency lighting system. Had natural gas been unavailable, those critical emergency generators would have been unavailable. Moreover, a disaster occurring during the summer months would have added challenges such as keeping the building air-conditioned to prevent equipment from overheating. A flooding disaster, although extremely rare considering Coriell's floodplain, would present different issues but would not greatly impact Coriell's information technology systems, biobank, or research labs as they are not located on lower levels of the building. Conversely, a fire within the Institute could be disastrous, with the extent of damage variable based on whether the fire was within an office, lab, or the Data Center, as well as the water damage caused by the sprinkler system and/or fire department response. Nonetheless, Coriell believes its best approach to preparedness remains scenario review, discussion, and planning. Such continued review of the Information Systems platform, for example, not only allows a faster recovery for normal production operations, but would also minimize the time interval between a “last good back-up” and the real-time usage of the system until the disaster occurred.

Coriell's preparedness was vital for protecting the entrusted biological specimens managed through Coriell's Biobank, which have allowed both Coriell scientists and research investigators around the world to advance research programs in genetics and cell biology. The Institute's laboratory activities were curtailed by Hurricane Sandy, but its fundamental biobanking responsibility to effectively manage specimens was unaffected.