Abstract
Introduction
Background and Significance
The IIWG was formed in 2009 by ISBER members to develop informatics best practices for biorepositories. The IIWG participated in the development of the ISBER Best Practices for Repositories: Collection, Storage, Retrieval, and Distribution of Biological Materials for Research (Best Practices), specifically with regard to issues of sample tracking, inventory management, and labeling. As an outgrowth of that initial focus, the IIWG has also begun to address the education and training needs of ISBER members in this area. In order to provide the most targeted and useful education and training materials, the IIWG sought to gather information about the current state of informatics in the ISBER community. The 2010 informatics survey was our first attempt to gather pertinent data about systems and to provide valuable information about the informatics priorities and education needs of the participants. Based on the results of the 2010 survey, we offered a workshop at the 2011 ISBER Annual Meeting & Exhibits entitled Evaluation and Design of Information Management Systems for Biorepositories. Based on feedback from the 2011 workshop, the 2012 survey was revised to collect more detailed information about the cost of implementing and supporting biospecimen management software.
There is an increasing demand in the ISBER community for more detailed information on software systems and the related costs of implementation and support. In the past year, there have been several email threads on the ISBER listserv on this topic. In addition, there has been at least one informal survey regarding the use of information management systems in biorepositories on LinkedIn® (started in late 2011). To our knowledge, there are no other formal market needs surveys of this type being conducted in the biorepository space.
Methods
Based on experience with implementing software systems for biorepositories, members of the IIWG collaboratively designed the contents of the survey. The ISBER Education and Training Committee reviewed and approved the survey, with minor modifications. The 2012 survey consisted of 53 questions covering the following topics:
• Basic descriptive information about the repository
• Importance of and satisfaction with various features of sample management software
• Cost, level of effort, and time for implementation and maintenance of current sample management software systems, subdivided into four categories - commercial, custom built by third party, open source, and developed in-house
In March 2012, the survey was distributed to the ISBER listserv, which has over 800 recipients. Seventy-two responses were received during the two-week survey period. The responses from the 2012 survey were compared, where applicable, to the responses from the 2010 survey to determine the stability of the results over time, and any significant recent changes. Although only 30% of respondents to the 2012 survey reported participating in the 2010 survey, the findings were remarkably similar. The results reported below are primarily from the 2012 survey.
Results
Basic descriptive information about the repository
• The respondents were primarily from organizations that included academic medical centers (45%), hospitals (32%), and research laboratories (26%). Pharmaceutical and biotechnology companies, government research centers, and outsource storage facilities were in the minority.
• Most of the repositories were storing specimens for future use (88%), supporting current intra-institutional studies for a particular department or laboratory (71%), and/or supporting inter-institutional collaborative projects (75%). Less common was providing previously banked samples to extramural investigators upon request (58%) or by contract to third parties (41%) although it was increased dramatically from 2010.
• Most of the respondents were banking human (92%) versus other animal (15%) or plant (4%) specimens, and the specimen types were blood (93%) or tissue (81%).
• The median size of the repositories in terms of number of banked biospecimens grew from a range of 10,000 to 50,000 in 2010, to a range of 50,000 to 100,000 in 2012.
• The median full-time equivalent (FTE) staff supporting these repositories was 2 to 5. For information technology (IT) support, 43% of repositories relied on departmental or organizational IT resources, 26% relied on third-party IT support from vendors, 22% had their own dedicated IT support, and 8% had no IT support whatsoever.
• For labeling technology, most respondents used printed 2D barcode labels (51%), text labels (47%), and linear barcode labels (40%). There was a drop in reported use of handwritten labels from 39% in 2010 to 24% in 2012.
• Most repositories used a combination of computerized database management systems (92%), electronic spreadsheets (34%), and paper (36%), with none relying solely on paper-based information management.
• Results were mixed for overall satisfaction with current repository information systems, with 57% reporting they were Somewhat or Very Satisfied, and 38% reporting Somewhat or Very Dissatisfied, with only 5% having no strong opinion.
Importance of and satisfaction with various features of sample management software systems
Although there has been much discussion about having the ability to search for and share data across networks and federations of sites, most respondents appeared much more concerned with sharing (83%) and searching (67%) for data within their own organizations. This compares to those who felt that the capability to share (60%) and search (29%) data internally was Important or Very Important. All respondents expressed dissatisfaction with the current performance of their systems on these features, rating them inadequate. The ability to share information with other departments within the organization was rated Very Important (highest possible rating) by 44% of respondents, while only 7% rated their current system performance in this area as Excellent (highest possible rating).
In our assessment of IT support and software customization features, an overwhelming majority of respondents reported that internal IT support for their repository system was Very Important (71%) or Important (22%). However, only 24% rated their current internal IT support as Excellent. Over 75% of respondents reported that the ability to customize their system was Important or Very Important, while only 11% rated their current system as Excellent in providing this ability. Less than half of the respondents reported their systems were meeting these requirements well.
In rating the importance of tracking specific types of data or information, a majority reported the following as Important or Very Important, in ranked order:
1. Information about the source of specimen (i.e. patient/clinical annotation)
2. Consent/authorization information
3. De-identification and control of identifiers
4. Biospecimen quality/metadata
5. Information from investigations/experiments on samples
6. Billing for repository and/or testing costs
7. Compatibility with controlled vocabulary [e.g. Biospecimen Reporting for Improved Study Quality (BRISQ) or common data elements (CDE) such as ICD-9]
8. Operational costs related to specimens
Of the most important data or information needs identified, only the tracking of billing and operational cost information were deemed inadequate for the majority of respondents. Detailed information about the source, quality, and location of the specimens was identified as an area for improvement.
A majority of respondents rated the following ranked list of features that are specific to sample management as Important or Very Important:
1. Storage location assignment
2. Tracking sample processing (sub-aliquot, extraction, etc)
3. Data edit checks (preventing entry of invalid data)
4. Barcode printing and scanning
5. Compliance with data security regulations (e.g. HIPAA, 21 CFR Part 11, etc)
6. Shipping and receiving
7. Configurability (adding fields, hiding fields, designing screens, etc)
8. Application Programming Interfaces (APIs) and systems integration
According to the majority of respondents, all of these features were at least adequate, with data edit checks, configurability and APIs noted as least satisfactory.
Cost in time and materials for implementation of current sample management software systems, subdivided into four categories: commercial, custom built by third party, open source, and developed in-house
Of the responses, 31% only developed their own custom system, 17% only implemented a commercial system, 8% only obtained open source software, and no respondents reported only purchasing a custom system. Overall, 52% reported developing their own system, 36% reported purchasing a commercial system, 24% reported purchasing custom software, and 24% reported adopting an open source system.
In terms of costs, respondents varied somewhat in their reporting of hardware, software, and services from internal staff or external consultants. Overall, the reported cost of implementation varied from zero to over $1 million. No clear correlation could be found between the type of software system built or purchased, and the reported cost. No correlation was identified between the reported cost and satisfaction with the system. Finally, no correlation was identified between the type of repository, size of repository, or level of activity, and cost reported by survey respondents.
The top reasons for choosing systems included cost, flexibility in customizing the system, ability to integrate with other systems, and vendor or supplier staff.
Discussion
The respondents predominantly represented repositories of human blood and tissue samples which were centered in academic medical centers, hospitals, and research laboratories. The IIWG may need to assess the needs of other animal, plant, and environmental repositories through more targeted methods.
The respondents appeared to have substantial and growing repositories run by relatively small staffs. Although new technologies like radio-frequency identification (RFID) and global sample identifiers have been discussed by the IIWG, the community appears to have more basic unmet needs at this point.
There was a clear discrepancy between the importance of, and the satisfaction with, nearly all system requirements.
The cost of implementation was self-reported. Therefore, there is wide variability in what was and was not included. For example, system hardware was included in some responses, but not in others. Additionally, there were respondents who reported zero cost and zero time to implement, which seems implausible to the authors.
For each type of software system (commercial, third-party custom built, open source, developed in-house), we compared (a) initial implementation costs, (b) time to implement, (c) cost of annual service and support, and (d) level-of-effort required by repository staff for implementation. While some anomalies were noted, we could find no significant differences that suggested one option was clearly more or less expensive to implement.
Satisfaction was measured on a number of individual factors spanning the features of the software system and support. Again, there was no significant predictor of satisfaction based on cost of the system.
Limitations of this work include the self-reporting noted above. In addition, multiple responses were received from the same organization which may have skewed some of the results, especially with regard to cost. The professional level of the responder (e.g. chief information officer vs. bench scientist) may also have affected the responses. The limited number of responses contributed to our inability to attribute cost to specific factors. The categorization of the software systems was also confusing to some respondents, which may have led to conflicting results. The survey instrument would need to be more specific to provide conclusive cause/effect relationships such as return on investment (ROI).
Conclusion and Next Steps
The IIWG will continue to gather and report this type of information in an ongoing effort to improve the quality of information management in biorepositories, and will use the results of the survey to further improve and clarify the Best Practices, the Checklist, and the content of future workshops.
Additionally, in order to continue to assess the state of informatics, we will improve the survey by:
1. Standardizing and testing the way in which the questions are asked
2. Adding specificity to the questions about cost
3. Re-categorizing the types of software systems evaluated
4. Investigating ways to increase participation by ISBER members
5. Exploring the possibility of surveying non-ISBER members
6. Conducting interviews to gather clarifying data from those respondents who provided contact information
7. Removing questions that are deemed superfluous
We plan to issue the survey again in 2013.