Abstract
The number of biobanks, in particular hospital-integrated tumor biobanks (HITB), is increasing all around the world. This is the consequence of an increase in the need for human biological resources for scientific projects and more specifically, for translational and clinical research. The robustness and reproducibility of the results obtained depend greatly on the quality of the biospecimens and the associated clinical data. They also depend on the number of patients studied and on the expertise of the biobank that supplied the biospecimens. The quality of a research biobank is undoubtedly reflected in the number and overall quality of the research projects conducted with biospecimens provided by the biobank. Since the quality of a research project can be measured from the impact factor of resulting publications, this also provides some indication of the quality of a research biobank. It is necessary for the biobank community to define “surrogate” quality indicators, and to establish systems of evaluation in relation to current and future resource requirements. These indicators will help in the realistic assessment of biobanks by institutions and funding bodies, and they will help biobanks demonstrate their value, raise their quality standards, and compete for funding. Given that biobanks are expensive structures to maintain, funding issues are particularly important, especially in the current economic climate. Use of performance indicators may also contribute to the development of a biobank impact factor or “bioresource research impact factor” (BRIF). Here we review four major categories of indicators that appear to be useful for the evaluation of a(m) HITB (quality, activity, scientific productivity, and “visibility”). In addition, we propose a scoring system to measure the chosen indicators.
Introduction
The purpose of this article is to review several potentially useful types of indicators, and to assess their advantages and caveats as tools for HITB evaluation. The creation and routine use of biobank performance indicators by different stakeholders and funding agencies should allow for optimization of HITB operations and ensure their sustainability.
Categories of Performance Indicators for a(m) HITB
To cover various aspects of HITB operations, four broad categories of performance indicator have been identified that describe factors related to: 1) quality; 2) activity; 3) scientific productivity; and 4) external dissemination and communication. Arbitrary scoring points (coefficients) have been assigned to items within each of the four categories according to their potential value and importance for the management of a biobank. The list of items is not exhaustive and many other criteria may be integrated in the future, depending on the strategic orientation of the HITB.
To provide comparable assessments of biobanks with different missions and strategic orientations, scoring systems will need to be adjusted. At one end of the spectrum are biobanks that act purely as service providers for private or public partners; at the other end are biobanks that play an active role in scientific projects and develop their own research. Depending on its internal organization, its motivation, and its business plan, an HITB may function at either end of the spectrum or as a hybrid of the two. It is important to measure the extent and proportion of the HITB's contribution to the research. Beyond the “simple” delivery of the biological resource, this contribution can include performing some of the analytical assays and experiments, the collection and update of clinical data, and the final interpretation of experimental data. The extent and quality of the connections and collaborations established between the HITB and other hospital-based resources and infrastructures (e.g., the pathology department, the molecular diagnostic department) and those resources and infrastructures purely dedicated to research activities should be evaluated.
Indicators Related to Quality
The performance indicators related to quality of an HITB and the corresponding scoring points (coefficient) assigned to each indicator are listed in Table 1.
A coefficient has been arbitrarily assigned to each category of indicator according to the average of the different notes given for each criteria. For this purpose and as an example, for the indicator concerning sample quality, a score out of 20 points can be attributed for each of the criteria (for example for DNA quality and control mode or for RNA quality and control mode), and then the average is calculated.
Sample quality
The quality of the morphology of frozen tissue samples is an essential parameter. This point highlights the obvious necessity to set up a tight collaboration between the pathologists and the biobankers. 5 The morphological controls collected must be representative of the frozen tissue. This is a critical point since different analyses performed with frozen tissue using biochemical and/or molecular biological approaches have to be done from representative samples of the lesions of interest. This step is particularly important when “omics” projects are developed further, since results can vary considerably depending on the relative proportion of tumor cells, stromal cells, and infiltrating inflammatory cells, as well as the presence of necrosis. The morphological assessment of tissue samples can be done in various ways depending on laboratory practice, and each method has advantages and disadvantages. One method is to obtain a formaldehyde-fixed, paraffin-embedded (FFPE) tissue sample to mirror the biospecimen dedicated to the freezing procedure, and to stain sections for assessment and archiving by the biobank. The disadvantage of this approach is that the FFPE tissue specimen may not accurately represent the whole frozen tissue specimen. However, this method allows in-parallel performance of immunohistochemical studies on whole sections and secondary tissue microarrays. It is probably good practice to keep these FFPE tissue blocks at 4°C for collection of high-quality and well-preserved tissues, in particular for long-term molecular biology analyses.
Another method is to perform imprint cytology of the tissue specimen before freezing. This method is rapid and allows one to check whether the tumor lesion is really present and whether there is a sufficient number of tumor cells before freezing. However, quality control of the morphology is only partially reliable since tissue architecture is not available on these imprints. Moreover, a potential risk of external contamination exists for tissue specimens when using this procedure.
Finally, another method is to cut a frozen section of the selected tissue immediately before nucleic acid and/or protein extraction. This approach is also called the “sandwich technique” since tissue sections for morphological analysis have to be cut on either side of the section cut for nucleic acid and/or protein extraction. This method allows for excellent quality control of the morphology of the frozen lesion (percentage of tumor cells, area of necrosis, stroma component). However, this raises the temperature of the whole frozen tissue sample from - 80°C or below, up to at least - 20°C (the temperature of the cryostat). This can be a disadvantage if the remainder of the tissue specimen needs to be returned to a lower temperature for future projects. Moreover, external contamination can occur using this procedure. Whatever the laboratory choice for control of morphology, it is critical to archive the corresponding images using software that allows comparison of histology with the biochemical and/or molecular data obtained from the corresponding frozen specimen. Interestingly, criteria concerning the microscopic review have been well established through the “Biospecimen Reporting for Improved Study Quality” (BRISQ) recommendations. 6
It is important to assess the quality of the nucleic acids obtained from the frozen biospecimen by extraction. DNA is more “resistant” than either RNA or protein to the time of warm or cold ischemia, and to formalin fixation. 7 It is important to note that even if the pre-analytical time is not optimal, certain DNA samples may be used successfully to detect genomic alterations in tumor samples. Conversely, other research projects, in particular those using RNA, need to be done using high-quality frozen biospecimens, and RNA quality must be controlled on a bioanalyzer. It is widely claimed that tissue specimens must have an RNA Integrity Number (RIN) above 7 for most transcriptomic projects. However, this assumption needs to be questioned because some tumor pathologies are often associated with necrotic areas or with a large number of hypoxic cells, and so a cut-off RIN value of 7 may be inappropriate for some human tissue specimens. RIN values above 7 are indicative of good quality RNA, but an elevated RIN does not reflect the reality of most of the tumor's “quality status” even before surgical resection. In this regard, tumor samples from some tissues (e.g., lung, pancreas, and central nervous system) will have an exceptionally low RIN even with a short ischemia time before freezing. 7 Whatever the requirements of a research project, it is critical that an HITB demonstrate the efficacy and management of the different pre-analytical steps and parameters of their samples, recording the time of cold ischemia, and if possible the time of warm ischemia, length of freezing, duration of formalin fixation before paraffin embedding, and archive period.
The number of specimens and quantity of tissue available is a quality indicator and the gross weight of the frozen tissue specimen in the cryotube should be evaluated before providing biospecimens for a research project. All cryotubes containing a tissue specimen should be weighed with a precision balance before freezing. In this regard, a number of HITBs only provide extracted nucleic acids and/or proteins to their partners in order to better control the quantity of biospecimen dispensed.
The availability of biospecimens of different origins such as tissues (frozen and/or FFPE); nucleic acids (somatic and germinal DNA, RNA, microRNA), and proteins; biological fluids (whole blood, plasma/sera, pleural liquid, urine, salivary fluid, etc.); and/or primary cell cultures from the same cohort of patients can provide added value for research projects. For example, projects targeting the discovery and/or validation of biomarkers can use data obtained from both tissues and biological fluids. Moreover, the possibility of obtaining frozen and fixed tissues (and potentially tissue microarrays) from the same population of patients can be useful to validate protein expression. Finally, the possibility of developing primary cell cultures in the HITB would be useful for pre-clinical research projects using mouse xenograft models or for in vitro cellular drug toxicity tests. However, it is critical for all collections to obtain optimal biospecimen quality control tools. 8
Frozen, matched “healthy” tissue (or at least matched nontumor tissue) together with a tumor tissue biospecimen is often used in transcriptomic projects. The availability of such control tissue is another quality indicator for HITBs.
Quality of associated clinical data
Apart from calibration and technical projects developed for biological test evaluation or validation, all translational and clinical research projects require the association of clinical, pathological, and/or biological data with the bioresource. For tumor tissue specimens, pathology data may relate to the gross morphology and/or the histology results (in particular, the pTNM staging determined according to the different updated WHO classifications defined for each organ). For histology data, the minimum data set should include the type of tumor (according to the latest international classification of tumors), and the codification (Cim-10 and CIMO). A “transcodification” table must be made available and used by the biobank software that can be employed at the national level. The percentage of tumor cells and the percentage of necrotic areas should be registered.
For some projects, the status of genomic alterations (mutations, translocations, etc.) associated with the tumors should be registered and made available.
The minimum data set linked to the bioresource must include basic demographic information (age, sex, place of birth) and some supplementary clinical data that add value to the collection and provide additional information that may be relevant to the pathology of the specimen. For instance, this additional data could include the place of birth of the parents or the ethnicity of the patient (according to the laws of the country); clinical data more specific to a certain pathology (e.g., work exposure or tobacco status for lung cancer patients); information obtained on patient follow-up (e.g., specific disease survival, overall survival, progression-free survival); and the different treatments received.
The availability of patient-specific information through a centralized, secure, and searchable database (including a security access code and a secure server environment) will contribute to the efficacy of an HITB.
A dynamic system for data recording and an associated quality management system, in particular for follow-up of clinical and biological data, is critical to ensure continual data updates. Moreover, the ability to obtain reports on performance and other quality data must be integrated into the system.
Ethics
There is general agreement that informed consent must be obtained before using a human bioresource for a research project. The demonstration that formal procedures are in place to obtain informed consent systematically, as appropriate, before use of patient samples for research is a strong indicator of good practice. This requirement has only recently been discussed in certain countries (e.g., France) and unfortunately is not mandatory in all countries. 9 Moreover, the laws in this field are changing rapidly; since 2004, it has been mandatory in France to provide detailed information to the patient before taking biospecimens for research. Moreover, a patient can refuse to allow the use of his/her biological resources for research. In some countries it is acceptable to obtain a waiver of consent from a formal process of ethics review as well as to obtain post-operative informed consent. Of course, in some situations the patient cannot give consent (e.g., autopsy).
The success rate of the informed consent procedure is an important performance indicator and the strategy used to obtain informed consent should also be evaluated.
Collection and storage in a secure place of the signed informed consent documents have to be formalized: hard copies signed at least twice must be stored in separate locations and/or scanned and registered in a secure hospital database. That informed consent has been obtained must be checked before using biospecimens for a research project.
The terms used on the informed consent documents must be appropriate and should be checked by the hospital's ethics committee. The use of broad consent for several research projects is possible, although this has been controversial in certain countries.10–12
The HITB must demonstrate its ability to protect the personal data of patients and show that the biological and clinical data shared with partners does not reveal a patient's identity. 13
Other parameters impacting on quality
The risk status of patients for human immunodeficiency virus, hepatitis B, and hepatitis C infection is often requested by industrial partners. However, as viral serological testing is not mandatory for all hospitalized patients, this may be an additional cost to some research projects.
The existence of fully or partially comprehensive back-up collections at a location that is different from that of the main biobank may also be considered a performance indicator. Back-up collections are particularly important for collections of rare tumors. If a different HITB holds a back-up collection, it needs to show the same level of certification or accreditation as the main HITB. For example, these HITBs need to be certified according to the NF S96-900 certification system in France.
The ease of locating biospecimens and the associated data resulting in a good turn-around-time between request and delivery of biospecimens needs to be evaluated.
Coefficients
The points for each category of indicator (coefficient) described in Table 1 have been assigned arbitrarily. For example, Sample Quality has been given a coefficient of 20 points. This means that we recommend that assessments of DNA quality, RNA quality, sample quantity, morphology, and all other measurements within the Sample Quality category, should be given a global score out of 20 points. We believe that the criteria corresponding to the category Ethics, which is primarily related to informed consent, should at least have the same coefficient as those indicators related to the intrinsic quality of the biospecimens.
Indicators Related to Activity
The performance indicators related to the activity of an HITB and the corresponding scoring points (coefficients) assigned to each indicator are listed in Table 2. The term “activity” is not only focused on an analytical count but includes other miscellaneous criteria.
A coefficient has been arbitrarily assigned to each category of indicator according to the average of the different notes given for each criterion. For this purpose and as an example, for the indicator concerning storage and supply, a score out of 10 points can be attributed for each of the criteria, and then the average is calculated.
Storage, supply, destocking activity, and existence of a dynamic strategy
These items include the number of biospecimens collected per year (tissues, biological fluids, etc.) and the number of samples supplied per year for use in research projects (“destocking activity” per year). This indicator is an important one and as an example is accounted for in the HITB evaluation process undertaken by the French National Cancer Institute (INCa). 14
The criteria used to measure the ratio between storage and supply are dependent on: 1) the type of collection being considered (organ, pathology); 2) the type of biospecimens (tissue, nucleic acids, biological fluids, etc.); or 3) the global activity of the HITB. The stored data need to include the number of collected biospecimens for each patient. In France, it is possible to use the budgetary nomenclature set up in Montpellier (France) to evaluate this activity. 15 Different activities can thus be evaluated, such as for example, the number of tissue biospecimens, aliquots of plasma/sera, paraffin sections, and patient tissue specimens included in tissue microarrays. This meticulous work is in fact useful to obtain an objective evaluation of the global activity performed in an HITB.
Strategic planning that accounts for the different results has to be conducted annually in order to reduce or increase the number of biospecimens for storage in the HITB. This “dynamic strategic approach” has to be linked to the number of projects developed each year using the same collection and also to the available annual HITB budget.
A decision may be made to destroy certain stored collections if there has been no biospecimen transfer for a long period of time.
Human resource management
The required number and type of HITB team workers can be directly determined from previously defined criteria for evaluation of activity. The sample turnover (which is objective proof that biospecimens stored in the HITB are used in research projects), is useful for anticipation of workload and planning purposes. The absence of specific expertise in management of the staff (pathologist, biologist, technician, data manager, secretary) can lead to a slowdown in the HITB activity despite growing demands for biospecimens.
Time related to the provision of samples
The turn-around-time (TAT) between the request for biospecimens and their transfer to the researcher/partner is an indicator of the efficiency of the whole HITB system, including the HITB team (general manager, data manager, pathologist, biologist, technician, secretary), and the scientific committee and hospital administration (for contract validation and signature of the material transfer agreement).
The TAT efficiency is evaluated according to the nature of the inquiry: number and type of biospecimens, the differing origins of the biological resources (e.g., tissue plus plasma plus RNA for the same patient), and the number and type of clinical data associated with the biospecimens. It is certainly advisable to provide the various TATs for different requests in the contract.
Biological resource cost recovery and business plan
The contribution of the cost recovery of human biospecimens is difficult to evaluate and depends on the organization of the HITB. However, setting up a “pricing policy” for an HITB is a key factor in determining where and how investments will be made. This point is critical in optimizing HITB infrastructure management. 16
The pricing policies developed by the HITB have to be approved by the hospital management. The pricing must be clearly stated in the contract before the formal request of biospecimens and before setting up a research project.
A clear business model is necessary to anticipate the sustainability of the expenses for manpower (can be maintained, reduced, or increased), for maintenance of the equipment of the HITB, and for purchase of consumables.
Dynamic monitoring
Selected miscellaneous indicators such as the number of contracts signed per year with private and/or public partners can be included in this category. The level of collaboration with local, regional, national, and/or international academic and/or industrial partners should also be monitored.
Indicators Related to Scientific Productivity
The performance indicators related to scientific productivity of an HITB and the corresponding scoring points (coefficients) are listed in Table 3. In assessing the quality of a public hospital biobank, it is critical that these indicators are evaluated. The use of specific indicators depends on the goals of the specific biobank. If the main goal of a biobank is to transfer high-quality human biological resources to academic and/or private partners in order to obtain funding, then scientific productivity is not a relevant or critical indicator. Conversely, scientific productivity is very important in the evaluation of academic biobanks. In this case, scientific productivity can also include the team's expertise in research and development. Development of translational research activities within the biobank provides added value and motivates the HITB team by increasing external visibility. For example, a point system called “SIGAPS” (a software system documenting the scientific publications according to different criteria such as journal impact factor) is used in France to assess a hospital's departments that may include the HITB. Depending on the number of points attained, more or less funding can be allocated to the different teams.
A coefficient has been arbitrarily assigned to each category of indicator according to the average of the different notes given for each criterion. For this purpose and as an example, for the indicator concerning scientific publications, a score out of 10 points can be attributed for each of the criteria, and then the average is calculated.
Scientific publications
This indicator category must be well defined since it is often quite difficult to know the precise contribution of an HITB to a publication. Moreover, the use and final outcome of biological resources after transfer to researchers can be difficult for the biobank to follow. The contribution of the HITB can take different forms. For example, biobankers can be listed as co-authors on publications; if the biobanker is the principal investigator of the research or if he/she is an associated contributor to the publication, the biobank member should be listed in association with the name of the biobank.
The contribution of the biobanker(s) listed as co-author(s) of a publication should be clearly indicated, as currently required by some scientific journals; in general, this contribution should be more substantive than the transfer of samples.
A biobank member (e.g., manager, pathologist, biologist, technician) can be cited in the “Acknowledgments” section of the publication and the HITB partnership should be included. The HITB itself can also be cited in this section. These arrangements should be clearly stated in the contracts.
If the biospecimens provided were used to obtain results for publication, the HITB should be listed in the “Materials and Methods” section.
The BRIF working group is currently developing a framework for 1) creating a tool for calculating the research impact of bioresources based on a metric (algorithm) and a unique digital resource identifier; and 2) assessing requirements for citation/acknowledgement of bioresources in order to track their use in research. 18 Each collection of a biobank would be assigned a digital object identifier (DOI) which could be systematically referred to when work that used the biological resource is submitted for publication (in which the work or part of the scientific work used biological resources).
BRISQ recommendations are to be followed by authors of manuscripts, so the biobank must provide all the information listed in the BRISQ (e.g., storage temperature, duration, shipping temperature, etc). 6
Impact factors
The quality and value of the scientific work are mostly linked to the current “scientific world system” and to the reputation of the journal in which the work is published. Each journal has an impact factor, which is higher in the more prestigious scientific journals. Although other criteria such as the citation index of the publication exist, the impact factor is the indicator most frequently used to evaluate individual researchers, research teams, and larger organizations (e.g., departments, hospitals, universities). A value for cumulative impact factors per year can be calculated in different ways (e.g., according to the presence or absence of a biobank member in the list of authors; according to the position among the authors of a biobanker member). The impact factor for a specific collection stored in an HITB, which would indicate its quality and richness, could be used to justify strategies for increasing the number of stored samples corresponding to a specific pathology or organ.
Finalized research
It is necessary to distinguish between patents obtained subsequent to research performed in the HITB (such as an innovative test for biospecimen quality evaluation) and patents obtained by research teams using biospecimens transferred from the HITB.
Books and book chapters in the field of biobanking and popular science can be listed.
Participation in scientific committees that organize conferences on biobanking themes is an indicator of scientific activity.
Communications
Presenting lectures and posters at national and international meetings can be considered, provided the authors are members of the HITB and/or the HITB is listed in the abstract.
Grants
Success in obtaining research grants is an important indicator for an HITB, with both national and international grants needing to be recognized. 19 Further distinction should be made between grants obtained by the HITB itself or by team researchers associated with the HITB (and using samples transferred from the HITB).
Coefficients
The coefficients could be based on the cumulative impact factor per year, and to the grants obtained.
Indicators Related to External Dissemination and Communication
The performance indicators related to external dissemination and communication of an HITB and the scoring points (coefficient) assigned to each indicator are listed in Table 4. This is a group of heterogeneous items aimed at ensuring the visibility of the HITB at the local, regional, national, and international levels. As the number of HITBs around the world is increasing, creating a competitive environment, increasing their visibility is an essential part of their work.
A coefficient has been arbitrarily assigned to each category of indicator according to the average of the different notes given for each option. For this purpose and as an example, for the indicator concerning certification, accreditation and labels, a score out of 20 points can be attributed for each of the criteria, and then the average is calculated.
Certification, accreditation, and labels
The certification of a HITB is primarily related to the use of biological resources for research projects and not to the immediate health care of patients. A biobank can be certified according to a variety of norms such as ISO 9001, a series of documents that comprise guidance on how to implement a quality management system. NF S 96-900 is a quality management system as well as other business requirements specific to biobanks in France; this norm could be upgraded in the future to become an ISO, or international standard.
The ISO 17025 standard, which provides accreditation of the competency of testing and calibration laboratories, can be used to establish the quality of the transferred products. Certain accreditations such as ISO 15189 are more appropriate for medical laboratories; this can be attributed to an HITB if samples are dedicated to diagnostics, prognostics, and/or theranostics of hospitalized patients.
The “IBISA” label used in France certifies that the biobank has a working technological platform and that it develops innovative projects using biological specimens (e.g., improvement of nucleic acid extraction)
Networking and collaboration
The participation of an HITB in national and international networks demonstrates recognition and involvement of its members in the biobanking field. 2 A large number of HITB networks exist around the world. In France, different thematic networks have been created for different types of biological samples such as liver or lung cancers, lymphomas, and sarcomas or mesotheliomas. 14
Networks of HITBs associated with research projects focused on the same pathological condition can be organized. Through these networks, an optimal number of biospecimens of rare pathologies can be obtained for a specific research project. 20
Membership in biobanking societies
In France, the INCa has assembled a group of experts working at the national level to promote good practices for the use of biological specimens. This group regularly provides detailed guidance in different areas of biobanking such as pricing and cost recovery for HITB managers, as well as hospital authorities.
In recent years, a variety of international biobanking organizations have been established, such as the Biobanking and Biomolecular Resource Research Infrastructure (BBMRI), the International Society for Biological and Environmental Repositories (ISBER), and the European, Middle East and Africa Society for Biopreservation and Biobanking (ESBB). Through the organization of symposia and meetings, as well as the dissemination of guidelines, these different organizations share procedures and ideas to improve the activities of HITBs.17,21 Belonging to one or more of these societies allows managers and staff to stay in touch with the latest developments in the field.
Educational programs and dissemination of information
The development of HITBs and the specificities of the work involved have led to the development of a new job description. It is obvious that the management of a HITB has to be done by a “biobanker” who has gained the necessary knowledge and technical skills to be successful. 22 Individuals working in HITBs represent a variety of professions (e.g., pathologist, biologist, technician, data manager, scientist, quality controller, statistician), each making a specific contribution to the endeavor. HITB managers and staff benefit from a comprehensive understanding of the breadth of responsibilities of a HITB including quality control procedures, data management, establishment of contracts and material transfer agreements, and business plan development. For this reason, it is critical to be able to offer opportunities for continuing professional training to HITB managers and staff.
Professional training for biobankers can be undertaken at different levels such as through the organization of different internal meetings within the HITB; or acquisition of diplomas such as the master's degree within faculties of medicine or the sciences. As an example, France has established a master's degree dedicated to biobank management. 23 Other biobanking education programs have also recently been developed such as the International Biobank Education Course (www.biobanking.org).
Competence of the HITB team
Attendance at HITB training courses and numbers of diplomas granted provide evidence of the competence of HITB personnel. It is also important to take into account any complementary competence of the HITB members (e.g., members with qualifications in medicine, informatics, biological sciences).
Integrated or associated technology platforms
HITBs can develop a number of different technical platforms that can be integrated into an HITB or can be associated with an HITB through contracts or partnerships. Examples include platforms for nucleic acid extraction and quality control (spectrophotometer, bioanalyzer); biopathology platforms (paraffin-embedded tissues, tissue micro arrays, immunohistochemistry, in situ hybridization); molecular biology platforms (DNA cycler, sequencing); transcriptomic platforms (scanner for different chips, bioinformatics expertise); and other more specialized platforms including laser capture microdissection, primary cell cultures, and proteomic analyses. This indicator is not used for HITBs that only function as secure storage areas for biological samples and do not participate in translational and clinical research projects.
Marketing activity and patient involvement
Increasing the visibility of biobanks can be done through different forms of marketing. Examples include maintaining a website, dissemination of an HITB brochure, and organizing meetings or publishing articles for the general public about the activity of the HITB. Biobanks should make clear in all their marketing materials their policies related to access and prioritization for obtaining samples. Active participation of patients and patient representatives in some HITB committees should to be considered.
Partnerships with industry
This is currently one of the most important indicators attesting to the attractiveness of the HITB in showcasing its capability to rapidly provide high-quality biospecimens. This indicator also provides evidence of the sustainability of, and capacity to obtain, private funding. Partnerships with industry can be set up through scientific collaboration (e.g., patents, communications, publications) or through contracts for sample transfer without scientific collaboration. It is necessary to make allowance for the fact that some HITBs may be prevented from developing such partnerships because of prevailing attitudes in the local community.
Coefficients
Within the category “Indicators of External Dissemination and Communication” we believe that involvement in international networks and scientific collaborations with industrial partners should be assigned relatively high coefficients.
Toward a New System of Appraisal of Performance of HITB Using the Coefficients Assigned to Each Category
Ratings systems for evaluation are never perfect and they can sometimes be considered as an arbitrary means of arriving at a conclusion. However, the establishment of different coefficients based on various items classified into large categories, as described above, can give an objective idea of the efficacy of an HITB. The most sensitive point probably lies in determining the value of each coefficient according to the considered item. The coefficients assigned are very informative, but they may be improved in the future. According to the total value obtained using each of the four large categories of performance indicator, five different ratings could be assigned (Fig. 1): A+ or “excellent” (>200 points); A or “very good” (150–200 points); B or “good” (120–150 points); C or “moderate” (100–120 points); and D or “weak” (<100 points).
Global proposal for an HITB evaluation scheme utilizing the coefficients assigned to each class of indicator.
Conclusion
The establishment of “performance indicators” for HITBs aims to maintain the quality of different procedures, in particular those set up prior to certification or accreditation. These indicators can assist in achieving durable optimization of the HITB. The indicators detailed above can also be used to evaluate other criteria related to the management of an HITB, such as the added value of the infrastructure through scientific publications. The list of items described above is not definitively or rigidly fixed and we hope to be able to add other criteria in the future that could be of potential interest for HITB evaluation. The research activity of a HITB can be evaluated in different ways: Based on assessment of resulting translational and clinical research and also based on additional indicators such as patents, consulting activities and different expertise, and organization of national or international symposiums or meetings.
Implementation of the BRIF project (http://www.gen2phen.org/groups/brif-bio-resource-impact-factor) has rapidly defined indicators for HITBs.18,24 In a complementary manner, transmission of an annual activity report (e.g., the requirement set out by the INCa in France in the last 2 years) is another indicator of HITB activity over time.
HITBs do not usually undergo scientific evaluation (e.g., in France, the research institutions Inserm or CNRS do not perform evaluations). Currently, there is no rating system that can lead to a “label of excellence” and thus it is not easy to obtain funding through this kind of notation. One reason for this could be the absence of an evaluation grid for HITBs that includes objective criteria. Thus, establishment of performance indicators could be of interest in creating such an evaluation process. The coefficients utilized in this review are open to discussion and could be modified in the future, in particular when taking into account the “type” and objective of the HITB. Indeed, some HITBs may not be involved in scientific research projects and may have as their unique goal the collection, storage, and transfer of biological samples to public and private partners. In this case, the indicators related to scientific productivity of the biobank would not be applicable.
Ratings that provide grades such as A+, A, B, C, or D may be welcomed by the “field workers,” and thus may give a reductionist view of an HITB's activity. Others may perceive the rating system in a negative way. However, the multiplicity of HITBs around the world has increased the overall cost of the infrastructures, and it is important to ensure the sustainability of the staff working in the HITB. This has created an urgent need to evaluate HITBs to at least check their efficacy in delivering high-quality samples for translational and clinical research. The consequences of evaluating HITBs in the manner described here could lead to an improvement in their functioning in different ways, in particular by optimizing certain targeted collections. Some synergistic programs involving different HITBs could be created to obtain large numbers of high-quality biospecimens to support excellent research projects.
Footnotes
Acknowledgments
The authors wish to thank the BRIF group for helpful discussion during the preparation of the manuscript
Author Disclosure Statement
The authors declare no conflict of interest