Abstract
With the increasing number of research biobanks and the importance of their role in supporting medical and biological research, the development and sharing of biobanking best practices and benchmarking standards has become paramount. To promote outstanding biobank services for research, the Research Biobank of the Year Competition (RBYC) has been inaugurated by the European, Middle-Eastern, and African Society for Biopreservation and Biobanking (ESBB) in October 2013. The procedures for the call and evaluation procedure, including the newly developed scoring system, are presented here. The statistics and evaluation results of the first year's applications, as well as the experiences of the jury are reported here, and improvements for the RBYC in subsequent years are proposed. Beyond offering a unique benchmarking opportunity for biobanks, the RBYC is discussed as a novel tool to enhance biobank quality, transparency, usage, connectivity, innovation, and sustainability.
Introduction
W
Methods/Procedure
Call for entries
The text of the call for entries 3 was drafted by a task force (Robert Hewitt, Christian Oste, and Christina Schröder) after the ESBB Council had decided on March 12, 2013 to endorse the competition. The Call was approved by the Council and published over the ESBB website on April 26, 2013, with a deadline of July 1, 2013. The Call was open for entries in two categories: 1) Biobanks for human samples, and 2) Biobanks for biological and environmental samples. The call was restricted to Biobanks for Research Purposes. In line with the geographic scope of ESBB as a society, eligibility was limited to biobanks situated in Europe, the Middle-East, and Africa. Biobanks both in public and private settings were eligible. In parallel to the competition, applicants were also invited to present a poster on their biobank at the Verona conference.
Jury
To assess the entries and present a shortlist, the ESBB Council agreed to invite a broad expert panel to balance any interests which might be involved. The Council discussed potential panel members and selected judges covering a broad range of biobanking expertise (including biobank users), and representing as many regions/countries covered by ESBB as possible. Reviewers (Table 1) committed themselves to treat entries and proposals confidentially (except the shortlist) and to refrain from voting where they might have a potential conflict of interest with respect to a specific biobank entry.
Assessment of entries
The entries received were checked for eligibility by the competition organizers (Robert Hewitt and Christina Schröder), and the eligible ones forwarded to the panel of judges. In addition, the upload of poster abstracts was verified for all applications. All panel members reviewed all applications to ensure consistent evaluation. After a preliminary review of the entries, the panel held two conference calls, one in mid-August 2013 to agree on the details of the evaluation and scoring procedure, and a second one early in September to decide on ranking and shortlist. After the first teleconference, a spreadsheet based on the list of entries and the six call criteria (quality, transparency, usage, connectivity, innovation, sustainability) was drafted and intensely discussed by the panel, who:
• decided to weight the six criteria according to importance,
• subcategorized each criterion further, and
• included biobank impact in terms of the scientific publications from supported research, even though this had originally not been mentioned in the Call.
Panel members felt that different criteria were not all equally important indicators of biobank performance and agreed upon the different maximum scores given in Table 2. The subcriteria (Table 3) provided an assessment catalog for judges to ensure consistent evaluation but were not given specific scoring ranges. Based on this differentiated scoring system, all entries were evaluated. If a panel member had refrained from voting for one or another biobank because of a potential conflict of interest, total scores for this biobank were normalized (i.e., divided by the number of votes given and multiplied by eight). All results were compiled in a cumulative spreadsheet. This revealed the ranking of applicants, which was agreed on unanimously (and kept confidential) by the panel in the second teleconference. It was decided to shortlist three biobanks and invite their representatives to give 10 minute presentations at the ESBB Annual Conference in Verona. After these short presentations, ESBB members present were invited to select the winner using an electronic voting system, which allowed audience participation in the judging process. To estimate the audience's perception of the competition, ESBB asked for their comments in the conference evaluation survey.
Each jury member has filled in this table for all 17 biobank applications (inserting “0” or “not scored” in lines representing biobanks where judges felt biased). Scores in the six columns on quality, transparency, usage, connectivity, innovation, and sustainability, yielded from the sub-scores shown in Table 3. As a score in the “impact” column, “0” was inserted if information was “unavailable”, “1” if information was “available” and “2” if it was excellent. For final ranking of biobanks, scores were summarized for each applicant across all eight judges' summary score sheets and normalized to the final values as explained in the text.
The jury had agreed upon the subcriteria prior to evaluation of entries and used these tables to support their individual evaluation, thus ensuring consistent decision-making. So the subcriteria have been weighted only on an individual level totaling in the overall range that had been agreed (see Table 2, column headers).
Although one jury member had proposed prior to the second conference call to inform applicants on their individual scores in relation to average scores, the panel decided not to communicate any scores, average scores, or ranking to applicants this year, and agreed upon respective texts for their notification.
Results
ESBB received 19 entries, 17 of which proved eligible (one proposal had been submitted twice, and another was from outside the geographic area covered by ESBB). Applications came from 11 countries (Table 4) and were all in the category Biobanks for Human Samples, none in the Enviro-Bio sector. So awareness of the Call has obviously been widespread, but motivation to participate in the biomedical research sector by far exceeded the biological research sector. The fact that there were 17 entries was considered a very positive and encouraging response to the call, given that participants were offered nothing in return apart from the chance of being shortlisted or winning the competition.
The jury found all applications were written to a consistent and highly professional standard and considered them the only basis for their scoring decision (i.e., additional information on the applicant biobanks such as websites and brochures were not taken into account). Evaluation was done on a basis of good faith, without any means of verification of the written information given in the entries. Applications were evaluated using the score sheets shown in Tables 2 and 3. Six out of eight jury members abstained from voting for a specific biobank because of a potential conflict of interest.
The three biobanks scoring highest were combined in the shortlist (Table 5) and published first in the Verona conference brochure, together with the shortlisted biobanks' poster abstracts. The corresponding posters were displayed in a prominent position, allowing conference attendees to discuss details with representatives of the shortlisted biobanks prior to their formal presentations in the Society Forum session.
At the beginning of the Society Forum, the RBYC evaluation procedure and scoring system were presented by panel members Julie Corfield and Andreas Tiran, who also chaired the subsequent presentations of the shortlisted biobanks. After these presentations, ESBB members voted over their electronic devices, selecting HUNT biobank from Norway as the winner. The platinum sponsor of the ESBB conference presented the HUNT representative with the Challenge Cup (Fig. 1) and a certificate for discount towards the purchase of the company's products. Compared with the interactive sessions of the ESBB meeting, where the electronic voting system had also been used, there were a relatively small number of participants voting. As also indicated by response to the conference evaluation survey (Table 6), this might have been due to the fact that this time only ESBB members in good standing were entitled to vote, and that the session on Friday evening was the last one of the conference. The overall response given by the survey and by feedback to both the conference and the competition organizers was very positive, suggesting that the RBYC should be continued in the future. Despite the jury's original decision not to communicate scores (cf. section “Assessment of Entries”), many applicants were very keen to be informed of the scores they had achieved. Obviously the benchmarking opportunity offered by the RBYC was much appreciated by participants. Therefore, the panel revised their decision and agreed to inform participating biobanks on their scoring, compared with average scores, as shown in Table 7.
The winner is… Kristian Hveem is presented the trophy by Josep Fernández, representative of ESBB's platinum sponsor.
With question 16 of the survey, conference attendants were asked: “Please comment on the “Research Biobank of the Year Competition”. Do you have any suggestions for improvement?” All answers given to this question are shown here in original wording/spelling.
in brackets: possible scores /single vote; **average scores all judges.
As required by many applicants. they were informed ex post on the scores they had reached, compared to average scores in the 7 categories reached by all participants of the competition and by the shortlisted ones, respectively.
Discussion
With the RBYC launch described above, ESBB and its ESBBtranslate Working Group have investigated how to:
• encourage the biobank community to increase their service standards;
• focus attention on refining assessment criteria towards an optimized use of biobank resources and enhanced collaboration of academic biobanks with industry.
In other words, we wanted to answer the question of whether an open competition and a public prize are appropriate and efficient tools to advance the biobanking-for-research field, and, if so, how and why?
With 17 eligible, consistent, and comprehensive entries from 11 countries, we have seen a very positive response to the first call in a previously unknown competition. The distribution of entries from across 11 countries within the limited geographic area covered by ESBB (Europe, Middle-East, and Africa) has demonstrated the biobanking community's broad and immediate awareness and acceptance of the call.
Nevertheless, the fact that no entries at all have been received from non-human biobanks obviously requires further investigation. To date, we can only speculate on the reasons: If at all, how far does the number of biobanks for medical research exceed the number of biobanks active in biological and environmental research? Do the latter deem competition and benchmarking not that relevant or applicable for their work? While these questions will certainly be considered by ESBB for future RBYC terms, this time only the prize in the category “Biobanks for human samples” could be presented and is discussed here.
Contestants and ESBB conference attendees have shown their interest and support for the RBYC in various ways:
1. The willingness of contestants to diligently prepare 6 page proposals as well as abstracts and posters demonstrates that the competition had a highly motivating effect. This is even more impressive, given that the call promised nothing in return except for the chance of being shortlisted and the possibility of being awarded the Challenge Cup.
2. The strong demand for scoring information shows that the benchmarking opportunity offered by the RBYC was highly valued by contestants.
3. Two out of three shortlisted biobanks have used their nomination for promotional purposes.4,5
4. Applicants have reported the value of the RBYC in encouraging them to “step back” from daily operations and reflect on their biobank's function in a more general sense.
5. Calling the RBYC a good idea and a “great innovative component of the conference”, respondents to the post-conference survey (Table 6) mirrored most conference participants' and exhibitors' spontaneous positive perception of the contest, which had also been communicated personally and directly to the organizers.
6. Many conference attendees suggested moving the competition presentations and voting procedure to a more central part of the conference programme to ensure maximum attendance. This demonstrates the high level of interest and support for the RBYC.
7. The platinum sponsor of the ESBB conference used the award-winning ceremony for promotional purposes. 6 After the conference, several companies declared their interest in sponsoring future RBYC winners and ceremonies.
We believe the success of the RBYC may be traced to several key factors:
1. Entries are assured because the possibility of being shortlisted or of winning the competition is a strong motivation to biobanks interested in demonstrating their value to potential biobank users.
2. The interest of the biobanking community is assured because the criteria being assessed are of widespread relevance and general importance. This is reinforced by the fact that any competition is by nature interesting and entertaining, particularly where audience participation is involved.
The call has obviously succeeded in addressing the major issues of the biobanking-for-research field by asking for the six criteria outlined in Table 2, thus achieving its primary goal to help optimize biobank services and usage.
When the jury developed the novel RBYC scoring system described above, they also evaluated existing tools and initiatives for the assessment of scientific biobank impact and performance, such as BRIF, 7 BRISQ, 8 and the ISBER self assessment tool. 9 The latter is meant for internal/confidential assessment of biobanks to help them identify potential deviation of their operations from ISBER best practices, while BRISQ is focused on correct and traceable reporting of biobank parameters. The Bioresource Research Impact Factor (BRIF) initiative aims at constructing a quantitative tool to calculate the research impact of bioresources, in order to document and acknowledge their quantitative use, the quality and importance of research results involving these resources, and the effort to make available a valid bioresource. BRIF is not designed to measure the quality of a biobank in a more general sense.
Although RBYC judges felt that the scoring system they had developed might help fill the need for a general benchmarking system for biobanks, they are fully aware that there is room for improvement in the competition call, procedure, and evaluation, as outlined below. In addition, ESBB should strive to give a stronger say to biobank users when selecting future panel members: Only two out of eight members of the 2013 RBYC Jury have not been involved with applicant biobanks, indicating that this has been more of a biobanking expert, rather than a biobank user panel.
Based on the jury's and the organizers' experiences, we propose the following improvements for upcoming RBYC terms:
1. The Call text should contain detailed information on the evaluation and scoring system. A clear and elaborate definition of the criteria and of their weight will make the work of next years' candidates and jury much easier.
2. The judging process should be facilitated and stream-lined by guidelines for judges to be issued by the 2013 jury as a first version. Guidelines should include advice on the integration of BRIF into the scoring system. Upcoming panels of judges should strike a balance between biobankers and biobank users.
3. The final voting procedure should be scheduled as a central part of the ESBB Annual Conference to allow for broad participation. ESBB should also decide if nonmembers could be entitled to vote.
4. Post-competition reporting should include information for applicants on the scores they achieved, compared to average scores. While the complete ranking will certainly not be published, the coverage of the shortlist should be discussed. With 3 out of 17 contestants, it was about a fifth (or 20%) in 2013, a proportion the judges deemed appropriate.
Conclusion
The RBYC attracted a lot of interest and put a spotlight on a key issue for biomedical research: the quality of biobank services. It was well received—comments were overwhelmingly positive.
There were many benefits:
• to the winner: promotion of own biobank;
• to the shortlisted entries: promotion of own biobank;
• to the contestants: benefit of viewing own biobank critically from a different perspective;
• to the judges: improving their knowledge about the assessment of biobanks;
• to the biobanking community: focusing attention on the quality of biobank services;
• to the research community: better quality biobank services;
• to the sponsor(s): promotion of own company.
As a result of the positive experience of the first RBYC, we intend to repeat the competition in future years, focusing attention on continuous improvement of the assessment criteria used.
Footnotes
Acknowledgments
The authors gratefully acknowledge the courage, time, and effort of all biobanks participating in this first RBYC term, thus supporting the work, results, and experiences described above, and advancing the biobanking-for-research field as a whole. Critical revision of the manuscript by representatives of shortlisted biobanks, Prof. Kristian Hveem, Dr. Balwir Matharoo-Ball, and Reinhard Thasler, is very much appreciated.
Author Disclosure Statement
No competing financial interests exist.