Biospecimen Data Reporting in the Research Literature

Abstract

A substantial fraction of biomedical research depends on the reliability of human biospecimens but variations in sample manipulation during collection, processing, and storage can differentially alter molecular integrity and influence interpretation of the resulting derived data. Details of biobanking processes are rarely adequately described in research publications, preventing reviewers, readers, and scientists seeking to replicate the findings, from appreciating and adequately considering preanalytical variations contributing to results. To address these shortcomings, a set of reporting guidelines, the Biospecimen Reporting for Improved Study Quality (BRISQ) criteria, were developed in 2011. In this study we evaluated the uptake and reporting of BRISQ criteria in 324 articles across four leading biomedical journals using human biospecimens and published before (161; in 2010) and after (163; in 2014) the delineation of the BRISQ guidelines. We found that even within journals recommending use of BRISQ, manuscript-level uptake. and reporting of the relevant biospecimen information is not widespread or uniform. In the future, an enhanced biospecimen reporting strategy to better serve the needs of researchers, reviewers, and journals may be considered to strengthen research reproducibility for the benefit of the research community at large.

Introduction

The global biomedical research enterprise is facing an existential challenge precipitated by a mounting awareness, both within the community¹ and increasingly outside² of it, of the scale and scope of irreproducibility of research findings. These concerns are superimposed on an environment of scarce funding and compounded by the move toward large-scale, high-throughput Omics-based research methodologies. We now have the technological capacity to produce low-quality data from low-quality inputs with unprecedented efficiency, resulting in an ability to get the wrong answers with unprecedented speed.³

Increased awareness of research irreproducibility has spurred initiatives to investigate its causes and prevalence.⁴ A recent analysis by Freedman et al. concluded that the cumulative prevalence of irreproducible preclinical research may exceed 50%, and causes delays in therapeutic drug development.⁵ In the United States alone this translates into US $28 billion/year spent on preclinical research that is not reproducible.⁵ Of several culprit causes, an estimated one third of irreproducibility can be attributed to “Biological reagents and reference material,” which includes biospecimens used as input for research studies.⁵

Addressing irreproducibility by improving the reporting of biospecimen variables that impact downstream data formed the rationale for the creation of the Biospecimen Reporting for Improved Study Quality (BRISQ) criteria. BRISQ was published in 2011^6–8/2012⁹ by a range of experts involved in research utilizing human biospecimens: laboratory scientists, clinicians, pathologists, biobankers, journal editors, and other key stakeholders.

The BRISQ criteria consist of three levels of data elements necessary for consistent documentation of biospecimen type and factors that could influence their molecular composition or integrity. None of the data elements is coded, and all information can be collated without constraints, as free text by the user.

The first tier consists of data elements that are recommended to be reported, and include information on biospecimen type and anatomical site from which the biospecimen was derived, and also the collection, stabilization, and preservation approaches. The second tier builds on this with information considered beneficial to report, this includes data elements such as timing between biospecimen collection and stabilization. The final tier of data elements consists of additional information that may be useful such as environmental exposures to which the biospecimen may have been subjected, or type of storage container in which the biospecimen was stored.^6–9

A number of journals have now included BRISQ reporting as a requirement for article publication and BRISQ is listed on the Equator Network site for reporting guidelines¹⁰: BRISQ was implemented as a requirement by the Journal of Pathology¹¹ and by Histopathology¹² in 2012. The Nature Publishing Group¹³ also implemented BRISQ in 2013, followed by the Biobanking and Biopreservation Journal¹⁴ in 2014. However, little is known about the scope and scale of BRISQ reporting uptake in these journals or others that have not formally endorsed these guidelines. Within the context of current concerns surrounding reproducibility of research, we sought to better understand the extent and quality of preanalytical variable reporting in the biomedical literature in response to existing BRISQ guidelines and to determine whether current reporting approaches are adequate.

Methodology

We examined the impact of BRISQ reporting guidelines on the quality of biospecimen reporting across a broad range of research studies (basic, translational, and clinical) spanning different disease foci (cancer and noncancer), in a representative selection of four high impact journals. The Journal of Pathology (JPath) and Nature Medicine (NatMed) were selected as representative translational research journals that have implemented BRISQ as a requirement for authors in 2012 and 2013, respectively. The New England Journal of Medicine (NEJM) and Cell were selected as journals with a focus on clinical or basic research, respectively, that have not implemented BRISQ. A literature review was then performed to evaluate the reporting of BRISQ Tier 1 Elements in articles published before (2010) and after (2014) the BRISQ guidelines were released.

All articles were accessed online and analysis was conducted between January 2015 and August 2015. The archives from 2010 and 2014 comprising articles and resources in Cell; original articles and short reports in Journal of Pathology; original articles and brief reports in the New England Journal of Medicine; and articles, letters, and technical reports in Nature Medicine, were reviewed. The following information was collected: journal name; year, volume, issue, first and last page; first author; title of article; article type; human biospecimen use described (yes/no); type of biospecimen; remarks; abstract.

Each article, including the supplements, protocol, and additional information was screened to identify those involving human samples. A detailed analysis was then conducted on a subset of these articles. This subset comprised eight publication groups of articles (two publication years, four journals). Each group included randomly selected articles (randomized with use of a random generator application) from all biospecimen-related articles in a given journal and year (up to a maximum of 50/year and 100 for each journal) (Cell = 29; JPath = 100; NatMed = 95; NEJM = 100). The articles assigned to each publication group were then evenly distributed among the four reviewers (A.J.M., D.S.-D., L.A.M., and S.C.).

All articles identified were analyzed for the presence of BRISQ tier 1 elements and for 3 additional non-BRISQ elements regarding patient informed consent, reporting of research ethics board approval, and naming of research ethics board, as previously described.¹⁵

A score of 1 was given if the required tier 1 information was available; an intermediate score of 0.3 was given in cases where the required information could reasonably be inferred by an experienced researcher; a score of 0 was given if there was no information available and it could not be inferred. If a tier 1 element was not applicable in a given case, N/A was assigned and the denominator adjusted accordingly. BRISQ reporting related to biospecimens collected for primary clinical use and secondary research data generation was excluded from the analysis. A total of 324 articles were evaluated, 32 of these were excluded from analysis because BRISQ did not apply. A summary of the analysis pathway is given in Figure 1.

FIG. 1.

Flowchart of article selection and analysis process. BRISQ, Biospecimen Reporting for Improved Study Quality.

Concordance and statistical analysis

Fleiss' kappa was calculated to assess the reliability of agreement between raters assigning categorical scores for the articles evaluated, and a value of 0.640 was obtained for the four reviewers, indicating strong agreement. These concordance calculations were performed on a subset of articles (n = 16), and calculated as mean value of scores assigned to biospecimens reported. Only biospecimen types identified by at least three of four of reviewers were included in the analysis.

Changes in BRISQ elements between time points were evaluated using t-tests. p < 0.05 was considered significant. Statistical analysis was performed using GraphPad Prism version 5.01 for Windows (Graphpad Software, La Jolla, CA).

Results

The proportion of articles published in these four journals that utilize human biospecimens ranged from ∼7% to ∼75% with an average of 42% across all journals (Table 1). The number of human biospecimen types used per published article also remained consistent between journals and over time (mean ∼1.54 ± 0.2).

Table 1.

The Total Numbers of Articles and the Proportions of These Articles Using Human Biospecimens in Two Time Periods (2010 and 2014) for the Four Journals Assessed

	2010		2014
	No. of articles	Percentage of articles using biospecimens	No. of articles	Percentage of articles using biospecimens
Cell	116	14	174	7
Journal of Pathology	101	75	109	62
Nature Medicine	113	40	137	50
New England Journal of Medicine	212	46	211	37

Overall the reporting of BRISQ tier 1 elements was found to be relatively consistent across journals. There was also no significant difference between journals or clear separation between journals that had or had not implemented BRISQ reporting (Fig. 2).

FIG. 2.

BRISQ element reporting in each of four journals. For all four journals [Cell, (A); Journal of Pathology, (B); Nature Medicine, (C); New England Journal of Medicine, (D)] and 2-year time periods, the mean BRISQ tier 1 element score is shown for each BRISQ element (bars represent means + standard error. Dark bars 2010, light bars 2014).

There was a clear divide between the reporting frequency of BRISQ elements derived mostly from clinical charts and elements derived mostly from the biobanking process. In general, most clinical elements regarding biospecimen type, anatomical site, clinical characteristics and diagnosis, and vital state of subject were relatively well reported in all journals reviewed. By contrast, laboratory and biobanking parameters were uniformly less well reported across all journals (Fig. 3). In addition, although overall there was no significant improvement in reporting seen in reporting the laboratory and biobanking parameters in three journals, a trend to improvement was clearly evident in Nature Medicine articles between 2010 and 2014 (Fig. 2C).

FIG. 3.

BRISQ element reporting relative to Clinical and Biobanking subcategories of BRISQ elements across all journals. (A) The overall mean BRISQ tier 1 element score across all journals for each BRISQ element. (B) The overall mean BRISQ tier 1 scores for elements grouped as clinical chart derived, biobanking derived, and ethics derived, for all journals and years (bars represent means + standard error).

The quality and extent of reporting of additional non-BRISQ elements related to ethics was used as a control in our analysis, as previously described.¹⁵ The level of ethics reporting was consistently high in all journals and at both time-points.

Discussion

We have studied the degree to which biospecimen-related data elements are reported in a set of four leading biomedical journals covering a spectrum of health research. Our results extend previous findings based on two journals dedicated to cancer research¹⁵ and confirm that the subset of biospecimen-related data that is derived from laboratory and biobanking processes (as delineated in the BRISQ standard) is mostly poorly reported. We also find that overall there has been relatively little improvement since the BRISQ standard was published in 2010. However there has been progress in at least one journal that implemented BRISQ.

A recent article¹⁶ examined the uptake of 37 oncology-specific reporting guidelines listed by the Equator Network and found that the number of citations for guidelines relevant to all aspects of a study were more highly cited (n = 7, median 151 citations) than those relevant to only certain parts of a study, such as methods or study characteristics (n = 24, median 27 citations).

The BRISQ guidelines were developed by an international committee with diverse research and biobanking experience and represent the first such standard focused specifically on biospecimens. BRISQ is a standard relevant to the methods of a study and has been cited by >40 research publications since its publication,^6–8 in addition to many reviews and articles on reporting guidelines and has been adopted by several journals. Therefore, although systematic adoption of BRISQ in research articles has been relatively limited, this is the same for most appropriate and relevant reporting standards.

The sheer breadth of research that BRISQ attempts to be applicable to, may be one of its key limitations. In addition, although other reporting frameworks provide specific guidance on their usage,¹⁷ BRISQ is inherently open to interpretation. It requires each use of the criteria to be independently “interpreted, modified, and applied according to the context of the study being reported”^6–9 by the user. The breadth and depth of reporting categories creates some challenges for biobanks and researchers to assimilate and implement and for reviewers and journals to evaluate.

Moreover, several categories are relatively vaguely defined, with items that are not comprehensive for a given category of information and repetitive across different items (e.g., clinical characteristics of patients vs. clinical diagnosis of patients). Some of the reporting items in BRISQ overlap with the elements in other reporting frameworks such as the Standard Preanalytical Code (SPREC^18–20) and the REporting recommendations for tumour MARKer prognostic studies (REMARK).^21–23 Implementation of BRISQ is further complicated by the lack of flexibility and concrete rules to determine which items are most important to report for a given type of research study.

Although BRISQ represents an important strategy toward improving communication of biospecimen information, lackluster uptake by the field also reflects the challenges of implementation of many competing reporting guidelines. Ours is not the first group to acknowledge the difference between creating guidelines and enforcing them, and this discrepancy remains an issue across the spectrum of research.^24–27 Evaluating the effects of recommendations and policies is critical in identifying areas of success and those requiring further focus.

The rationale behind the creation of BRISQ remains pertinent. But in common with other comparable guidelines several barriers to its uptake have become evident and addressing these might improve uptake of BRISQ.

One approach would be to clarify the definitions and establish common data elements for existing categories, combined with providing tools to make it easier for researchers to address the relevant data elements recommended. An online BRISQ tool has been created and mounted in the Biobank Resource Centre.²⁸ This allows the user to enter the BRISQ criteria in the online tool, download and print out, or share the report. Taking this a step further, harmonizing the data elements to a globally agreed upon guidelines could generate recognized drop-down options that could facilitate use of the BRISQ criteria and an online tool.

Another approach could be revising the application of BRISQ with a simpler checklist approach as used by some other standards. This checklist version of BRISQ or “BRISQ 2.0” could build on and complement the foundations developed by existing guidelines that have addressed specific issues within the lifecycle of biospecimen use in research. SPREC, developed by the International Society for Biological and Environmental Repositories (ISBER) Biospecimen Science Working Group in 2009, and published in 2010,^18–20 has established a codified method of evaluating the preanalytical variables associated with biospecimen processing. It integrates with existing laboratory management systems, can be predefined in biospecimen processing protocols, and is flexible enough to allow for uncoded processing steps to be reported.

Similarly, REMARK^2–23 has established a checklist approach to tumor marker study reporting that incorporates a framework for reporting the clinical characteristics of study groups from which biospecimens are derived. The REMARK recommendations have had considerable uptake in the literature. The elements missing from SPREC and REMARK are those that are specific to biospecimen collection and storage, these reflect and parallel the traditional role of biobanking within translational research infrastructure.

Finally, a more flexible and researcher focused approach to implementing BRISQ reporting guidelines could be based on study type. The level of evidence being provided by a given study varies based on design and intent, but not necessarily analytical approach/technique, and using a system to define study types based on level of evidence they provide would facilitate implementation of a graduated system of extent of variable reporting: the extent of biospecimen variable reporting should reflect the rigor of study being performed. Biospecimen quality should be fit-for-purpose, and as the purpose of a given study can range from exploratory or discovery work, to clinical validation trials, the extent of data associated with the provenance of biospecimens should reflect this range.

Characteristics of a potential next generation of biospecimen reporting system (Table 2) can be summarized as including the following; a predefined checklist of required variables that reflects a balance between the reporting burden and the potential relative clinical impact of reproducibility of the study, and it should be simple for biobanks to fulfill data requests, for researchers to report, for reviewers to verify, and for journals to implement.

Table 2.

Characteristics of Ideal Biospecimen Reporting Schema

Predefined checklist of required variables	√
Simple for biobanks to fulfill data requests	√
Simple for researchers to report	√
Simple for reviewers to verify	√
Simple for journals to implement	√
Reporting burden reflects clinical impact	√

This system could provide specific requirements for researchers to follow based on study type/objective; have increasing reporting requirements with increasing clinical impact of research/decreasing distance to clinic. Such a system developed to reflect study type would allow journals publishing different types of research (i.e., basic discovery work vs. clinical trial validation studies) to require different levels of biospecimen reporting.

Defining study type would form the cornerstone of such an approach. As an example of such a system, we have defined three major study types: Exploratory, Descriptive, and Analytic, adapted from the work of Burkett²⁹ (Table 3). The reporting elements required for each study type might draw upon existing BRISQ, SPREC, and REMARK recommendations and tie together these overlapping efforts with a harmonized set of reporting elements. The minimum data element set would be limited for exploratory studies, intermediate for descriptive studies, and extended for analytic studies.

Table 3.

Study Type Definitions


Studies conducted when researcher lacks sufficient information about a subject to attempt more precise measurement of its characteristics. Exploring a field to determine or sharpen the definition of the relevant questions to be asked and the most meaningful way to ask them. That is, is the protein/gene/element of interest expressed/present in given cell, tissue, organ, etc. • Studies identifying a novel gene, genetic modification, protein, splice variant, SNP, protein interaction, etc. through a method that does not directly involve human biospecimens, and then confirming presence or absence of this element in a human biospecimen.	Studies seeking to characterize the distribution of some set of variables within a setting or population. Once concepts can be operationalized and measured as variables, the study has progressed from exploratory to descriptive level. That is, when/where/how/etc. is the element expressed, characterizing the expression pattern of a given element of interest. How can it be best measured? What is its variation between individuals, over time, etc. • Studies characterizing the expression levels of a given element of interest in human biospecimens without direct clinical application, that is, without proposing any observed differential expression to be predictive, prognostic, or diagnostic for clinical outcomes or management. This would include studies where human biospecimens are used to compare expression level, distribution, or modification of a gene/protein of interest between groups, such as between diseased and nondiseased patients, age distributions, sex distributions, etc. • Studies detailing a disease process or mechanism in animal models that use human biospecimens in support of their other observations.	Studies seeking to determine relationships, ultimately causal relationships, between variables. These studies can be experimental, interventional or observational. Requires valid and reliable measurement of variables before testing relationships between them. That is, is the protein/gene/element of interest associated with a disease state, developmental stage, treatment response, physiologic or pathologic state, etc. Is it correlated or associated with other proteins/genes/elements of interest, etc. • Studies characterizing the expression levels of a given element of interest, or panel of elements in human biospecimens AND associating these findings with the presence of a disease, predictive or prognostic or diagnostic biomarkers, a change in clinical management or clinical outcomes. That is, the authors propose their findings can directly impact clinical care in some way.

Note: The area under the curve (shaded area) represents the relative impact on clinical care and need for variable reporting for research within each of the three research study categories.

Despite the growing recognition of the potential impact of variability in biobanking processes on interpretation of analyses of biospecimens, adequate documentation of important preanalytical factors in research publications is not common. The BRISQ guideline remains a relevant and important reference point for variables that are likely to be important to report in most cases. Whatever strategy is adopted to improve biospecimen data reporting, development of biospecimen reporting guidelines cannot be divorced from the realities of the research laboratory, the funding environment, or the article review process. The content and processes involved in meeting requirements of guidelines must be straightforward for researchers and reviewers to consider.

Footnotes

Authors' Contributions

A.J.M. contributed to study design, literature review, data analysis, and article writing and review. D.S.-D. contributed to study design, literature review, and article writing and review. L.A.M. contributed to study design, literature review, and article writing and review. S.C. contributed to study design and literature review. P.H.W. contributed to study design, data analysis, and article writing and review. P.H.W. and L.A.M. gratefully acknowledge support for this work by the Biobanking and Biospecimen Research Program at BC Cancer (supported by the Provincial Health Services Authority), the Canadian Tissue Repository Network (funded by grants from the Institute of Cancer Research, Canadian Institutes of Health Research and the Terry Fox Research Institute, and from the Canadian Cancer Research Alliance), and the Office of Biobank Education and Research, University of British Columbia (supported by the Department of Pathology and Laboratory Medicine, University of British Columbia).

Author Disclosure Statement

No conflicting financial interests exist.

References

Baker

. 1,500 scientists lift the lid on reproducibility. Nature, 2016; 533:452–454.

Engber

. Cancer research is broken. Slate. 2016; Available from: www.slate.com/articles/health_and_science/future_tense/2016/04/biomedicine_facing_a_worse_replication_crisis_than_the_one_plaguing_psychology.html (accessed March 13, 2019).

Compton

. Getting to personalized cancer medicine: Taking out the garbage. Cancer, 2007; 110:1641–1643.

Repetitive flaws. Nature 2016;529:256.

Freedman

, Cockburn

, Simcoe

. The economics of reproducibility in preclinical research. PLoS Biol, 2015; 13:e1002165.

Moore

, Kelly

, Jewell

, et al. Biospecimen reporting for improved study quality. Biopreserv Biobank, 2011; 9:57–70.

Moore

, Kelly

, Jewell

, et al. Biospecimen reporting for improved study quality. Cancer Cytopathol, 2011; 119:92–101e.

Moore

, Kelly

, Jewell

, et al. Biospecimen reporting for improved study quality (BRISQ). J Proteome Res, 2011; 10:3429–3438.

Moore

, Kelly

, McShane

, Vaught

. Biospecimen reporting for improved study quality (BRISQ). Clin Chim Acta, 2012; 413:1305.

10.

Equator Network. BRISQ Guidelines [Internet]. [cited December 11, 2018]. Available from: https://www.equator-network.org/reporting-guidelines/brisq (accessed March 13, 2019).

11.

Simeon-Dubach

, Burt

, Hall

. Quality really matters: The need to improve specimen quality in biomedical research. J Pathol, 2012; 228:431–433.

12.

Simeon-Dubach

, Burt

, Hall

. Quality really matters: The need to improve specimen quality in biomedical research. Histopathology, 2012; 611003–1005.

13.

Nature Editorial. Announcement: Reducing our irreproducibility. Nature, 2013; 496:398.

14.

Simeon-Dubach

, Moore

. BIO comes into the cold to adopt BRISQ. Biopreserv Biobank, 2014; 12:223–224.

15.

Cheah

, Dee

, Cole

, Matzke

, O'Donoghue

, Watson

. An online tool for improving biospecimen data element reporting. Biopreserv Biobank, 2012; 10:501–510.

16.

Maccarthy

, Kirtley

, De Beyer

, Altman

, Simera

. Reporting guidelines for oncology research: Helping to maximise the impact of your research. Br J Cancer, 2018; 118:619–628.

17.

Sauerbrei

, Taube

, McShane

, Cavenagh

, Altman

. Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK): An abridged explanation and elaboration. J Natl Cancer Inst, 2018; 110:803–811.

18.

Betsou

, Lehmann

, Ashton

, et al. Standard preanalytical coding for biospecimens; Defining the sample PREanalytical code. Cancer Epidemiol Biomark Prev, 2010; 199:1004–1011.

19.

Lehmann

, Guadagni

, Moore

, et al. Standard preanalytical coding for biospecimens: Review and implementation of the Sample PREanalytical Code (SPREC). Biopreserv Biobank, 2012; 10:366–374.

20.

Nussbeck

, Benson

, Betsou

, Guadagni

, Lehmann

, Umbach

. Is there a protocol for using the SPREC?. Biopreserv Biobank, 2013; 11:260–266.

21.

McShane

, Altman

, Sauerbrei

, Taube

, Gion

, Clark

. Reporting recommendations for tumor marker prognostic studies (REMARK). J Natl Cancer Inst, 2005; 97:1180–1184.

22.

McShane

, Hayes

. Publication of tumor marker research results: The necessity for complete and transparent reporting. J Clin Oncol, 2012; 30:4223–4232.

23.

Altman

, McShane

, Sauerbrei

, Taube

. Reporting recommendations for tumor marker prognostic studies (REMARK): Explanation and elaboration. BMC Med, 2012; 10:1–39.

24.

Vaux

. Research methods: Know when your numbers are significant. Nature, 2012; 492:180–181.

25.

Baker

, Lidster

, Sottomayor

, Amor

. Two years later: Journals are not yet enforcing the ARRIVE guidelines on reporting standards for pre-clinical animal studies. PLoS Biol, 2014; 12:e100756.

26.

Begley

, Ioannidis

. Reproducibility in science: Improving the standard for basic and preclinical research. Circ Res, 2015; 116:116–126.

27.

Norlin

, Fransson

, Eriksson

, et al. A minimum data set for sharing biobank samples, information, and data: MIABIS. Biopreserv Biobank, 2012; 10:343–348.

28.

Biobank Resource Centre. Available from: https://www.biobanking.org (accessed March 13, 2019).

29.

Burkett

. Classifying basic research designs. Fam Med, 1990; 22:143–148.