Abstract
The Common Rule, revised extensively to enhance human subjects protections and to reduce burdens to investigators and institutional review boards (IRBs), was implemented on January 19, 2019. We analyzed IRB performance metric data from 2016 through 2021 to evaluate the potential impact of the revised Common Rule on the quality and performance of IRBs. From 2016 to 2021, exempt protocols increased by 159% and protocols requiring IRB continuing reviews decreased by 28%. As only 48% of all protocols in 2021 were subjected to the revised Common Rule requirements, numbers of exempt protocols and protocols requiring IRB continuing reviews will continue to increase and decrease, respectively, in the next few years. Among a total of 16 IRB performance metrics studied, 4 improved, 4 deteriorated, and 8 remained unchanged from 2016 through 2021. This study represents the first effort to evaluate the impact of revised Common rule on IRB quality and performance.
Keywords
Introduction
The institutional review board (IRB) system in the United States was first established in 1974 under the National Research Act (1974) for the primary purpose of protecting human subjects participating in research. With the publication of the Belmont Report in 1979, defining the ethical principles of respect for persons, beneficence, and justice for human research, the Department of Health and Human Services and the Food and Drug Administration revised their existing human subjects regulations in 1981 using the Belmont Report as the foundational background (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979; Office for Human Research Protections, 2016). In 1991, Subpart A of the Department of Health and Human Services Federal Policy for the Protection of Human Subjects (45 Code of Federal Regulations (CFR) 46 Subpart A), which outlined the basic provisions for IRB, informed consent form, and assurance of compliance, became the Common Rule. The Common Rule was subsequently revised extensively, adopted by 20 U.S. Departments and Agencies, and implemented effective January 19, 2019 (Menikoff, Kaneshiro, & Pritchard, 2017; U.S. Department of Health and Human Services, 2018). For the purpose of this report, the revised Common Rule will be designated as the 2018 Common Rule or 2018 Requirements, and the pre-revised version will be designated as the pre-2018 Common Rule or pre-2018 Requirements.
The 2018 Requirements apply to human research activities that are exempt or required to be approved by an IRB on or after January 19, 2019, and those that have been approved by an IRB prior to January 19, 2019, but are determined and documented by an IRB to be transitioned to the 2018 Requirements. Human research activities that are exempt or approved by an IRB prior to January 19, 2019, will continue to be subjected to pre-2018 Requirements, unless they are transitioned to the 2018 Requirements as described above (U.S. Department of Health and Human Services, 2018). Thus, there will be a period after January 19, 2019, during which some human research activities are subjected to the 2018 Requirements, while others are subjected to the pre-2018 Requirements.
The 2018 revision of the Common Rule was intended to modernize “the regulations by enhancing protections for human research subjects and reducing unnecessary burden and ambiguity for researchers” (Menikoff et al., 2017, P. 613). To accomplish these objectives, the revisions included, but were not limited to, the following:
Improving informed consent by:
Giving prospective participants information a reasonable person would want to know to make a decision whether to participate in research, Providing sufficient detail, Reorganizing consent form to facilitate understanding, and Presenting concise and focused key information at the beginning. Reducing burdens of low-risk studies to allow IRBs and researchers to focus on high-risk studies by:
Increasing exemptions for low-risk studies, Reducing IRB continuing reviews, Allowing new option for screening of potential participants, and Using “broad consent” for secondary research. Requiring the use of single-IRB review for multi-institutional studies in the U.S., eliminating the time and effort associated with multiple IRB reviews and the need for reconciling different IRB determinations. (Menikoff et al., 2017; U.S. Department of Health and Human Services, 2018).
With the implementation of the revised Common Rule, it is reasonable to expect a profound impact on the quality and performance of IRBs.
The Department of Veterans Affairs (VA) health care system is the largest integrated health care system in the U.S. with more than 100 facilities conducting research involving human subjects. Since 2010, as part of its quality assurance program, VA has collected yearly performance metric data on its human research protection programs, for which IRB is a major component (Tsan & Nguyen, 2017; Tsan, Smith, & Gao, 2010). In the current study, we analyzed IRB performance metric data from 2016 through 2021 as an initial effort to evaluate the impact of the revised Common Rule on the quality and performance of IRBs.
Methods
Data Collection
Collection of VA human research protection program performance metric data was carried out as described previously (Tsan & Nguyen, 2017; Tsan, Nguyen & Brooks, 2013). Briefly, as part of the VA quality assurance program, VA research facilities were required to conduct audits of all informed consent documents annually and regulatory audits of all human research protocols approximately once every three years by research compliance officers, who reported directly to institutional officials, i.e., medical center directors, and functioned independently of research offices (U.S. Department of Veterans Affairs, 2020). For protocols that had been active for more than three years, protocol regulatory audits were limited to the last three years of the protocols. Approximately one third of all active human research protocols were audited each year. Audit tools were developed for the annual informed consent document as well as the triennial protocol regulatory audits. Facility research compliance officers were then trained to use these tools to conduct audits throughout the year. Results of the audits conducted between June 1 and May 31 of each year were collected through a web-based system from all VA research facilities (Tsan & Nguyen, 2017).
A total of 25 human research protection program performance metrics were collected, of which 16 metrics were related to IRB performance (Tsan & Nguyen, 2017). Information collected in these 16 IRB performance metrics (see Tables 2–5) included metrics assessing compliance with informed consent document and Health Insurance Portability and Accountability Act authorization requirements; compliance with requirements for IRB initial approval and continuing review of human research protocols; compliance with selected informed consent requirements; for-cause suspension or termination of human research protocols; research-related serious adverse events; and subject enrollment according to inclusion and exclusion criteria.
Informed Consent Document and Health Insurance Portability and Accountability Act Authorization.
Determined using analysis of ordered categories for the trend of changes from 2016 through 2021.
Percent change from 2016 to 2021.
Starting 2021, protocols that have had at least one prior informed consent document audit and are open only for long term follow up or data analysis, no longer require annual informed consent document audit. This accounted for the decrease in the number of protocols audited in 2021.
Abbreviations used: ICD, informed consent document; HIPAA, Health Insurance Portability and Accountability Act; LAR, legally authorized representative.
The numbers in parentheses were the numbers of informed consent documents per protocol audited.
The numbers in parentheses were the percentages of the total number of protocols audited or HIPAA authorization required.
Institutional Review Board Initial and Continuing Reviews.
Determined using analysis of ordered categories for the trend of changes from 2010 through 2016.
Percent change from 2016 to 2021.
Abbreviations used: IRB, institutional review board.
The numbers in parentheses were the percentages of the total number of protocols audited or requiring IRB continuing reviews.
N/A denotes not applicable.
Protocol Suspension and Serious Adverse Events.
Determined using analysis of ordered categories for the trend of changes from 2010 through 2016.
Percent change from 2016 to 2020.
The numbers in parentheses were the percentages of the total number of protocols audited.
Data not collected.
N/A denotes not applicable.
Informed Consent Requirement and Inclusion/Exclusion Criteria.
Determined using analysis of ordered categories for the trend of changes from 2016 through 2021.
Percent change from 2016 to 2021.
The numbers in parentheses were the percentages of the total number of case histories reviewed.
The numbers in parentheses were the percentages of the number of case histories reviewed with inclusion/exclusion criteria.
Data of these 16 IRB performance metrics collected from 2016 to 2021 were analyzed to evaluate the potential impact of the revised Common Rule on IRB quality and performance. Data from 2016–2018 were collected prior to the implementation of the revised Common Rule, while 2020 and 2021 data were collected after its implementation. Twenty nineteen (2019) was a transitional year. It contained 7 months (from June 1, 2018, to January 18, 2019) of pre-implementation data and 5 months (from January 19, 2019, to May 31, 2019) of post-implementation data.
Protection of Human Subjects
As this was a quality assurance project, it did not involve human subjects and no individually identifiable information was collected. Therefore, no IRB review and approval was required (Tsan & Puglisi, 2014).
Data Analysis
All data collected were entered into a computerized database for analysis. We used the analysis of ordinal categorical data to determine the trend of changes from 2016 through 2021 (Agresti, 1984). This was performed using JavaStat ordinal contingency table analysis available at www.statpages.info. A p value of < 0.05 was considered to be statistically significant. For those performance metrics with statistically significant changes, we also calculated percent changes from 2016 through 2021 using the following formula: Percent change = [(rate in 2016 – rate in 2021) ÷ rate in 2016] x 100 (Tsan & Nguyen, 2017).
Results
Number of Exempt Protocols and Continuing Reviews
The revised Common Rule expanded the exempt categories from 6 to 8 by adding 2 new categories, i.e., storage of identifiable private information or identifiable biospecimens for secondary research for which broad consent is required (Category 7), and secondary research using identifiable private information or identifiable biospecimens for which broad consent is required (Category 8) (U.S. Department of Health and Human Services, 2018).
Prior to 2018, the Common Rule required IRBs to conduct continuing review of ongoing research at intervals appropriate to the degree of risk, but not less than annually. The revised Common Rule made two changes affecting the IRB continuing review requirements: (1) the expansion of exempt research categories as described above, and (2) the removal of the annual IRB continuing review requirements for studies that undergo expedited review and for studies that have completed study interventions and are merely analyzing study data or involve only observational follow up in conjunction with standard clinical care (Menikoff et al., 2017; U.S. Department of Health and Human Services, 2018).
The above changes in 2018 Requirements should markedly affect the numbers of exempt protocols and the number of protocols requiring IRB continuing reviews. However, the extent of these changes is not clear.
Table 1 shows data collected from 2016 through 2021 on numbers of facilities; total numbers of active protocols each year, numbers and rates of exempt protocols, numbers of protocols audited, numbers and rates of protocols subjected to 2018 Requirements, and numbers and rates of protocols requiring IRB continuing reviews.
Impact of Revised Common Rule on Exempt Protocols and IRB Continuing Reviews.
Determined using analysis of ordered categories for the trend of changes from 2016 through 2021.
Percent change from 2016 to 2021.
The numbers in parentheses were the percentages of the total number of active protocols or protocols audited.
Data not collected.
Abbreviation used: IRB, institutional review board.
As shown in Table 1, the number of exempt protocols increased from approximately 6% of the total number of active protocols in 2016–18 to almost 16% (i.e., 15.84%) in 2021, an increase of 159%. In contrast, the number of protocols requiring IRB continuing reviews decreased from 83.2% in 2016 to 60.2% in 2021, a reduction of 28%. In 2020, only 28.70% of all active protocols were subjected to 2018 Requirements. This number increased to 48.02% in 2021.
Informed Consent Documents and Health Insurance Portability and Accountability Act Authorizations
Table 2 shows data collected from 2016 through 2021 on numbers of informed consent documents audited each year; numbers and rates of informed consent documents that were missing; numbers and rates of incorrect informed consent documents used as well as informed consent documents that were not signed and dated by subjects; numbers of Health Insurance Portability and Accountability Act Authorization required; and numbers and rates of these authorizations that were not obtained.
All 3 performance metrics related to informed consent document requirements showed statistically significant deterioration from 2016 through 2021, ranging from 52% (from 1.25% in 2016 to 1.84% in 2021 for rates of incorrect informed consent documents used) to 1,300% (from 0.02% in 2016 to 0.28% in 2021 for rates of informed consent documents not signed and dated by subjects). Similarly, compliance with Health Insurance Portability and Accountability Act authorization requirement also deteriorated significantly from 0.56% of required authorizations being not obtained in 2016 to 1.43% in 2021, a deterioration of 155%.
Institutional Review Board Initial and Continuing Reviews
Table 3 shows data from 2016 through 2021 on numbers and rates of protocols conducted and completed without IRB approval; numbers and rates of protocols initiated prior to IRB approval; numbers and rates of protocols requiring IRB continuing reviews, and numbers and rates of protocols lapsed in required IRB continuing reviews as well as those protocols that investigators continued research activities during lapse in required IRB continuing reviews.
The 2 performance metrics related to IRB initial reviews did not show any statistical significant difference from 2016 through 2021. Thus, there was no improvement or deterioration. For the IRB continuing reviews, despite a significant decrease in the numbers of protocols requiring IRB continuing reviews, i.e., a 28% decrease from 2016 to 2021, lapse rates and rates of investigators who continued research activities during the lapse remained unchanged from 2016 through 2021.
Protocol Suspensions and Serious Adverse Events
Table 4 shows data from 2016 through 2020 on numbers and rates of protocols suspended or terminated by IRB, those that were suspended due to human subject protection concerns or due to investigator-related concerns; numbers and rates of local adverse events that were determined to be serious, unanticipated, and related or probably related to research; as well as those serious local adverse events resulted in hospitalization or death. With the exception of the number and rate of local adverse events that were determined to be serious, unanticipated, and related or probably related to research, other data were not available in 2021 due to administrative reasons.
The rate of protocols suspended or terminated by IRB showed a statistical significant decrease from 0.24% in 2016 to 0.10% in 2020, an improvement of 58%. This was primarily due to an improvement in the protocol suspension due to investigator-related concerns, i.e., an improvement of 76% from 2016 (0.21%) to 2020 (0.05%). None of the performance metrics related to local adverse events that were determined to be serious, unanticipated, and related or probably related to research showed any statistical significant differences from 2016–2020 or 2021.
Informed Consent Requirement and Inclusion/Exclusion Criteria
Review of Case histories was performed to determine whether investigators obtained informed consent prior to enrolling subjects in the study and whether subjects not meeting inclusion/exclusion criteria were enrolled into the research. Table 5 shows data obtained from 2016 through 2021 on numbers and rates of informed consent not obtained prior to the initiation of research; and numbers and rates of subjects not meeting inclusion/exclusion criteria that were included in research.
There was a statistical significant difference in the rates of informed consent not obtained prior to initiation of studies from 216 through 2021, i.e., 0.04% in 2016 and 0.02% in 2021, a 50% improvement. Similarly, there was a statistical significant difference in the rates of subject enrolled in studies not meeting inclusion/exclusion criteria from 2016 through 2021, i.e., 0.08% in 2016 and 0.07% in 2021, a 13% improvement.
Discussion
The results presented in this report demonstrated that since implementation of the revised Common Rule on January 19, 2019, there had been an increase of 159% in the number of exempt protocols and a reduction of 28% in the number of protocols requiring IRB continuing reviews from 2016 to 2021. As only 48.02% of all active protocols in 2021 were subjected to 2018 Requirements, the number of exempt protocols is expected to continue to increase and the number of protocols requiring IRB continuing reviews is expected to continue to decrease over the next few years. Among a total of 16 IRB performance metrics studied, 4 improved, 4 deteriorated, and 8 remained unchanged from 2016 through 2021. This study, while limited in scope, represents the first effort to evaluate the impact of revised Common Rule on the quality and performance of IRBs.
Increasing the number of exempt protocols and reducing the number of protocols requiring IRB continuing reviews is one of the stated purposes for revising the Common Rule, namely, to reduce unnecessary burdens of low-risk studies, allowing investigators and IRBs to focus on high-risk research (Menikoff et al., 2017). Indeed, in 2021, the third year after the implementation of the revised Common Rule, there was a 159% increase in the number of exempt protocols and a 28% reduction in the number of protocols requiring IRB continuing reviews. However, there was no improvement in IRB continuing review lapse rates. The reason for this is not clear. We have previously demonstrated that implementation of effective remedial actions can result in lasting improvement in lapse in IRB continuing reviews (Tsan & Nguyen, 2019). However, this has not lead to improvement in overall IRB continuing review lapse rate. It is possible that a lapse in IRB continuing review with the resultant interruption of research activities for a few months does not significantly affect investigators’ research. Further study is necessary to answer this question.
In this study, we also noted that of the 16 IRB performance metrics that we monitored each year from 2016–2021, 8 were unchanged, 4 improved, and 4 deteriorated. The 4 performance metrics that deteriorated were all related to informed consent documents and Health Insurance Portability and Accountability Act Authorization requirements, ranging from 52% to 1,300% deterioration from 2016 to 2021. This was quite unexpected. In a previous study, we observed that from 2010 to 2016, these same 16 IRB performance metrics showed improvement in 12, unchanged in 4, and none deteriorated (Tsan & Nguyen, 2017). A number of factors could potentially contribute to this undesirable observation. First, it was a lot of work for the investigators and IRBs to implement all the changes made in the revised Common Rule, including extensive changes in informed consent requirements. Second, from January 19, 2019 to May 31, 2021, IRBs and investigators had to follow two set of rules, pre-2018 Requirements for some studies and 2018-Requirements for other studies. This could be quite confusing for all parties involved. Third, from October 2020 through September 2021, the VA Office of Research and Development implemented a new VA Innovation and Research Review System (VAIRRS), providing an enterprise platform to support management of research oversight committees, including IRBs. This could potentially affect IRB performance during the implementation period. Forth, the coronavirus disease 2019 (covid-19) pandemic in the U.S. also caused considerable disruption in 2020 and 2021. However, missing informed consent documents, using incorrect informed consent documents, informed consent documents not signed and dated by subjects, and not obtaining required Health Insurance Portability and Accountability Act authorization are basic deficiencies that should not have been affected by changes in 2018-Requirements, implementation of new IRB management system, or covid-19 pandemic. Facility IRBs and research compliance officers should develop remedial action plans, including education and training, to address these deficiencies.
These 16 IRB performance metrics were originally developed as part of the VA human research protection program quality indicators. We stated at that time that, “Although it is difficult to directly measure the extent to which human subjects are protected from research-related harms, we believe it is possible to identify indicators that can be used to assess the quality of an institution's human research protection program, which includes attention to factors that are likely to or may lead to research-related harm” (Tsan et al., 2010, P. 16). This statement was true in 2010, and remains true today. Nonetheless, in the last 10 years, we have learned that we can do much more to measure the quality and performance of IRBs. In 2013, Lidz et al. (2012) assessed how well IRBs followed the Common Rule protocol approval criteria, by measuring whether IRBs ensured protocols had satisfied all eight Common Rule approval criteria, thus, satisfying the three ethical principals of the Belmont Report, before making a decision whether to approve the research. They observed that of the 104 protocols reviewed by IRBs from 10 U.S. major academic institutions, only 20% had satisfied all 8 Common Rule approval criteria. In 2020, Tsan et al. (2020) reported that of the 60 protocols that had been deemed to be exempt by 10 VA IRBs, 10 required IRB reviews and approval, i.e., 4 requiring full board reviews and 6 requiring expedited reviews, according to the Office for Human Research Protections (OHRP) human subject regulations decision charts. These results strongly suggest that the quality and performance of the U.S. IRBs are in need of improvement (Tsan, 2021).
The 2018 Common Rule has made extensive revisions with the goal of enhancing human subjects protections while reducing IRBs and investigators’ burdens allowing them to concentrate on high-risk studies. Thus, in addition to evaluating the effectiveness of the revised Common Rule, such as whether it has achieved its goal of improving potential participants’ understanding of the informed consent, thereby facilitating their decisions on whether to participate in the research, it is important to also evaluate the impact of the revised Common Rule on the quality and performance of IRBs, such as we reported here. Further studies should include the revised Common Rule's impact on the quality of IRB reviews and the quality of IRB decisions.
In 2007, Taylor (2007) proposed to develop a tool to measure the quality of IRB reviews without defining what constitutes the quality of IRB reviews. More than a decade later, no such tool has been developed and no study measuring the quality of IRB review has ever been published. Similarly, Rosenfeld and Gerhart (2016) proposed that IRBs should be judged by how well and consistently IRBs make decisions. What constitutes the quality of IRB decisions has not been defined and no study measuring the quality of IRB decisions has ever been published.
We have previously defined IRB quality as “how well IRBs implement the Common Rule” (Tsan, 2021, P. 479). Based on this definition, a quality IRB review can be defined as:
A review that is performed by a duly constituted IRB with all necessary expertise and free of conflict of interest for the protocol being reviewed, and The IRB has systematically assessed each of the eight Common Rule approval criteria and determines whether each criterion has been satisfied or the IRB stipulation of what changes are necessary to ensure meeting the three ethical principals of the Belmont Report, namely respect for persons, beneficence, and justice.
A decision made after a quality IRB review as described above is a quality IRB decision.
With the above definition of quality IRB reviews, non-quality IRB reviews can be categorized into three groups: non-quality IRB reviews due to deficiency in the composition of IRB membership such as lack of necessary expertise or with conflict of interest for the protocols being reviewed; non-quality IRB reviews due to failure to ensure that all eight Common Rule approval criteria have been satisfied; and non-quality IRB reviews due to deficiencies in IRB membership composition as well as inadequate IRB reviews.
Best Practices
Based on the results of this study, we recommend that institutions routinely collect IRB performance metric data, so that institutions know how well their IRBs and their research programs are performing.
Research Agenda
Future research should be conducted to determine:
The impact of the revised Common Rule on exempt protocols, namely, the extent of non-exempt human subjects research that is carried out intentionally or unintentionally as exempt protocols without prior IRB approval before and after the implementation of the Revised Common Rule, as described in details previously in a recent publication (Tsan et al. 2020). The impact of the revised Common Rule on the quality of IRB reviews. Specifically, we suggest the following studies from a representative sample of IRBs:
From a sample of full board reviewed protocols (both before and after the implementation of revised Common Rule), review the meeting minutes as well as the recordings of the meetings to evaluate whether the IRB membership composition was appropriate for the protocol being reviewed and whether the IRB systematically assessed all eight Common Rule approval criteria prior to rendering a determination on the disposition of the protocol. Based on the findings of the above, determine whether the review was a quality IRB review or a non-quality IRB review, and if it was a non-quality IRB review, whether it was due to inadequate IRB membership composition, inadequate review, or both. Compare pre- and post-revised Common Rule results to determine the impact of the revised Common Rule.
Educational Implications
Investigators and designated individuals who are authorized to make determinations on exempt status should be educated and trained on the various exempt categories and how to determine whether a study is qualified to be exempt using Office for Human Research Protections Human Subject Regulations Decision Charts (U.S. Department of Health and Human Services, 2016, 2020).
Investigators and IRB members should also be educated on the criteria for quality IRB reviews, including the required eight Common Rule approval criteria that satisfy the three ethical principles of the Belmont Report.
Footnotes
Acknowledgement
The authors thank John Thomas Puglisi, Ph.D., for his critical review of the manuscript, and Yen B. Nguyen, Pharm.D., and all VA research compliance officers for their contributions in conducting the audits and collecting the data presented in this report.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
