Improving the Quality and Performance of Institutional Review Boards in the U.S.A. Through Performance Measurements

Abstract

Performance measurement leads to quality improvement, because performance measurement can identify areas of vulnerability to guide quality improvement activities. Recommendations from empirical institutional review board (IRB) performance measurement data on research approval criteria, expedited review protocols, exempt protocols, and IRB continuing review requirements published over the past 10 years are reviewed here to improve the quality and efficiency of IRBs. Implementation of these recommendations should result in improvements that can be evaluated by follow-up performance measurements.

Keywords

IRB quality IRB performance improving IRB quality and performance performance measurement

The institutional review board (IRB) or its equivalent, the research ethics committee or ethics review board, is a key component of institutional human research protection programs and plays a pivotal role in protecting the rights and welfare of human subjects participating in research (Institute of Medicine, 2001, 2002). In the United States, the role of IRBs is described in detail in the Federal Policy for the Protection of Human Subjects, also known as the Common Rule, which was adopted by 20 U.S. Departments and Agencies (U.S. Department of Health and Human Services, 2018). Technically, the Common Rule applies only to federally funded research. However, in practice most U.S. institutions adopt it as an overarching regulatory/ethical framework. Thus, the quality of an U.S. IRB can be defined by how well it implements the Common Rule (Tsan, 2019a).

Performance measurement is a well-established tool for the improvement not only of health care, but also of human research protection programs (Cassel et al., 2014; Tsan & Nguyen, 2017). Because of its importance, there has also been considerable interest in measuring the quality and performance of IRBs. While human subject protection is not a parameter for the quality of IRBs, it is reasonable to expect that high-quality IRBs are likely to exercise better human subject protections (Tsan, 2019a, 2019b). Here I summarize existing IRB performance measurement data and recommend how the quality and efficiency of IRBs can be improved.

Research Approval Criteria

The Common Rule requires that no research involving human subjects, unless deemed to be exempt from IRB review, can be initiated until it has been reviewed and approved by an IRB. It further requires that in order to approve research, the IRB shall determine that all of the following requirements are satisfied:

risks to subjects are minimized,

risks to subjects are reasonable in relation to anticipated benefits, if any, to subjects, and the importance of the knowledge that may reasonably be expected to result,

selection of subjects is equitable,

informed consent will be sought from each prospective subject or the subject's legally authorized representative,

informed consent will be appropriately documented or appropriately waived,

when appropriate, the research plan makes adequate provision for monitoring the data collected to ensure the safety of subjects,

when appropriate, there are adequate provisions to protect the privacy of subjects and to maintain the confidentiality of data, and

when some or all of the subjects are likely to be vulnerable to coercion or undue influence, such as children, prisoners, individuals with impaired decision-making capacity, or economically or educationally disadvantaged persons, additional safeguards have been included in the study to protect the rights and welfare of these subjects (U.S. Department of Health and Human Services, 2018).

Thus, the Common Rule mandates that US IRBs conduct in-depth ethics review of proposed research and approve research only when all of the above criteria are determined to be satisfied, meeting the three basic ethical principles of the Belmont Report, namely, respect for persons, beneficence, and justice (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979; U.S. Department of Health and Human Services, 2018).

In a study evaluating how closely IRBs followed the Common Rule approval criteria when reviewing research protocols, Lidz et al. (2012) analyzed audio-recorded IRB meetings of 104 protocols from 20 IRBs at 10 major U.S. academic institutions between 2006 and 2009. They observed that in their review of protocols that required IRB discussion because the principal investigators did not adequately satisfy all required approval criteria, IRBs mostly discussed the informed consent documents (102/104 or 98% of the time), but failed to address risk minimization (17/82 or 21% of the time), the risk-to-benefit ratio (52/91 or 57%), equitable subject selection (31/52 or 60%), data monitoring (32/59 or 54%), privacy and confidentiality (13/52 or 25%), and protection of vulnerable populations (7/55 or 13%). These IRBs made clear determinations that all eight Common Rule approval criteria were satisfied in only 20 of the 104 (20%) protocols (Lidz et al., 2012).

While the above finding was based on the authors’ assumption that the absence of specific IRB discussion areas indicated that the IRB had not examined those areas closely (rather than because the protocol was satisfactory in those areas, warranting no discussion), the above observation is striking. To improve the quality of IRB reviews, and to avoid the above assumption about “silent” areas, I recommend that in their review of protocols, IRBs systematically address each of the eight approval criteria and only approve the protocol when all these criteria are satisfied.

Specifically, for full-board review protocols:

In primary reviewer's reports, each primary reviewer should include an assessment of each approval criterion, whether the criterion has been satisfied and if not, what suggested changes are proposed. The use of a simple checklist is not adequate, unless the reviewers include their assessments of each approval criterion as described above.

At the IRB meeting, the chair should lead the IRB to systematically assess each approval criterion, and discuss and determine whether the criterion is satisfied or the IRB stipulation of what changes are necessary.

The IRB meeting minutes should document IRB's determinations on each approval criterion.

For expedited review protocols:

The reviewer, either the chair or the designated IRB member, should document a systematic assessment of each approval criterion and whether the criterion is satisfied or needs modification.

It should be pointed out that the above recommendation is not to limit the IRB review of research to the eight Common Rule approval criteria. Other ethical issues such as conflict of interest, community involvement, etc., should also be adequately considered (Emanuel et al., 2008).

Expedited Review Protocols

Research activities that present no more than minimal risk to human subjects and involve only procedures listed in one or more of the Common Rule's expedited review categories may be reviewed by the IRB using the expedited review procedure. Under this procedure, the review may be carried out by the IRB chair or by one or more experienced reviewers designated by the chair from among IRB members. In reviewing the research, reviewers may exercise all of the authority of the IRB except that the reviewers may not disapprove the research (U.S. Department of Health and Human Services, 2018). Thus, under the Common Rule, the IRB chair may decide whether to use the expedited review procedure or recommend full board review at a convened meeting to review those protocols that qualify for expedited review.

In a study evaluating how well IRBs followed Common Rule criteria for levels of initial protocol review, Tsan et al. (2020) reviewed and analyzed 140 protocols that had been approved by full-board IRBs at convened meetings from nine Department of Veterans Affairs (VA) facilities and the VA Central IRB in 2010 and 2011. Of these 140 protocols, only 71 protocols (50.7%) required actual full board review at a convened meeting according to the Common Rule as determined using the Office for Human Research Protections (OHRP) Human Subject Regulations Decision Charts (U.S. Department of Health and Human Services, 2016), as they were neither eligible for expedited review nor qualified for exemption. However, 66 protocols (47.1%) were eligible for expedited review, but were subjected to full board review. In addition, two of the 10 IRBs did not conduct expedited review of any protocols as a matter of local institutional policy.

The expedited review procedure has the following advantages:

Protocols can be reviewed at any time after submission, instead of waiting until the next regularly scheduled IRB meeting date.

Either the chair or an experienced IRB member designated by the chair can review the protocol, instead of the full board members at a convened meeting. Expedited review is thus much less labor intensive and more cost-effective.

It has been shown that from submission to final approval, on average expedited review is 44 days faster than full board review (Varley et al., 2016).

Under the revised Common Rule, the annual IRB continuing review requirement of ongoing research for studies that undergo expedited review is no longer required (U.S. Department of Health and Human Services, 2018).

Thus, while it is permissible to use full board, convened meetings to review protocols that are eligible for expedited review, this is problematic because:

it takes much longer to review and approve a protocol, and thus, delays research;

it is not cost-effective; it unnecessarily consumes IRB time and resources, and diverts IRB energy and attention from more deserving full board review protocols.

Based on the above, I recommend the following:

Investigators, IRB chairs, and designated experienced IRB members, who are authorized to review expedited review protocols, should be educated and trained on the various expedited review categories and how to determine whether a study is eligible for review by the expedited review procedure using criteria specified in OHRP Human Subject Regulations Decision Charts (U.S. Department of Health and Human Services, 2020a). This will enable investigators to know when to request expedited review for their protocols, and IRB chairs and/or designated IRB members to be sure whether protocols under their review are in fact eligible for expedited review.

The IRB office should prescreen all submitted protocols for research that involve no more than minimal risk and determine whether they meet criteria for expedited review. This will ensure that the IRB office is able to identify all protocols that are eligible for expedited review.

IRBs should utilize the expedited review procedure to the greatest extent permissible by the regulations, so that IRBs take full advantage of the expedited review procedure.

Exempt Protocols

Research activities in which the only involvement of human subjects is in one or more of the Common Rule exempt categories are exempt from IRB review. Prior to 2018, there were six exempt categories. The revision in 2018 expanded the number of exempt categories from six to eight (Menikoff et al., 2017; U.S. Department of Health and Human Services, 2018). These changes included:

adding some restrictions on Category 1. Educational practices;

expanding Category 2. Educational tests, surveys, interviews, and observation of public behavior,

replacing Category 3. Research on public officials, with research involving benign behavioral interventions,

expanding Category 4. Research on existing data, to secondary research that does not require consent,

expanding Category 5. Public benefit or service programs,

no change on Category 6. Taste and food evaluation,

adding new Category 7. Storage of identifiable private information or identifiable biospecimens for secondary research for which broad consent is required,

adding new Category 8. Secondary research using identifiable private information or identifiable biospecimens for which broad consent is required, and

some research activities listed in Exempt Categories 2 and 3, and all research activities listed in Categories 7 and 8 require limited IRB review (Anderson, 2018).

The Common Rule does not define who should make the exempt determination. Because of the potential for conflict of interest, OHRP recommends that investigators not be given the authority to make an independent determination that human subjects research is exempt and that institutions should implement exemption policies that most effectively address the local setting and programs of research (U.S. Department of Health and Human Services, 2020b). At the Department of Veterans Affairs, VA policies require that the exempt status be determined by the IRB chair, an experienced IRB member, or qualified administrative staff with expertise in applying human research exempt regulations (U.S. Department of Veteran Affairs, 2019).

Ample examples exist in the literature of research that was conducted under the pretense of exempt protocols, but was later found to require IRB approval (Cooper & McNair, 2020). Conducting nonexempt research without prior IRB approval constitutes serious noncompliance that should be reported to OHRP (U.S. Department of Health and Human Services, 2011). Neither the Common Rule nor OHRP provides any direction regarding how data collected from such noncompliant research should be handled. However, investigators of such research often experience difficulties in publishing the data, because most institutions, as a matter of institutional policy, prohibit data obtained through such research from being published or used in degree dissertations (Tsan, 2020). Some even require such data to be destroyed (Cooper & McNair, 2020). In addition, most, if not all, reputable journals increasingly require the author's assurance that research has been conducted with prior IRB approval as a condition for publication (Rowan-Legg et al., 2009).

In the above-cited study evaluating how well IRBs follow Common Rule criteria for levels of initial protocol review, Tsan et al. (2020) reviewed and analyzed 60 exempted protocols from nine VA research facilities and the VA Central IRB in 2010 and 2011. They observed that of these 60 protocols, 10 (16.7%) protocols were found to require IRB review according to the Common Rule as determined using OHRP Human Subject Regulations Decision Charts (U.S. Department of Health and Human Services, 2016). Specifically, six protocols required expedited review and four protocols required full board review.

This finding is striking because prior to 2018, VA policies required that protocol exempt status be determined by the IRB chair or an experienced IRB member, and that the applicable exempt category/categories be documented (U.S. Department of Veteran Affairs, 2019). Conducting nonexempt research without prior IRB approval constitutes serious noncompliance and there are few options available to remedy the noncompliance. Prevention remains the best strategy to ensure that no nonexempt research is initiated prior to IRB approval (Tsan, 2020). I therefore recommend the following:

Non-VA institutions should designate qualified individuals such as IRB chairs, experienced IRB members, or other experienced personnel, but not the investigators, to determine the exempt status of a research.

Designated individual(s) who are authorized to make determination on exempt status and investigators should be educated and trained on the various exempt categories and how to determine whether a study is qualified to be exempt using OHRP Human Subject Regulations Decision Charts (U.S. Department of Health and Human Services, 2020a).

Institutions and/or IRBs should regularly audit exempted protocols to ensure that no non-exempt research is initiated prior to IRB approval. This will also help empirically identify grey areas where making the exempt/non-exempt distinction is difficult.

Institutions and IRBs should conduct research to determine the impact of the revised Common Rule on the quality and performance of IRBs, especially its impact on exempt research.

Continuing Reviews

Prior to 2018, the Common Rule required IRBs to conduct continuing review of ongoing research at intervals appropriate to the degree of risk, but not less than annually. Two changes in the revised Common Rule should markedly reduce the number of protocols requiring IRB continuing review: (1) the expansion of exempt research categories as described above and (2) the removal of the annual IRB continuing review requirements for studies that undergo expedited review and for studies that have completed study interventions and are merely analyzing study data or involve only observational follow up in conjunction with standard clinical care (Menikoff et al., 2017; U.S. Department of Health and Human Services, 2018).

Continuing review is one way by which IRBs ensure protection of human subjects. When continuing review and approval do not occur prior to the approval expiration date, all research activities involving human subjects must stop, unless the IRB determines that it is in the best interests of subjects who are already enrolled in the study, to continue participating in the research (U.S. Department of Health and Human Services, 2010). Stoppage may also cause costly disruption of the research.

Since 2010, VA has collected data on 25 human research protection program performance metrics including annual IRB continuing reviews as part of the VA quality assurance program (Tsan, 2017). For example, a review and analysis of 3,558 human research protocols from 107 VA research facilities in 2011 revealed that 2,942 (82.7%) protocols required IRB continuing review, of which 208 (7.07%) protocols had lapsed and six (0.20%) protocols continued research activities during the lapse. This IRB continuing review lapse rate was the highest type of noncompliance of 25 performance metrics measured (Tsan & Nguyen, 2015, 2017).

Subsequent studies revealed that the type of IRB used, namely VA IRB or affiliate university IRB, or the sizes of human research programs did not correlate with IRB continuing review lapse rates. While approximately 60% of facilities with protocols requiring continuing reviews had no lapses, approximately 20% of facilities had annual lapse rates exceeding 10% annually (Tsan & Nguyen, 2015).

In 2013, 10 facilities with IRB continuing review lapse rates that were higher than the VA national average for 3 consecutive years from 2011 to 2013 were asked to develop and implement remedial action plans to improve their continuing review lapse rates. Eight of these 10 facilities showed markedly improved (lower) lapse rates after implementing remedial action plans (28.5% ± 13.9% in 2013 vs. 6.7% ± 8.6% in 2014, p = .0017). The other two facilities failed to fully implement their remedial action plans in 2013 and showed no improvement in 2014 (15.8% and 48.4% in 2013 vs. 47.1% and 55.6% in 2014) (Nguyen et al., 2016).

Using data from 2011 through 2018, it was further demonstrated that 70% of these facilities’ lapse rates significantly improved. In contrast, none of the 10 facilities with lapse rates higher than the national averages in 2 of 3 years from 2011 to 2013 that did not implement remedial action plans showed any improvement. This shows empirically that implementation of effective remedial measures in facilities with high lapse rates can result in long-lasting improvement in the majority of these facilities (Tsan & Nguyen, 2019).

Remedial measures that these facilities found most helpful included:

tracking expiration dates of IRB approval,

notifying investigators at least 60 days prior to approval expiration,

investigator training and education,

follow-up with investigators to ensure that the continuing review applications were submitted in time for IRB review, and

Suspending all research activities when lapse occurs (Nguyen et al., 2016).

Based on the above VA experience, I recommend the following:

institutions should regularly monitor their IRB continuing review lapse rates,

institutions or IRBs with high annual continuing review lapse rates, such as more than 6–10%, should implement the remedial measures recommended above to improve their compliance with IRB continuing review requirements, and

institutions should evaluate the impact of the revised Common Rule on continuing review lapse rates.

Summary

Performance measurement leads to quality improvement, because performance measurement can identify areas of vulnerability to inform quality improvement priorities. Based on the available IRB performance measurement data from the empirical literature in the last 10 years on research review criteria, expedited review protocols, exempt protocols, and continuing reviews, specific recommendations have been made above to improve the quality and efficiency of IRB performance. It is hoped that implementation of these recommendations will result in improvements that can be documented by follow-up performance measurement.

With the implementation of the revised Common Rule requiring single IRB review of multiinstitutional studies conducted in the United States (Menikoff et al., 2017; U.S. Department of Health and Human Services, 2018), increasing numbers of institutions are relying on freestanding commercial IRBs for such studies. It is important that the quality and performance of these IRBs also be monitored closely (Klitzman et al., 2020; Tsan, 2019b).

Footnotes

Acknowledgment

The author wishes to thank John Thomas Puglisi, PhD, for his critical review of the manuscript.

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Author Contributions

Min-Fu Tsan is responsible for the research idea, collection, analysis, and interpretation of data, and preparation of the manuscript.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Min-Fu Tsan

Author Biography

Min-Fu Tsan is Senior Research Scientist at the McGuire Research Institute, Richmond, VA, USA. From 2009 to 2014, he served as the Deputy Chief Officer, VA Office of Research Oversight. His research interests include, but are not limited to, protection of human subjects participating in research.

References

Anderson

M. A.

. (2018). Overview of the revised common rule. Harvard Catalyst. https://catalyst.harvard.edu/pdf/regulatory-events/Applying%20the%20Regulations_May%201%202018.pdf (accessed on July 7, 2020).

Cassel

C. K.

Conway

P. H.

Delbanco

S. F.

Jha

A. K.

Saunders

R. S.

Lee

T. H

. (2014). Getting more performance from performance measurement. New England Journal of Medicine, 371(23), 2145–2147. https://doi.org/10.1056/NEJMp1408345

Cooper

J. A.

McNair

. (2020). Practical ethicist: Retrospective review of research. Journal of Empirical Research on Human Research Ethics. 15(3), 227–228. https://doi.org/10.1177/15562646/9891343

Emanuel

E. J.

Wendler

Grady

(2008). An ethical frame-work for biomedical research. In Emanuel

E. J.

Grady

Crouch

R. A.

Lie

R. K.

Miller

F. G.

Wendler

(eds.), The Oxford textbook of clinical research ethics (pp. 123–133). Oxford University Press.

Institute of Medicine. (2001). Preserving public trust: Accreditation and human research participant protection programs. National Academies Press.

Institute of Medicine. (2002). Responsible research: A systems approach to protecting research participants. National Academies Press.

Klitzman

Appelbaum

P. S.

Murray

Pivovarova

Stiles

Lidz

C. W

. (2020). When IRBs say no to participating in research about single IRBs. Ethics & Human Research. 42(1): 36–40. https://doi.org/10.1002/eahr.500041

Lidz

C. W.

Appelbaum

P. S.

Arnold

Candilis

Gardner

Myers

Simon

(2012). How closely do institutional review boards follow the common rule? Academic Medicine 87(7), 969–974. https://doi.org/10.1097/IACM.0b013e3182575e2e

Menikoff

Kaneshiro

Pritchard

. (2017). The common rule, updated. New England Journal of Medicine, 376(7), 613–615. https://doi.org/10.1056/NEJMp1700736

10.

National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. (1979). The Belmont report: Ethical principles and guidelines for the protection of human subjects of research. Government Printing Office.

11.

Nguyen

Grabenbauer

Tsan

M. F

. (2016). Improving rates of lapse in IRB continuing reviews: The department of veterans Affairs’ experience. IRB: Ethics & Human Research, 38(4), 17–20. http://www.thehastingscenter.org/irb_article/improving-rates-lapse-irb-continuing-reviews-department-veterans-affairs-experience/

12.

Rowan-Legg

Weijer

Gao

Fernandez

. (2009). A comparison of journal instructions regarding institutional review board approval and conflict-of-interest disclosure between 1995 and 2005. Journal of Medical Ethics. 35(1), 74–78. https://doi.org/10.1136/me.2008024299

13.

Tsan

M. F.

(2017) Assessing the quality of human research protection programs. Scholars’ Press.

14.

Tsan

M. F

. (2019a) Measuring the quality and performance of institutional review boards. Journal of Empirical Research on Human Research Ethics. 14(3), 187–189. https://doi.org/10.1177/1556264618804686

15.

Tsan

M. F

. (2019b) From moving beyond compliance to quality to moving beyond quality to effectiveness: Realities and challenges. Journal of Empirical Research on Human Research Ethics. 14(3), 204–208. https://doi.org/10.1177/1556264619850710

16.

Tsan

M. F

. (2020) Comment on the proposed institutional review board retrospective review of research. Journal of Empirical Research on Human Research Ethics. 15(3), 229–231. https://doi.org/10.1177/1556264620908153

17.

Tsan

M. F.

Ling

Feske

Zickmund

Stone

Sonel

Arnold

R. M.

Fine

Hall

D. E

. (2020). Assessing the quality and performance of institutional review boards: Levels of initial reviews. Journal of Empirical Research on Human Research Ethics. 15(5), 407–414. https://doi.org/10.1177/1556264620956795

18.

Tsan

M. F.

Nguyen

. (2015). Lapse in institutional review board continuing review approval. IRB: Ethics & Human Research, 37(2), 14–19. https://doi.org/10.1213/ANE.0b013e3182095c5d

19.

Tsan

M. F.

Nguyen

. (2017). Effectiveness of human research protection program performance measurements. Journal of Empirical Research on Human Research Ethics, 12(4), 217–228. https://doi.org/10.1177/1556264617720387

20.

Tsan

M. F.

Nguyen

. (2019). Improving compliance with institutional review board continuing review requirements. Journal of Empirical Research on Human Research Ethics, 14(4), 365–371. https://doi.org/10.1177/1556264619837108

21.

U.S. Department of Health and Human Services. (2010). Office for Human Research Protections Guidance on IRB Continuing Review of Research. https://www.hhs.gov/ohrp/regulations-and-policy/guidance/guidance-on-continuing-review-2010/index.html (accessed December 1, 2020)

22.

U.S. Department of Health and Human Services. (2011). Office for Human Research Protections Guidance on reporting incidents to OHRP (2011). https://www.hhs.gov/ohrp/compliance-and-reporting/guidance-on-reporting-incident/index.html (accessed December 1, 2020)

23.

U.S. Department of Health and Human Services. (2016). Human subject regulations decision charts-Pre-2018 requirements. https://www.hhs.gov/ohrp/regulations-and-policy/decision-charts/index.html (accessed December 1, 2020)

24.

U.S. Department of Health and Human Services. (2018). Federal policy for the protection of human subjects: 45 code of Federal Registration (CFR) 46. http://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr-46/index.html (accessed December 1, 2020)

25.

U.S. Department of Health and Human Services. (2020a). Human subject regulations decision charts-2018 requirements. https://www.hhs.gov/ohrp/regulations-and-policy/decision-charts-2018/index.html (Accessed December 1, 2020)

26.

U.S. Department of Health and Human Services. (2020b). Office for Human Research Protections Exempt Research Determination FAQs. https://www.hhs.gov/ohrp/regulations-and-policy/guidance/faq/exempt-research-determination/index.html (accessed December 1, 2020)

27.

U.S. Department of Veterans Affairs. (2019). Requirements for the protection of human subjects in research. VHA Directive 1200.05(1). https://www.va.gov/vhapublications/publications.cfm?pub=2 (access December 1, 2020)

28.

Varley

P. R.

Feske

Gao

Stone

R. A.

Zhang

Monte

Arnold

R. M.

Hall

D. E.

(2016). Time required to review research protocols at 10 Veterans Affairs Institutional Review Boards. Journal of Surgical Research, 204(2), 481–489. https://doi.org/10.1016/J.jss.2016.06.004