Poor Oral Health in Trauma Intensive Care Unit Patients: Application of a Novel Oral Health Score

Abstract

Background:

Although oral hygiene in patients in the intensive care unit (ICU) has been shown to reduce hospital-associated infections, baseline and progressive oral health are often not reported because of lack of a standardized tool. The Oral Health Risk Assessment Value Index (OHRAVI) is a comprehensive oral assessment validated by dental providers. This study hypothesizes that non-dental providers can use OHRAVI in trauma ICU patients with minimal training and acceptable inter-rater reliability (IRR).

Patients and Methods:

Dentulous adult patients in the ICU at a level 1 trauma center were scored, excluding those with severe orofacial trauma. The eight categories of the OHRAVI were scored 0 to 3 (best to worst) with summed total and index (average) score. Index scores 1 or less need routine oral care; greater than 1–2 require moderate care; and greater than 2–3 require extensive oromaxillofacial care. Inter-rater reliability was assessed by two to three raters with Krippendorff's α (≥0.80 for good and ≥0.667 for acceptable).

Results:

Eighty-four ratings were completed across 34 patients, with 16 patients (47%) scored by all three raters. Ten patients (29%) had an index score <1. The average index score for patients was 1.28 (median, 1.34; range, 0.63–2). Krippendorff's α for index score was 0.86. For individual categories, α ranged from 0.44 to 1, with six of the eight categories achieving an α ≥ 0.667.

Conclusions:

With minimal training, non-dental providers were able to use OHRAVI with a good IRR for index score and an acceptable/good IRR for most individual categories. This novel, simple, comprehensive oral health score could help standardize oral assessment and facilitate future studies of peri-operative oral hygiene interventions.

Poor oral health is one of the most common health conditions worldwide and has been linked to multiple chronic diseases such as diabetes mellitus and heart disease.^1–5 Increasing evidence suggests that poor oral health and hygiene are also associated with acute health conditions, such as health-care–associated infections (HAIs).^6–10 Postulated mechanisms include direct seeding, aspiration, and transient bacteremia.^8,11,12 More importantly, there are data suggesting that oral hygiene interventions reduce the risk of HAIs, particularly ventilator-associated pneumonia, in critically ill patients.^6,13 More recently, use of oral hygiene interventions in the pre- and peri-operative setting have been shown to reduce HAIs, including surgical site infections, in patients undergoing elective surgery such as those undergoing procedures for cancer.^9,10,14–17 However, there is a lack of data regarding the impact of oral hygiene interventions on HAIs other than pneumonia in acute care surgical patients, a population that is at high risk for HAIs and who are not eligible for pre-operative risk factor modification.

There is still no strong clinical agreement or guidelines as to what components of oral health and hygiene are most important, what infections may be affected, and how aggressively oral care should be administered.^6,8,18–22 Despite many promising studies in favor of peri-operative oral hygiene to prevent HAIs, studies have conflicting results and recommendations for the type and timing of oral care.^18,23–27 The discrepancies could be related to differences in baseline oral health status; quantity, quality and type of oral hygiene interventions; or complexity of oral disease requiring a more multidisciplinary approach to risk factor modification. However, these hypotheses have not been tested. In fact, few studies evaluating the effect of oral hygiene on post-operative infectious complications include any baseline or progressive oral health information.^10,16 One reason for this limitation is the lack of a widely accepted standardized oral assessment.

The Oral Health Risk Assessment Value Index (OHRAVI), is a simple, comprehensive, oral assessment, designed to be useable by non-dental healthcare providers with minimal training and without specialized equipment or imaging. It evaluates real-time patient oromaxillofacial clinical status and includes components of prior dental interventions. The OHRAVI has been validated by dental health professionals on representative ambulatory patients with good inter-rater reliability (IRR). However, the OHRAVI has not previously been used by non-dental professionals and has not been applied to an inpatient population. The primary hypothesis of this study is that non-dental healthcare professionals can use the OHRAVI in surgical trauma ICU patients and have acceptable IRR. The secondary hypothesis is that critically ill trauma patients have a high incidence of poor oral health suggesting that oral hygiene interventions may be an effective strategy for reducing HAIs.

Patients and Methods

A prospective, Institutional Review Board (IRB)-approved, observational study at a busy, metropolitan level 1 trauma center was performed (May to June 2022) (IRB: HSC-MS-22-0192). Adult (≥16 years old) dentulous, trauma patients requiring surgical trauma ICU admission were included. Patients who were edentulous or presented with severe orofacial trauma (e.g., fractures requiring mandibulomaxillary fixation) were excluded. Patients were examined and scored by two to three raters within 72 hours of admission. The OHRAVI consists of eight categories to examine oromaxillofacial health that are scored 0 to 3 (best to worst) to obtain a summed total score and index (average) score. The categories include: missing/restored/caries; salivary secretion; caries; periodontal condition; oral lesions; pain; bite/occlusion; and oral hygiene. (Table 1). Intubated or obtunded patients received “NA” for pain and their index was calculated from seven categories. Bite/occlusion was not excluded for intubated patients because this category also includes attritions, crowding, gaps, rotation, erosion, and visible fractures, which can indicate malocclusive disease and be detectable despite the endotracheal tube. Index scores less than 1 are healthy, primarily routine oral care needed; greater than 1–2, moderate, require moderate care, possible multidisciplinary care; and greater than 2–3, severe, require extensive, complex level of oral/maxillofacial care.

Table 1.

OHRAVI Examination Sheet

	0 (Healthy)	1 (Mild)	2 (Moderate)	3 (Severe)
Missing/restored/caries	No missing, restored, or carious teeth	1–4 missing, restored, or carious teeth	5–8 missing, restored, or carious teeth	>8 missing, restored, carious teeth
Salivary secretion	Glossy. No obvious signs of salivary deficiency	Matte or thick saliva. Minor salivary deficiency	Plaque buildup. Moderate salivary deficiency	Dry and cracked. Severe salivary deficiency
Visible caries	No clinical evidence of caries	1 tooth clinical sign of caries	2–4 clinical signs of caries	Rampant: >4 clinically present
Soft tissue intraoral lesions (includes traumatic lesions)	No intraoral lesions	One non-gingival or periodontal inflammatory lesion	Few (2–3) inflammatory lesions on soft tissue	Multiple (>3) lesions, potentially malignant
Orofacial pain(scale 1–10) (intubated = NA)	No intra-oral pain	Mild 1–3	Moderate 4–6	Severe 7–10
Periodontal condition	No periodontitis or gingivitis	Only gingival inflammation	Mild/moderate periodontitis	Severe periodontitis
Occlusal/bite condition	No occlusal deficiency, erosion, attritions, or malocclusion	Minor crowding, attritions, erosions, visible tooth fracture	Moderate crowding, attritions, erosions, fractures	Severe occlusal disorders; possible skeletal condition
Oral hygiene/ cleanliness	Excellent oral hygiene	Minor plaque and/or calculus buildup	Moderate plaque/calculus	Severely compromised and poor oral hygiene
			Total score: _______________
			Index score: _______________

Three raters received three short training sessions by dental health professionals in the dental clinic and ICU on examining and scoring oral health, prior to and during the study. The study was designed to determine if non-dental health professionals could obtain appropriate IRR with a few training sessions so that this scoring system could be used to examine oral health formally in relation to patient outcomes, namely HAIs. The primary outcome of this study was IRR of index score, and secondary outcomes was IRR of the eight OHRAVI categories and total score. The IRR was assessed with Krippendorff's α, with a cutoff ≥0.80 for good and ≥0.667 for acceptable.^28,29

Statistical analysis was conducted using R Statistical Software R version 4.1.2 (2021-11-01) in RStudio (RStudio 2022.02.0 + 443 “Prairie Trillium” Release) with irr packages for Krippendorff's α: kripp.alpha.^30–33 If poor IRR was found at time of evaluation, additional training and reassessment of specific score components were planned.

Results

Thirty-four patients were included, with a total of 84 ratings completed. Sixteen patients (47%) were scored by all three raters. The patients were primarily male, suffered blunt injury, had a median age of 49, and one-half had at least one prior medical condition (Table 2). At time of examination, 18 patients (53%) were intubated and received “NA” for pain. Of the 16 non-intubated patients, 10 (63%) reported at least daily teeth brushing and nine (56%) reported seeing a dentist within one year. The average index score was 1.28 (median, 1.34; range, 0.63–2) with 38% (13/34) having a healthy index score of 1 or more, 62% (21/34) having moderate index score, and no patients having a severe score greater than 2 (Table 3).

Table 2.

Patient Information

Patient information (n = 34)
Age (y)	48.5 (32.5, 59.3)
Race/ethnicity
White	13 (38.2%)
Black	11 (32.4%)
Hispanic	2 (5.9%)
Asian	5 (14.7%)
Other	3 (8.8%)
Gender (male)	26 (76.5%)
Mechanism of injury (blunt)	26 (76.5%)
BMI (kg/m²)	28 (23.5, 32)
Tobacco use	7 (20.6%)
Diabetes mellitus	6 (17.6%)
Cardiovascular disease	10 (29.4%)
Any medical history	18 (52.9%)
ED temperature (°F)	98 (97.4, 98.7)
ED systolic blood pressure	114 (102, 142)
ED heart rate	101 (81, 130)
ED respiratory rate	19 (16, 22)
ED GCS	14 (6, 15)
ED lactate	3.4 (2.7, 6.2)
ED base deficit	−3 (−6.5, −1.5)
ED pH	7.29 (7.27, 7.34)
Injury Severity Score total	22 (18.5, 29)
Transfusion^a	21 (62%)
Underwent operation	25 (74%)
Total length of stay	13 (7, 21.5)
ICU length of stay	7 (3.5, 13.5)
Ventilator duration (d)	1 (0, 5)
Hospital-associated infections^b	14 (41.2%)

Categorical variables shown as count (percent), continuous variable shown as median (interquartile range).

ED = emergency department; GCS = Glasgow Coma Scale; ICU = intensive care unit.

Transfusion of blood products within first 4 hours.

Hospital-associated infections include: device and non-device pneumonia, urinary tract infections, primary blood stream infections, and surgical site infections.

Table 3.

Oral Health Scores

	Oral health score
	0	1	2	3
Missing/restored/caries	1 (2.9%)	7 (20.6%)	5 (14.7%)	21 (61.8%)
Salivary secretion	21 (61.8%)	9 (26.5%)	3 (8.8%)	1 (2.9%)
Visible tooth caries	6 (17.6%)	11 (32.4%)	12 (35.3%)	5 (14.7%)
Soft tissue intra-oral lesions	19 (55.9%)	10 (29.4%)	5 (14.7%)	0 (0%)
Orofacial pain (NA: 18)	10 (62.5%)	2 (12.5%)	3 (18.8%)	1 (6.2%)
Periodontal condition	6 (17.6%)	9 (26.5%)	16 (47.1%)	3 (8.8%)
Occlusal/bite condition	1 (2.9%)	19 (55.9%)	14 (41.2%)	0 (0%)
Oral hygiene/cleanliness	2 (5.9%)	19 (55.9%)	8 (23.5%)	5 (14.7%)
Index score	0–1: 13 (38%)
	>1–2: 21 (68%)
	>2: 0 (0%)

Oral health categories shown with count (percent) of patients in each oral health score, with 0 = best and 3 = worst.

Krippendorff's α for the IRR of index scores was 0.86. The individual categories that also had a Krippendorff's α indicating good IRR were orofacial pain (1.0) and missing/restored/caries teeth (0.87). The individual categories of salivary secretion (0.69), visible tooth caries (0.67), soft tissue intra-oral lesion (0.72), and periodontal condition (0.69) were ≥0.667 and met Krippendorff's α criteria of acceptable whereas oral hygiene (0.62) and occlusal/bite condition (0.44) were fair/poor (Table 4). Overall, Krippendorff's α for individual categories ranged from 0.44 to 1, with six of the eight categories achieving an α of ≥0.667. Additionally, exclusion of the pain category, given its subjectiveness as being reported by patients and inability to score on intubated patients, also did not affect the α for index score IRR (0.853).

Table 4.

Krippendorff's α Interrater Reliability

Category	Krippendorff's α
Missing/restored/caries	0.87
Salivary secretion	0.69
Visible tooth caries	0.67
Soft tissue intraoral lesions	0.72
Orofacial pain	1
Periodontal condition	0.69
Occlusal/bite condition	0.44
Oral hygiene/cleanliness	0.62
Total score	0.88
Index score	0.86

Discussion

A novel, standardized oral health assessment tool, OHRAVI, was able to be taught easily to non-dental professionals and applied to an inpatient trauma ICU population. The tool demonstrated good IRR for the overall index score and an acceptable or good IRR for most individual category scores. Additionally, the tool was able to be modified for patients who were intubated, without any effect on the useability or IRR. Approximately two-thirds of critically ill trauma patients evaluated had moderate or more severe oral health problems. If oral hygiene interventions are more effective in patients with suboptimal baseline oral health, then most patients would be potential subjects for future interventional studies of oral hygiene interventions. Furthermore, given that trauma patients are often younger with few comorbidities, application of the OHRAVI score to other populations at high-risk for infectious complications may demonstrate a higher prevalence of poor oral health.

The OHRAVI is a scoring system that has the potential to fill the important gap of standardizing oral health assessment. Existing oral health indices often focus on specific components of oral health and require dental training, tools, or imaging.^10,28 Other oral indices designed for non-dental health professions, such as the modified Beck Oral Assessment Scale and Mucosa Plaque Score, were not designed for the ICU but have been adapted to use there to help also guide oral care frequency.^26,34 However, they have not been used to examine associations of oral health with outcomes such as HAIs.³⁴ Although they could be, oral and maxillofacial conditions can be complex or require multidisciplinary evaluation and care, and overspecialized evaluation may miss relevant components of oral health. Therefore, use of a comprehensive score such as OHRAVI may be better for the evaluation of oral health as a risk factor for HAIs or other complications.

The OHRAVI is designed to require minimal training, to avoid the need for special imaging, and to evaluate overall oromaxillofacial health. Although it was designed to be used on a long-term scale, routine use of specific score components could allow the score to be used to monitor progressive oral health status on a daily or weekly basis. The OHRAVI score may benefit from modifications for inpatient use, particularly in intubated patients. For example, pain was not able to be assessed in sedated patients. Bite/malocclusion, although evaluable, was more difficult to assess in intubated patients which was likely reflected in the lower IRR. Both OHRAVI and suggested modifications should be validated in additional inpatient populations. Furthermore, OHRAVI has potential to be used to incorporate oral health assessment into risk models, to evaluate effects of acute oral hygiene interventions on oral health, and to determine whether there is effect modification of existing and novel interventions based on oral health status.

For this study, Krippendorff's α, was chosen to evaluate IRR over Cohen's κ for several reasons. It can handle more than two raters and varying numbers of raters, while Cohen's κ is designed for only two raters.^28,35 It can ignore missing data, such as not having every patient graded on pain. It can also be used with different types of data, including nominal, ordinal, interval, or ratio data.^28,29,35,36 Additionally, Krippendorff's α is based on the observed disagreement corrected for disagreement expected by chance, rather than using agreement such as Cohen's κ or Fleiss' κ. Finally, its cutoffs are considered to be more conservative, and thus are less likely to overestimate the findings.

There are several weaknesses to this study. A small sample size of patients was included and not all patients were examined by all three raters. However, use of Krippendorff's α allowed for comparisons with varying raters and number of raters. Raters discussed patient scores after each examination and re-review of mouths, when necessary, to facilitate learning and improve examination skills. Some examinations were limited, such as those in patients who were intubated, had other medical devices present, or experienced facial swelling. Examinations were improved with the use of a light and mirror.

Conclusions

In conclusion, non-dental healthcare providers, with minimal training, were able to use OHRAVI with good IRR for index score and an acceptable or good IRR for most of the individual categories. This novel, simple, comprehensive oral health score may help standardize oral assessment, identify high-risk patients in whom to test oral hygiene interventions, and guide future studies on the effectiveness of peri-operative oral hygiene interventions.

Footnotes

Acknowledgments

The authors thank Julian N. Holland III PhD for assistance with statistical analysis.

Previously presented Quickshot Presentation, Academic Surgical Congress, February 7–9, 2023, Houston, Texas.

Authors' Contributions

Data curation (supporting): Quinton, Syed. Data curation (lead): Guy-Frank. Investigation (supporting): Quinton. Investigation (lead): Guy-Frank. Writing—original draft (equal): Quinton, Guy-Frank. Writing—review and editing (supporting): Quinton, Klugh, Dhanani, Adibi. Writing—review and editing (equal): Guy-Frank, Kao. Conceptualization (supporting): Dhanani, Adibi, Kao. Conceptualization (lead): Guy-Frank. Methodology (supporting): Adibi. Methodology (lead): Guy-Frank. Software (lead): Guy-Frank. Formal analysis (lead): Guy-Frank. Visualization (supporting): Kao. Visualization (lead): Guy-Frank. Supervision (supporting): Adibi. Supervision (lead): Kao.

Funding Information

No funding was received.

Author Disclosure Statement

All authors have no conflicts to report.

References

Wolf

, Cagetti

, Fisher

, et al. Non-communicable diseases and oral health: An overview. Front Oral Health, 2021; 2:725460; doi: 10.3389/froh.2021.725460

Wahid

, Chaudhry

, Ehsan

, et al. Bidirectional relationship between chronic kidney disease & periodontal disease. Pak J Med Sci, 2013; 29(1):211–215; doi: 10.12669/pjms.291.2926

Oral health in America: A report of the surgeon general. J Calif Dent Assoc, 2000; 28(9):685–695.

Oral Health in America: Advances and Challenges. National Institute of Health: Bethesda, MD; 2021.

Lee

, Jung

, Kim

, et al. Association between oral health and colorectal adenoma in a screening population. Medicine (Baltimore), 2018; 97(37):e12244; doi: 10.1097/MD.0000000000012244

Zhao

, Wu

, Zhang

, et al. Oral hygiene care for critically ill patients to prevent ventilator-associated pneumonia. Cochrane Database Syst Rev, 2020; 12:CD008367; doi: 10.1002/14651858.CD008367.pub4

Liang

, Zhang

, Hu

, et al. Association between perioperative chlorhexidine oral care and postoperative pneumonia in non-cardiac surgical patients: A systematic review and meta-analysis. Surgery, 2021; 170(5):1418–1431; doi: 10.1016/j.surg.2021.05.008

Pedersen

, Larsen

, Hakonsen

. The effectiveness of systematic perioperative oral hygiene in reduction of postoperative respiratory tract infections after elective thoracic surgery in adults: A systematic review. JBI Database System Rev Implement Rep, 2016; 14(1):140–173; doi: 10.11124/jbisrir-2016-2180

Nobuhara

, Matsugu

, Soutome

, et al. Perioperative oral care can prevent surgical site infection after colorectal cancer surgery: A multicenter, retrospective study of 1,926 cases analyzed by propensity score matching. Surgery, 2022; 172(2):530–536; doi: 10.1016/j.surg.2022.02.015

10.

Nishikawa

, Honda

, Kimura

, et al. Clinical impact of periodontal disease on postoperative complications in gastrointestinal cancer patients. Int J Clin Oncol, 2019; 24(12):1558–1564; doi: 10.1007/s10147-019-01513-y

11.

Lockhart

, Brennan

, Thornhill

, et al. Poor oral hygiene as a risk factor for infective endocarditis-related bacteremia. J Am Dent Assoc, 2009; 140(10):1238–1244; doi: 10.14219/jada.archive.2009.0046

12.

Nishikawa

, Honda

, Kimura

, et al. The bacterial association with oral cavity and intra-abdominal abscess after gastrectomy. PLoS One, 2020; 15(11):e0242091; doi: 10.1371/journal.pone.0242091

13.

Bellissimo-Rodrigues

, Menegueti

, Gaspar

, et al. Effectiveness of a dental care intervention in the prevention of lower respiratory tract nosocomial infections among intensive care patients: A randomized clinical trial. Infect Control Hosp Epidemiol, 2014; 35(11):1342–1348; doi: 10.1086/678427

14.

Hasegawa

, Takeda

, Tanaka

, et al. Effects of preoperative dental examination and oral hygiene instruction on surgical site infection after hepatectomy: A retrospective study. Support Care Cancer, 2021; 29(2):653–659; doi: 10.1007/s00520-020-05525-7

15.

Ishikawa

, Yamamori

, Takamori

, et al. Evaluation of effects of perioperative oral care intervention on hospitalization stay and postoperative infection in patients undergoing lung cancer intervention. Support Care Cancer, 2021; 29(1):135–143; doi: 10.1007/s00520-020-05450-9

16.

Janaphan

, Hashem

, Smith

, et al. Periodontal disease as a primary cause of surgical site infection in fractures of the mandible: Is smoking a confounding variable?. Br J Oral Maxillofac Surg, 2022; 60(10):1424–1429; doi: 10.1016/j.bjoms.2022.08.001

17.

Kurasawa

, Maruoka

, Sekiya

, et al. Pneumonia prevention effects of perioperative oral management in approximately 25,000 patients following cancer surgery. Clin Exp Dent Res, 2020; 6(2):165–173; doi: 10.1002/cre2.264

18.

Dale

, Rose

, Carbone

, et al. Effect of oral chlorhexidine de-adoption and implementation of an oral care bundle on mortality for mechanically ventilated patients in the intensive care unit (CHORAL): A multi-center stepped wedge cluster-randomized controlled trial. Intensive Care Med, 2021; 47(11):1295–1302; doi: 10.1007/s00134-021-06475-2

19.

Bardia

, Blitz

, Dai

, et al. Preoperative chlorhexidine mouthwash to reduce pneumonia after cardiac surgery: A systematic review and meta-analysis. J Thorac Cardiovasc Surg, 2019; 158(4):1094–1100; doi: 10.1016/j.jtcvs.2019.01.014

20.

Wren

, Martin

, Yoon

, et al. Postoperative pneumonia-prevention program for the inpatient surgical ward. J Am Coll Surg, 2010; 210(4):491–495; doi: 10.1016/j.jamcollsurg.2010.01.009

21.

Ogawa

, Satomi-Kobayashi

, Yoshida

, et al. Impact of oral health status on postoperative complications and functional recovery after cardiovascular surgery. CJC Open, 2021; 3(3):276–284; doi: 10.1016/j.cjco.2020.10.007

22.

Nishikawa

, Honda

, Kimura

, et al. The effects of intensive oral care before surgery for gastric cancer patients. Oral Dis, 2021; 27(7):1847–1853; doi: 10.1111/odi.13722

23.

Parreco

, Soe-Lin

, Byerly

, et al. Multi-center outcomes of chlorhexidine oral decontamination in intensive care units. Surg Infect, 2020; 21(8):659–664; doi: 10.1089/sur.2019.172

24.

Bellissimo-Rodrigues

, Menegueti

, de Macedo

, et al. Oral mucositis as a pathway for fatal outcome among critically ill patients exposed to chlorhexidine: Post hoc analysis of a randomized clinical trial. Crit Care, 2019; 23(1):382; doi: 10.1186/s13054-019-2664-6

25.

Klompas

. Oropharyngeal decontamination with antiseptics to prevent ventilator-associated pneumonia: rethinking the benefits of chlorhexidine. Semin Respir Crit Care Med, 2017; 38(3):381–390; doi: 10.1055/s-0037-1602584

26.

Kelly

, Blackwood

, Credland

, et al. Oral health care in adult intensive care units: A national point prevalence study. May 1, 2023. Nurs Crit Care [Epub ahead of print; DOI: 10.1111/nicc.12919].

27.

Winning

, Lundy

, Blackwood

, et al. Oral health care for the critically ill: A narrative review. Crit Care, 2021; 25(1):353; doi: 10.1186/s13054-021-03765-5

28.

Hallgren

. Computing inter-rater reliability for observational data: An overview and tutorial. Tutor Quant Methods Psychol, 2012; 8(1):23–34; doi: 10.20982/tqmp.08.1.p023

29.

Beckler

, Thumser

, Schofield

, et al. Reliability in evaluator-based tests: Using simulation-constructed models to determine contextually relevant agreement thresholds. BMC Med Res Methodol, 2018; 18(1):141; doi: 10.1186/s12874-018-0606-7

30.

Team

RStudio: Integrated Development Environment for R. Boston, MA; 2022.

31.

Team

RC.

R: A Language and Environment for Statistical Computing. Vienna, Austria; 2021.

32.

Gamer

, Lemon

, Fellows

, et al. CRAN. irr: Various Coefficients of Interrater Reliability and Agreement. 2012. Available from: https://cran.r-project.org/web/packages/irr/irr.pdf [Last accessed: August 23, 2023].

33.

Krippendorff

Content analysis: An introduction to its methodology (4th ed). Sage Publications: Los Angeles; 2019.

34.

Ames

, Sulima

, Yates

, et al. Effects of systematic oral care in critically ill patients: A multicenter study. Am J Crit Care, 2011; 20(5):e103–114; doi: 10.4037/ajcc2011359

35.

Zapf

, Castell

, Morawietz

, et al. Measuring inter-rater reliability for nominal data: Which coefficients and confidence intervals are appropriate?. BMC Med Res Methodol, 2016; 16:93; doi: 10.1186/s12874-016-0200-9

36.

Harvey

. A Simple Guide to Inter-rater, Intra-rater and test-retest Reliability for Animal Behavior Studies. OSF Preprints, 2021; doi: 10.31219/osf.io/8stpy