Impact of the ICD-11 on the accuracy of clinical coding in Korea

Abstract

Background: ICD-11 was officially released at the World Health Assembly on 25 May 2019. Objective: To find effective ways to increase the accuracy of coding for diagnostic terms in Korea for a stable transition from Korean modification of ICD-10 (7th Revision of Korean Classification of Disease, KCD-7) to ICD-11. Method: A total of 27 skilled Korean health information managers performed KCD-7 and ICD-11 coding simultaneously (line coding [56]; case coding [17]). Accuracy rates and percentage agreements were calculated, and granularity and difficulty of the ICD-11 were rated by participants. Results: The average accuracy rate of line coding was 71.6 % in ICD-11 and 80.2% in KCD-7, which was similar to results in other studies. The mean percentage agreements for ICD-11 and KCD-7 for line coding were 64.2% and 72.1%, respectively; while for case coding it was 15.3% and 26.6%. Selection criteria for the case scenarios may have influenced the low agreements in case coding. Cluster coding, changes of terms in ICD-11 and removal of codes used in ICD-10 contributed to low agreement in ICD-11 (46.6% of participants reported that granularity of ICD-11 was similar to ICD-10, while 36.9% reported that ICD-11 had finer granularity). In terms of difficulty, 15.3% of participants found line coding difficult and 10.9% found case coding difficult. Conclusion: Provision of more detailed reference guidelines and efficient training for coding professionals by the World Health Organization would enable ICD-11 to be an excellent tool for gathering relevant information about diseases in Korea.

Keywords

International classification of diseases ICD-10 ICD-11 clinical coding health information management Korea

Introduction

The International Statistical Classification of Disease and Related Health Problems, Tenth Revision (ICD-10) has been used in many countries for 30 years (Tanpowpong et al., 2021) as a basic tool for coding mortality and morbidity problems, and can be used to guide policies in the public health environment for specific and impactful changes (Almeida et al., 2020; Barke et al., 2018; Barraza et al., 2020). The development of ICD-11 started in 2007. Following an alpha draft in May 2001 and a beta draft in May 2012, a stable version of the ICD-11 Mortality and Morbidity Statistics (ICD-11 MMS) was released in June 2018 to enable member states to begin planning for implementation. ICD-11 contains over 55,000 codes. It is considerably more complex than ICD-10 (Balhara et al., 2020; Lou et al., 2017; Reed et al., 2018a). It addresses the changing needs of digitised healthcare systems and healthcare innovation and is capable of coding detailed features of a patient’s clinical condition (Krawczyk and Swiecicki, 2020). In 2017, the World Health Organization (WHO) performed a field trial prior to the release of ICD-11, involving 1673 coding experts from 31 countries (including Korea) performing 112,383 code assignments (IFHIMA, 2018). In total, 298 generic line coding and 30 case scenarios were included, and user guidelines, ICD-11 tools and training materials were changed in line with the results of the trial.

Korean participants experienced great difficulty in post-coordination, which was newly introduced into ICD-11 and required the use of multiple codes to describe some clinical concepts (WHO, 2021). Education about the ICD-11 structure was found necessary (Korean Medical Record Association, 2017) and a domestic field trial for diagnoses that are frequent in Korea was also required. Statistics Korea supported two domestic ICD-11 field trials in 2018 and 2019. The study chapters and volume were decided by the needs and annual budget of Statistics Korea. A university conducted the first trial (Choi, 2018) for Chapter 1 (Certain infectious and parasitic diseases), Chapter 2 (Neoplasms), Chapter 3 (Diseases of the blood or blood forming organs) and Chapter 4 (Diseases of the immune system). In 2019, the Korean Health Information Management Association performed the second field trial for Chapter 5 (Endocrine, nutritional or metabolic diseases), Chapter 9 (Diseases of the visual system) and Chapter 10 (Diseases of the ear or mastoid process). The current study reports the results of the second trial, the aim being to find effective ways to increase the accuracy of coding of diagnoses that are frequent in Korea, in order to support a stable transition to ICD-11 by examining coding differences between KCD-7 and ICD-11.

Method

Trial diagnoses selection and standard responses

Six health information managers working in six tertiary hospitals and general hospitals with more than 10 years of clinical coding experience selected target diagnoses according to the following criteria:

1. frequent diseases in six research hospitals;

2. diagnoses with different descriptions of the same concept in ICD-11 and KCD-7;

3. diagnoses present in KCD-7 only;

4. code pairs as a diagnosis – the dagger and asterisk system in KCD-7;

5. chapter transition;

6. changed or removed diagnosis from KCD-7; and

7. different mapping chapters between ICD-11 and KCD-7 for same diagnosis.

For line coding, 56 diagnostic terms were selected, with 17 case scenarios for case coding prepared from patient records, including pathology reports and procedure notes. In all, there were 111 diagnoses. Selected diagnostic terms by chapters are shown in Box 1. Box 1.

Number of selected diagnostic terms by chapter.

Coding	ICD-11 chapter				Total
Coding	Chapter 5	Chapter 9	Chapter 10	Other	Total
Line coding	25	25	3	3^a	56
Case coding
1	1			2	3
2	2			2	4
3	2			2	4
4	2	1			3
5	1		1	2	4
6	2	1		1	4
7	1	2			3
8	2			2	4
9	2	1		1	4
10		3			3
11	2	2			4
12		1	1		2
13			1		1
14			1	2	3
15	1	1		2	4
16	1			1	2
17			3		3
case coding total	19	12	7	17^b	55
Total	44	37	10	20	111
%	39.6	33.3	9.0	18.0	100.0

^aOther chapters in line coding: Codes moved to other chapters.

^bOther chapters in case coding: Codes from other chapters required to complete clinical descriptions.

A code cluster is a group of codes composed by post-coordination. Cluster coding presents the code cluster using a specific syntax to indicate which codes belong together when post-coordination is used. The WHO guidelines for cluster coding were not sufficiently clear to enable their application for the field trial at that time. Standard responses for line coding were developed based on a consensus of six health information managers (HIMs), who consulted relevant doctors for the 20 coding cases on which they did not reach consensus. Standard responses for the case coding were not able to be set.

Field trial

The ICD-11 browser (MMS 2019 April) was provided with Korean translation in a web-based Korean ICD-11 field trial programme performed over a one-week period in July 2019. Participants were required to log in to the programme with their own ID and password. The programme had a Questions and Answers (Q & A) bulletin board, where participants in the programme could be supported in real time, including coding method, case coding and the survey. Diagnostic terms and case scenarios were provided to 27 participants with more than five years of clinical coding experience, who worked in one of 13 tertiary hospitals or general hospitals. Participants used both ICD-11 and KCD-7 to code the same diagnosis. For each code or code cluster, they were required to answer survey questions about the granularity difference between ICD-11 and KCD-7, and the difficulty of ICD-11 clinical coding. Face-to-face training was provided for: ICD-11 features; 2018 field trial results in Korea; introduction and use of the Korean ICD-11 field trial programme; ICD-11 conventions; ICD-11 reference guide; and field trial examples.

Statistical analysis

The accuracy rates of ICD-11 and KCD-7 line coding for each of the 56 diagnostic terms were calculated. Because of difficulty in reaching a consensus on standard responses for case coding, we could not provide accuracy rates for case coding. Percentage agreements for each line coding or case coding case were calculated as the inter-rater agreements of both line coding and case coding in ICD-11 and KCD-7, as outlined below.

A g r e e m e n t p r o p o r t i o n = \frac{N u m b e r o f p a i r s w i t h s a m e c o d e o r c l u s t e r}{T o t a l n u m b e r o f p a i r s}

The total number of pairs for the percentage agreements was ₂₇C₂ (=351). Accuracy rates and percentage agreements were calculated with and without the order of codes in each cluster. R version 4.0.5 was used for statistical analysis.

Ethics approval

This study was determined to be exempted from deliberation in accordance with Article 13, Paragraph 2 of the Enforcement Rule of Bioethics and Safety Act among the Ordinance of the Ministry of Health and Welfare, Korea.

Results

Accuracy of line coding

The average accuracy rate of ICD-11 coding was 74.5% when the order of the codes in a cluster was not considered and 71.6% when order was considered. The accuracy of 14 diagnostic terms was 100%. However, nine terms showed accuracy of less than 30%. The average accuracy rate of KCD-7 coding was 80.2%, slightly higher than that of ICD-11.

There were diagnostic terms classifiable only in ICD-11. Accuracy rates of KCD-7 for these terms were lower than those of ICD-11. In cases where a code cluster was needed or a code had been deleted, added, or moved in ICD-11 compared to ICD-10, the accuracy rate was especially low. Several examples have been provided in 1 and 2 (see Table 1 and Table 2). ‘Severe non-proliferative retinopathy in diabetes mellitus’ was a good example of cluster coding. The correct coding in ICD-11 was 9B71.00&XK6G&XS25/5A14 (Diabetic retinopathy & unspecified laterality & severe/diabetes) and the accuracy rate of ICD-11 was only 22.2%. There were various code clusters missing the required stem code or with a different order of stem codes or extension codes. “Furuncle in the external ear” was only listed in Korean modification as H60.02. Accuracy of coding in KCD-7 was 100%. In ICD-11, AA00 (Abscess of external ear), AA3Z&XK8G&XA6ZY6 (Abscess of external ear & left & external ear), and other codes were assigned. AA3Z&XK8G&XA6ZY6 was suggested as a standard answer and the accuracy was 22.2%. The accuracy rate of ICD-11 for “eyelid apraxia” was 96.3%. However, the accuracy of KCD-7 coding was 44.4% because this diagnosis was not listed in the KCD-7.

Table 1.

Selected line coding examples of ICD-11.

ICD-11 coding	Description		N (%)
Severe non-proliferative retinopathy in diabetes mellitus
5A14/9B71.00&XK6G^**&XS25	Diabetes/Diabetic retinopathy and unspecified laterality and severe	8	(29.6)
9B71.00&XK6G&XS25/5A14	Diabetic retinopathy and unspecified laterality and severe/Diabetes	6	(22.2)
9B71.00&XS25&XK6G/5A14	Diabetic retinopathy and severe and unspecified laterality/Diabetes	4	(14.8)
9B71.00&XK6G&XS25	Diabetic retinopathy and severe and unspecified laterality	2	(7.4)
5A14/9B71.00&XS25&XK6G	Diabetes/Diabetic retinopathy and severe and unspecified laterality	2	(7.4)
Other		5	(18.5)
Furuncle in the external ear, left
AA00&XK8G	Abscess of external ear and left	11	(40.8)
AA3Z&XK8G&XA6ZY6	Otitis externa, unspecified and left and external ear	6	(22.2)
1B75.0&XA6ZY6&XK8G	Furuncle and external ear and left	2	(7.4)
Other		8	(29.6)
Eyelid apraxia
9A05.1&XK9K	Eyelid apraxia and right	26	(96.3)
No answer		1	(3.7)

XK6G was in the 2019 MMS frozen version.

Table 2.

Selected line coding examples of KCD-7.

KCD-7 coding	Description	N(%)
Severe non-proliferative retinopathy in diabetes mellitus
E14.31, H36.0	Diabetes mellitus with retinopathy	20	(74.1.)
Others		7	(25.9)
Furuncle in the external ear, left
H60.02	Furuncle in the external ear	27	(100.0)
Eyelid apraxia
H02.88	Other specified disorders of eyelid	12	(44.4)
H51.8	Other specified disorders of binocular movement	9	(33.3)
H02.58	Other disorders affecting eyelid function	3	(11.1)
Other		3	(11.1)

We also discovered some term changes, removal of codes, and change of coding rules. For example, ‘globe’ was used in the Chapter 9 (Diseases of the eye and adnexa) of KCD-7. However, in ICD-11, for some codes, only ‘eyeball’ was used, while ‘globe’ was not provided even in index terms. Codes ‘H04.0 Dacryoadenitis’ and ‘E28.0 Oestrogen Excess’ were not available in ICD-11. For ‘J40 Bronchitis, not specified acute or chronic’, a coding note (‘Bronchitis not specified as acute or chronic in those under 15 years of age can be assumed to be of acute nature and should be classified to J20.-.’) was provided in KCD-7 to clarify the use of the code. However, in ICD-11 ‘CA20.Z Bronchitis, unspecified’, the coding note was removed, which confused participants who followed the coding note in KCD-7.

Percentage agreement of clinical coding

Line coding

The mean percentage agreement of 56 diagnostic terms of line coding was 64.2% in ICD-11, and 72.1% in KCD-7. The diagnostic terms that showed low accuracy rates generally showed low percentage agreements. Post-coordination and cluster formation had negative effects on percentage agreement in ICD-11. For 20 diagnostic terms (22%), the percentage agreement was above 90%, and for two diagnoses (primary hypothyroidism, eyelid apraxia), the percentage agreement of ICD-11 was 20% higher than KCD-7.

Case coding

The percentage agreement of ICD-11 was 15.3% and that of KCD-7 was 26.6% (see Table 3). When anatomical concept was complicated or KCD-7 unique terms were provided, ICD-11 percentage agreement was lower than those of KCD-7.

Table 3.

Percentage agreement of ICD-11 and KCD-7 for each case scenarios (N = 351).

No	Number of diagnoses	ICD-11(%)		KCD-7 (%)
No	Number of diagnoses	Without order of codes	With order of codes	KCD-7 (%)
1	3	5.7	5.7	48.7
2	4	2.5	3.1	7.4
3	4	3.7	3.7	7.9
4	3	3.1	4.3	24.8
5	4	0.8	1.7	4.0
6	4	1.9	1.9	1.7
7	3	2.5	3.1	5.4
8	4	11.1	11.1	25.4
9	4	3.3	3.7	7.9
10	3	9.7	9.7	45.3
11	4	3.1	3.1	40.5
12	2	54.4	54.4	66.1
13	1	42.2	42.2	54.4
14	3	20.8	20.8	26.5
15	4	11.4	11.4	39.6
16	2	72.1	72.1	33.0
17	3	3.1	3.1	13.7
Total	55	14.8	15.3	26.6

Review of cluster coding in case coding

Participants submitted different numbers of code clusters depending on the way they combined the Principal Diagnosis and other diagnoses. Participants who combined the most diagnoses submitted 39 clusters for 55 diagnoses, whereas others who combined the least diagnosis submitted 54 clusters. Where there was a cause-and-effect relationship between diagnoses, each code should be combined as a cluster. However, the number of code clusters and combinations were different, according to participants’ ways of combine diagnoses or codes. The percentage agreement was low when the cause-and-effect relationship between the Principal Diagnosis and other diagnoses was unclear.

Survey results

Participants’ responses indicted that the granularity of ICD-11 was similar to KCD-7 or finer in both line coding (46.6%, 36.6%) and case coding (46.8%, 34.8%), while the proportion similar to KCD-7 and finer was higher in case coding than in line coding (p < 0.001) (see Table 4). A greater number of participants reported moderate difficulty in case coding (36.9%) than in line coding (29.5%) (p < 0.001). ICD-11 line coding was considered difficult or very difficult by 15.3% of participants, while only 10.9% of participants answered difficult or very difficult in case coding. When the diagnosis contained detailed multiple concepts such as ‘diabetic peripheral angiopathy with type 1 diabetes mellitus, without gangrene, Femoro-popliteal’ and diabetic complication, participants experienced coding difficulties.

Table 4.

Difficulty and granularity of ICD-11 coding, N (%).

	Line coding (N = 1512)	Case coding (N = 1480)	χ²	p-value
Granularity of ICD-11
Similar to KCD-7	704 (46.6)	693 (46.8)	20.5	<0.001
Finer	553 (36.6)	515 (34.8)
Coarser	222 (14.7)	126 (8.5)
No answer†	33 (2.2)	146 (9.9)
Difficulty level
Very difficult	30 (2.0)	11 (0.7)	57.2	<0.001
Difficult	201 (13.3)	151(10.2)
Moderate	446 (29.5)	546 (36.9)
Easy	610 (40.3)	535 (36.1)
Very easy	193 (12.8)	95 (6.4)
No answer^†	32 (2.1)	142 (9.6)

†

No answer was excluded from χ² analysis

Discussion

Language barrier

The biggest obstacle to the Japanese ICD-11 field trial in 2017 was English proficiency (Sato et al., 2019). Raters had problems understanding ICD-11. About 70% of participants in the additional questionnaire said they were not confident with their English skills. In Korea, we used the ICD-11 browser and Korean translation in the Korean ICD-11 field trial programme. In addition, participants were familiar with the English version of ICD-11 because diagnostic terms in Korean hospitals are usually written in English. Thus, English was not a barrier in the field trial in Korea.

Accuracy and agreement of clinical coding

The average accuracy rate of ICD-11 line coding was 74.5% when the order of the codes in a cluster was not considered, and 71.6% when order sequence was considered, both of which were higher than the accuracy of a field trial for pain (63.2%) (Barke et al., 2021). Mean percentage agreement of line coding was 64.2% in ICD-11 and 72.1% in KCD-7. For case coding, it was 15.3% in ICD-11 and 26.6% in KCD-7. Other studies have showed that the agreement with ICD-11 was better than that with ICD-10 (Barke et al., 2021; Gaebel et al., 2020), although the percentage agreements of case coding were very low, in spite of other studies adopting different inter-rater reliability coefficients. One explanation for these results may have been the selection criteria, which made coding challenging for the participants. Other explanations could include post-coordination and code differences in ICD-11. In the main, we choose a single code from pre-listed disease and disability codes in KCD-7 coding. Post-coordination is a newly introduced capability of ICD-11 and is a generalised form of the ‘dagger and asterisk’ system in ICD-10. ICD-11 post-coordination capability allows more detailed description of diseases (Almeida et al., 2020; Fung et al., 2020). On the other hand, diverse post-coordination may exist according to coders’ selection and combination of stem codes and extension codes, which can decrease the accuracy of ICD-11 clinical coding. The same issues relating to post-coordination have been raised by Germany (Gaebel et al., 2018) and Japan (Nishio et al., 2019). Participants also coded in various ways when the title in a code was changed. They suggested adding terms in KCD-7 to index terms of ICD-11 to keep coding consistent. Where a code in the previous version was removed or changed, or a coding rule was changed, as in J40, a clear instruction or guidance should be prepared. Percentage agreement cannot eliminate agreement by chance (Suen and Lee, 1985) when used in multiple-choice questions with limited number of answers. Other researchers used Krippendorff’s Alpha (Eisele et al., 2019) or intraclass kappa coefficient (Reed et al., 2018b) to assess the agreement of ICD-11 coding. However, we adopted percentage agreement in this study to calculate inter-rater reliability because participants assigned codes using their coding knowledge and each answer was different as determined by the coding cases, which should have minimised agreement by chance.

Survey results of granularity and difficulty

Approximately half of all participants reported that the granularity of ICD-11 was similar to that of KCD-7. Before the trial, we had expected that participants would report that ICD-11 had finer granularity because the number of codes in ICD-11 is much greater than in KCD-7; but that they might also feel that granularity was almost the same because of the limited codes used in the field trial (only three chapters from ICD-11 were included). The Japanese field trial in 2017, which covered all chapters, reported that granularity of ICD-11 was different by chapters and suggested the need for more detailed analysis of the different chapters (Sato et al., 2019).

Most participants in the current study reported that ICD-11 was easy or moderately difficult for the study chapters included. They reported difficult or very difficult in line coding (15.3%) and case coding (10.9%). However, participants may have reported less difficulty with case coding than with line coding because with case coding they were more easily able to locate the correct ICD-11 code using additional information such as chief complaint and present illness.

Suggestions for accurate and reliable coding

Need for detailed guidance or support system for post-coordination and/or cluster coding

Post-coordination was the main reason for the inconsistencies found in ICD-11 coding. Therefore, detailed guidance or support system for post-coordination or clustering is required (Chen et al., 2019). Coding results changed according to the way participants used post-coordination. Most participants had difficulties because there was no detailed guidance for using post-coordination. For example, for the diagnosis ‘severe non-proliferative retinopathy in diabetes mellitus’, 45% of participants selected diabetes as the stem code and diabetic retinopathy was added as post-coordination, whereas 41% put diabetic retinopathy as the stem code. Although the participants selected the same stem code, the way the extensions were added was different. Some put laterality first, others put severity first, while others only put one of two extensions. This was an important reason for inconsistency of coding results.

For case coding, the effect of post-coordination was stronger. The cause-and-effect relationship between diagnoses was an important factor to inform the creation of a cluster code. For example, for case number #2, the Principal Diagnosis was ‘chronic kidney disease, stage 5’ with three other diagnoses ‘Non-insulin-dependent diabetes mellitus with established diabetic nephropathy’, ‘glomerular disorder in diabetes mellitus’, and ‘Acute renal failure’, 20 among 27 participants submitted different cluster codes. Participants had difficulties making clusters because the diseases had cause and effect relationships and they did not know how to combine each disease code in the right order in a cluster (see Table 4). In KCD-7, some codes have the symbol ‘*(asterisk)’ and ‘+(dagger)’ so that coders can easily select related codes. However, in ICD-11, all related codes should be selected in the post-coordination disease list (associated disease, manifestation). Since the provided disease range in the post-coordination list included all diabetic and all body systems, coders needed more time to find the correct manifestation from a long list. At the time of the study, the ICD-11 coding tool had limited functionality to support post-coordination, and the participants did not use it.

Provision of guidelines for changed code or coding rule

Where there were changes in the code or coding rule, many coding specialists who were accustomed to KCD-7, showed confusion in using ICD-11. A special section or booklet with information about changed codes or coding rules might help coders reduce their confusion in coding.

Need for intensive education to provide clinical knowledge

Doctors in Korea do not always record all information needed for coding, such as the cause-and-effect relationship between diseases or severity of disease, and some coders need more clinical knowledge to enable them to raise an efficient query to doctors. For example, the case coding case #2 (see Box 2), the clinical knowledge of participants concerning the relationship between diabetes, chronic kidney disease and acute kidney disease affected the coding result. Greater clinical knowledge was needed for clinical coders to select the stem code and to apply post-coordination in ICD-11, indicating that further study and education in clinical knowledge should be provided to clinical coders before transition to ICD-11. Box 2.

Case coding example: case #2.

Principal diagnosis: Chronic kidney disease, stage 5Other diagnosis 1: Non-insulin–dependent diabetes mellitus, with established diabetic nephropathyOther diagnosis 2: Glomerular disorder in diabetes mellitusOther diagnosis 3: Acute renal failure
Number	Cluster	Count	Number of clusters
1	5A11/GB61.5/GB4Z/GB60.Z	1	1
2	5A11/GB61.Z	1	1
3	GB61.5/5A11	1	1
4	GB61.5/5A11/MF83/GB60.Z	1	1
5	5A11^a/GB61.5/MF83 \| GB60.Z	1	2
6	5A11^a/MF83/GB61.5 \| GB60.0/1G40	1	2
7	5A11/GB61.5 \| MF83 \| GB60.Z	1	3
8	GB61.5/5A11 \| MF83/5A11 \| GB60.Z	1	3
9	GB61.5/5A11 \| MF83 \| GB60.Z	2	3
10	GB61.5 \| 5A11/GB61.Z/MF83 \| GB60.Z	2	3
11	GB61.5/5A11 \| 5A11/GB61.5^b \| 5A11/GB4Z \| GB60.Z	1	4
12	GB61.5/5A11 \| 5A11/GB61.5^b \| MF83/5A11 \| GB60.Z	1	4
13	GB61.5/5A11 \| 5A11/GB61.5^b \| MF83 \| GB60.Z	1	4
14	GB61.5/5A11 \| 5A11/GB61.Z^b \| MF83/5A11 \| GB60.Z	2	4
15	GB61.5/5A14 \| GB61.Z/5A14^b \| MF83/5A14 \| GB60.Z	1	4
16	GB61.5 \| 5A11/GB61.5^b \| MF83/5A11 \| GB60.Z	1	4
17	GB61.5 \| 5A11/GB61.5^b \| MF83 \| 5A2Y	1	4
18	GB61.5 \| 5A11/GB61.5^b \| MF83 \| GB60.Z	1	4
19	GB61.5 \| 5A11/GB61.Z^b \| MF83/5A11 \| GB60.Z	4	4
20	GB61.5 \| 5A11/GB61.Z^b \| MF83/5A14 \| GB60.Z	1	4
21	GB61.5 \| 5A15A11/GB61.5/GB60.0 \| MF83/5A11 \| GB60.0	1	4

1G40: Sepsis without septic shock, GB4Z: Glomerular disease, unspecified, GB60.0: Acute kidney failure stage 1, GB60.Z: Acute kidney failure, GB61.5: Chronic kidney disease, stage 5, GB61.Z: Diabetic nephropathy, 5A11: Non-insulin–dependent diabetes mellitus, 5A14: Diabetes mellitus, type unspecified, 5A2Y: Other specified acute complications of diabetes mellitus, MF83: Glomerular disorder in diabetes mellitus.

Slanted bar (/) is used to combine two or more stem codes in cluster coding. Vertical bar (|) separates clusters. Participants were asked to make appropriate clusters to express all diseases.

^aParticipants made cluster with principal disease, diabetes, and diabetic nephropathy (# 5, 6).

^bParticipants tried to put codes for each diagnosis expressing cause and effect relationship leading to duplicate coding (# 11- 20).

Limitations

In the absence of an official gold standard for clinical coding, senior health information managers decided on the most suitable coding responses for line coding, with additional information provided by consulting doctors. For case scenarios, no consensus was reached for standard responses. As the study did not allow enough time for training, results did not fully reflect participants’ classification capacity. Appropriate training for clinical knowledge and ICD-11 coding might lead to higher quality coding results in future studies. Results of this study may not be applicable to more recent versions of ICD-11 because ICD-11 is continuously updated. Therefore, the possibility that some issues discussed in this report may have already been updated should be taken into account.

Conclusion

As the ICD-11 contains very detailed codes, participants in this study were easily able to code some terms. However, if there were any changes in terms in ICD-11 or the code used in KCD-7 was not present in ICD-11, participants expressed difficulties. To ensure a smooth transition from KCD-7 to ICD-11, it is necessary to examine the consistency of terms and codes. Lacking detailed instructions on how to add ‘post-coordination’ to stem code was another major factor that made coding difficult. If the WHO could provide more detailed reference guidelines and more efficient training for clinical coding professionals in each country, ICD-11 would be an excellent tool for gathering relevant information about diseases.

Footnotes

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This research project was funded by Statistics Korea (11-1240000-001341-01). The results of this study are not associated with the Statistics Korea’s official position and the authors are solely responsible for its content and conclusions.

ORCID iDs

Hyunkyung Lee

Sukil Kim

References

Almeida

MSC

Sousa Filho

Rabello

, et al. (2020) International Classification of Diseases-11th revision: from design to implementation. Rev Saude Publica 54: 104.

Balhara

YPS

Singh

Saini

, et al. (2020) Development and validation of gaming disorder and hazardous gaming scale (GDHGS) based on the WHO framework (ICD-11 criteria) of disordered gaming. Asian Journal of Psychiatry 54: 102348.

Barke

Korwisi

Casser

, et al. (2018) Pilot field testing of the chronic pain classification for ICD-11: the results of ecological coding. BMC Public Health 18(1): 1239.

Barke

Korwisi

Jakob

, et al. (2021) Classification of chronic pain for the International Classification of Diseases (ICD-11): results of the 2017 International WHO Field Testing. Pain 163: e310–e318. Epub ahead of print 2021/04/18. DOI: 10.1097/j.pain.0000000000002287.

Barraza

JAM

Benardis

CNS

Jeria

RSA

, et al. (2020) Concordance in Discriminating Recordings of Different Lung Sounds Between Physiotherapists. Respir Care 65(2): 177–182.

Chen

Zhang

Zhao

, et al. (2019) A Bibliometric Analysis of the Development of ICD-11 in Medical Informatics. Journal of Healthcare Engineering 2019: 1649363.

Choi

(2018) The First Study of Korean Version Field Trial for the Eleventh Revision of the International Classification of Diseases(ICD-11). Geneva, Switzerland: WHO.

Eisele

Dereskewitz

Oberhauser

, et al. (2019) Reliability, usability and applicability of the ICD-11 beta draft focusing on hand injuries and diseases: results from German field testing. International Journal for Quality in Health Care 31(10): G174–G179.

Fung

Bodenreider

(2020) The new International Classification of Diseases 11th edition: a comparative analysis with ICD-10 and ICD-10-CM. Journal of the American Medical Informatics Association: JAMIA 27(5): 738–746.

10.

Gaebel

Riesbeck

Zielasek

, et al. (2018) [Web-based field studies on diagnostic classification and code assignment of mental disorders: comparison of ICD-11 and ICD-10]. Fortschr Neurol Psychiatr 86(3): 163–171.

11.

Gaebel

Stricker

Riesbeck

, et al. (2020) Accuracy of diagnostic classification and clinical utility assessment of ICD-11 compared to ICD-10 in 10 mental disorders: findings from a web-based field study. European Archives of Psychiatry and Clinical Neuroscience 270(3): 281–289.

12.

IFHIMA (2018) IFHIMA Globl News. Sydney, NSW: FHIMA. Available at https://ifhimasitemedia.s3.us-east-2.amazonaws.com/wp-content/uploads/2018/07/20033704/IFHIMA-Newsletter-Issue1-2018_FINAL2-copy.pdf

13.

Korean Medical Record Association (2017) ICD-11 Second Field Trial. Seoul, Korea: Korean Medical Record Association.

14.

Krawczyk

Swiecicki

(2020) ICD-11 vs. ICD-10 - a review of updates and novelties introduced in the latest version of the WHO International Classification of Diseases. Psychiatr Pol 54(1): 7–20.

15.

Lou

Nyulas

, et al. (2017) Use of ontology structure and Bayesian models to aid the crowdsourcing of ICD-11 sanctioning rules. J Biomed Inform 68: 20–34.

16.

Nishio

Kimura

Seto

, et al. (2019) The Survey for Determining Knowledge-Related Problems in the Dissemination of ICD-11. Studies in Health Technology and Informatics 264: 1865–1866.

17.

Reed

Keeley

Rebello

, et al. (2018a) Clinical utility of ICD-11 diagnostic guidelines for high-burden mental disorders: results from mental health settings in 13 countries. World Psychiatry 17(3): 306–315.

18.

Reed

Sharan

Rebello

, et al. (2018b) The ICD-11 developmental field study of reliability of diagnoses of high-burden mental disorders: results among adult patients in mental health settings of 13 countries. World Psychiatry 17(2): 174–186.

19.

Sato

Mizushima

Ogata

, et al. (2019) ICD-11 Field Trial in Japan. Geneva, Switzerland: WHO|. Available at: https://www.whofic.nl/icd-11-field-trial-in-japan

20.

Suen

Lee

(1985) Effects of the use of percentage agreement on behavioral observation reliabilities: A reassessment. Journal of Psychopathology and Behavioral Assessment 7(3): 221–234.

21.

Tanpowpong

Lertudomphonwanit

Phuapradit

, et al. (2021) Value of the International Classification of Diseases code for identifying children with biliary atresia. Clin Exp Pediatr 64(2): 80–85.

22.

World Health Organization (WHO) . (2021) ICD-11 Reference Guide. Geneva, Switzerland: WHO|. Available at: https://icd.who.int/icd11refguide/en/index.html#1.2.4GeneralfeaturesofICD-11|extension-codes|c1-2-4-2