Utility of SNOMED CT in automated expansion of clinical terms in discharge summaries: Testing issues of coverage

Abstract

Objective:

This study tests coverage of SNOMED CT as an expansion source in the process of automated expansion of clinical terms found in discharge summaries. Term expansion is commonly used as a technique in knowledge extraction, query formulation and semantic modelling among other applications. However, characteristics of the sources might affect credibility of outputs, and coverage is one of them.

Method:

We developed an automated method for testing coverage of more than one source at a time. We used several methods to clean our corpus of discharge summaries before we extracted text fragments as candidates for clinical concepts. We then used Unified Medical Language System (UMLS) sources and UMLS REST API to filter concepts from the pool of text fragments. Statistical measures like true positive rate and false negative rate were used to decide on the coverage of the source. We also tested the coverage of the individual SNOMED CT hierarchies using the same methods.

Results:

Findings suggest that a combination of four terminologies tested (SNOMED CT, NCI, LNC and MSH) achieves over 90% of coverage for term expansion. We also found that the SNOMED CT hierarchies that hold clinically relevant concepts provided 60% of coverage.

Conclusion:

We believe that our findings and the method we developed will be of use to both scientists and practitioners working in the domain of knowledge extraction.

Keywords

SNOMED CT patient discharge summaries vocabulary semantics health information management

Introduction

SNOMED CT is claimed to be the most comprehensive single resource of clinical terms available today (IHTSDO, 2020a). It has been created and is maintained by the SNOMED International organisation that currently has 37 countries as governing members. The number of concepts in the January 2019 edition of this resource was 349,548 and it is showing a steady growth in the number of concepts added in every edition (IHTSDO, 2020b). Its content is clinically verified and enables accurate representation of clinical concepts in electronic clinical records and clinical analytics systems (Lee et al., 2013). SNOMED CT’s structure consists of 19 large hierarchies that are joined through the top-level concept, called the SNOMED CT Concept. The hierarchies are made of concepts connected using the Is_A attribute. Hierarchies themselves are large acyclic graphs in nature and no concept from any of the hierarchies connects to any of the concepts from the other 18 hierarchies. In addition to the Is_A attribute, which is used purely for depicting hierarchy, other attributes, listed in the SNOMED CT Model Component (metadata) hierarchy, are used to depict relationships. In contrast to the Is_A attribute, these attributes can “connect” concepts located in different hierarchies. For example, Finding site (attribute) is used to represent the relationship between the concepts Pulmonary infarction (disorder) and Structure of pulmonary blood vessel (body structure) that reside in different hierarchies.

The ability of SNOMED CT to provide axioms using a concept’s hierarchical and non-hierarchical relationships allows for reasoning and classifies SNOMED CT as an ontology rather than just a taxonomy (Bodenreider, 2018). This is supported by the fact that the axioms in SNOMED CT are presented using Compositional Grammar that can be interpreted using Description Logic (SNOMED International, 2019). Considering that the number of concepts covered in SNOMED CT is significant and that SNOMED CT is a quality-controlled resource created by experts and scrutinised by many, it can be assumed that the axioms postulated in SNOMED CT are a close representation of the propositions in reality or in its representative ontology. These characteristics make SNOMED CT a suitable source for use in knowledge extraction and for term expansion in particular. Term expansion is a frequently used technique for knowledge extraction. Term expansion is used to overcome vocabulary mismatch issues where expansion increases the scope of analysis to cover syntactic patterns that are not otherwise present in the text. This technique keeps compound or obscure terms from preventing or confusing concept identification, making processes like knowledge extraction more effective (Alani et al., 2003). In term expansion, a term is supplemented with associated terms found in a source selected as an expansion repository. This source can be any body of text; however, the most commonly used sources are domain-specific and domain-independent ontologies and databases. SNOMED CT, in particular, is used for work in the domain of medicine and human biology (Stokes et al., 2009; Wang et al., 2008).

As part of our work on annotation of clinical datasets using openEHR Archetypes (Zivaljevic et al., 2015), we need a repository that we can use as a source for expansion of clinical terms found in the free-text discharge summaries that we use as a sample. We expect that the source will be comprehensive and have its concepts and their relationships presented as closely as possible to reality, and thus be a valid representation of the clinical domain. Considering its comprehensiveness, quality control and involvement of clinicians in its creation, SNOMED CT is presumed to be an obvious choice. It is expected that its 19 hierarchies, that each cover different subdomains relevant to the clinical domain, will provide valuable guidance in noise minimisation efforts as they will allow for different weighting to be assigned to different (sub)domains of interest. However, the research community reports issues that can potentially render SNOMED CT unsuitable for term expansion. Rodrigues et al. (2018) raise the question of modelling issues in SNOMED CT and suggest quality assurance improvements. However, their conclusions are based on the limited sample, comparison of SNOMED CT concepts and mapped ICD11 classes from the circulatory and digestive chapters only.

Miñarro-Giménez et al. (2018) conducted a qualitative analysis testing the utility of SNOMED CT in coding clinical text by measuring inter-annotator disagreement. Their results show that their annotators matched what was defined in the reference standard only in 21.6% of the codes assigned to the terms found in the provided sample. This, as the authors term it, “astonishingly low” result could mean that SNOMED CT features a high level of ambiguity, which would render it unsuitable for our research due to the potential introduction of noise in the term expansion process. However, the issues that the authors list as factors causing this are mostly subjective or of a research methods nature and do not depict a failure of SNOMED CT. The factors include annotators’ lack of domain knowledge, carelessness, annotation guideline issues, interface term issues and language issues. Bona and Ceusters (2018) demonstrated that some of the concept tags in SNOMED CT incorrectly identify a concept’s place in the hierarchy. The semantic tag is part of the concept’s description and comes surrounded in parentheses. It functions to disambiguate the description of the particular concept from the descriptions of other concepts that might be the same but might belong to other concepts. The example that the authors give are the concepts [35566002 | Hematoma (morphologic abnormality)] and [385494008 | Hematoma (disorder)]. It was found that 89 of over 300,000 concepts in the SNOMED CT 20170131 release have mismatched semantic tags. However, based on the magnitude of the problem (or lack of it), we discard this issue as minor to our study.

The issue of SNOMED CT’s coverage, which we consider to be SNOMED’s ability to capture clinical information in a given text, has been assessed by many (Khorrami et al., 2018). Some deficiencies of SNOMED CT is reported to have practical implications on its utilisation in the clinical domain. For example, Liu et al. (2018) established that SNOMED CT’s adoption among ophthalmologists in the United States has not reached expected levels and find that the reason for that is the coverage of SNOMED CT’s ophthalmology component. Similarly, Rastegar-Mojarad et al. (2017) raised an issue of coverage in their attempt to map the list of procedures from their Gynaecology Surgery Registry to SNOMED CT. They found that only a small percentage of procedure names can be mapped to the concepts in SNOMED CT. The reason behind this issue was not the absence of a suitable terms in SNOMED CT but the format in which the procedures are represented and use of what they call procedure modifiers. This issue observed by Rastegar-Mojarad et al. is an important one as it indicates that the issues seen as SNOMED CT coverage could actually be unrelated to SNOMED CT and related to how concepts are presented in the corpus. This particular case highlighted the importance of stemming and lemmatisation of the text fragments that are to be mapped to the terms in SNOMED CT. In their comparison of Nomenclature for Properties and Units (NPU), Logical Observation Identifiers Names and Codes (LOINC) and SNOMED CT for use in representing laboratory results, Bietenbeck et al. (2018) suggest that for the best coverage SNOMED CT and LOINC should be used together as some of the concepts in SNOMED CT are represented in rather complex fashion, using post-coordinated, rather than pre-coordinated expression. However, they note that this particular issue is by design and is to ensure that duplication between SNOMED CT and LOINC is minimal.

The issues of SNOMED CT coverage raised in the literature have prompted us to question its suitability to our study. As our plan was to conduct ontological term expansion of the terms found in the sample corpus, we needed an ontology that incorporated as many of our target concepts as possible. Ideally, the ontology would have all of the target concepts incorporated, but the possibility for that to be the case was low. In this work, we tested the suitability of SNOMED CT for use in expansion of clinical concepts in terms of its coverage. The main goal was to test whether the coverage of SNOMED CT would allow for the majority (if not all) of the clinical terms from our sample corpus of discharge summaries to be recognised. We expected that at least 90% of the clinical terms found in the discharge summaries would be available in SNOMED CT. This was an arbitrary number based on a consensus of the research team members and work of the authors mentioned later in the text. If less than 90% coverage was found, we would consider terms from other ontologies for inclusion in SNOMED CT using extension.

Method

The sample we used in our study was the set of discharge summaries provided by the i2b2 National Center for Biomedical Computing (Uzuner et al., 2007). The sample contains 889 unannotated, de-identified discharge summaries provided as free text. We extracted each record’s text and stored it into an SQL database along with the related RECORD ID. The free text extracted needed to be cleansed of elements that did not hold any value in the process. This particular corpus had section captions that divided free text. The captions were not standard and varied from discharge summary to discharge summary. Due to the fact that the captions contained no medical concepts related to the patient health record, but generic descriptions of the sections (Diagnosis, Family History, etc.), they were temporarily removed from the corpus. Some value was seen in classification of the concepts recognised under the headings, but we ignored that knowing that the headings were specific to this particular corpus and might not be found in other corpuses of clinical free text. This step would most likely be different for different corpuses, due to structural variations that can be expected. We called this step Text Cleaning.

Boundary detection took place next. In this step, we split the text in the array of sentences. For that to be possible, line breaks were normalised and changed to full stops if they were not present and double spaces were replaced with single spaces. It was ensured that the sentence boundaries were kept intact so that multi-word text fragments would not cross them. The candidates for clinical concepts were extracted next in the step we called Fragments Extraction. The algorithm extracted text fragments that were up to five@@ words long, contained only a-z, A_Z, -, ', and space. We kept the interpunctuation in the sentence intact, which ensured that the extracted text fragments would not span across sentence segments. The minimal number of characters for the text fragments that were one word in length was set to three. The minimum length for the words in the text fragments made of two or more words has not been set. Text fragments are recorded once only, regardless of how many times they appeared in the corpus. However, the number of times that text fragments appeared in the corpus has been recorded. The algorithm used to extract the text fragments tokenised the text into one-word tokens and then, starting from each token, the algorithm selected all n-word text fragments up to n = 5. The duplicates were then removed and text fragments of one word in length were filtered for stop words. The fragments that contained more than one word were not filtered of stop words. The final array was cached into a database.

Due to our focus on expressiveness of terminology rather than higher level semantics of the text fragments retrieved, we were not concerned with recognising negation in the corpus. We viewed negation as a topic on its own and a question separate to this stage of our research. Each of the extracted text fragments was then used to formulate a query to the Unified Medical Language System (UMLS). UMLS Representational State Transfer REST application program interfaces APIs are used to query whether an exact match for the extracted text fragment exists in the UMLS Metathesaurus. If the exact match or matches existed, the source terminologies were recorded as associated with the text fragment used in the query. The information was returned by the UMLS in the JavaScript Object Notation (JSON) format and JSONs were saved in the database for future reference. If SNOMED CT was listed as a source terminology for the discovered concept, we made a recursive query that revealed to which of 19 SNOMED CT hierarchies the concept belonged. The version of SNOMED CT available in UMLS was the US Edition version 20190301.

As our aim was to learn how well SNOMED CT would perform as a term expansion source for our study, we wanted to establish the level of coverage that SNOMED CT would provide to our corpus. Elkin et al. (2006) conducted similar research testing how well SNOMED CT provided coverage for coding the most common conditions seen at one of their large clinics. They found that the coverage was 92.3% and declared that as sufficient coverage. Following Elkin et al., we used sensitivity as one measure. For us, sensitivity answered the question of how to quantify a UMLS source’s (the Source) coverage. For example, for SNOMED CT, we calculated Sensitivity Ratio as the number of concepts found in SNOMED CT divided by the number of concepts found in the UMLS as if it was without SNOMED CT. We calculated Sensitivity Ratio for every Source where at least one text fragment matched a concept (see Box 1, equation (1)). We also wanted to know the rate of false negatives as an indicator of the number of concepts that will not be recognised if the particular source is used. False negatives are the concepts found in the UMLS but not in the tested source divided by the total number of concepts found in the UMLS (see Box 1, equation (2) for calculation of false negatives). As we were also interested in the coverage of distinct SNOMED CT hierarchies (organised as SNOMED CT axes), we calculated sensitivity of each one of the SNOMED CT hierarchies. We defined sensitivity of each hierarchy is the number of concepts found in that hierarchy over the number of total SNOMED CT concepts recognised expressed as percentage (see Box 1, equation (3)).

Considering that the corpus of text we worked on consisted of discharge summaries, we expected that significant sensitivity would be shown by the hierarchy with “Clinical finding (finding)” as its root concept. We placed a threshold at 40% sensitivity per source ( ${TPR}_{{FNR}_{SNOMED}} \geq 40 %$ ). In other words, if the ratio between the number of false negatives found in the Source X and the total false negatives was equal to or over 0.4, we would consider inclusion of the recognised concepts from the Source X into the SNOMED CT extension along with their relationships with the SNOMED CT concepts as they are depicted in the UMLS (Box 1, equation (4)). If, after including sources with ${TPR}_{{FNR}_{SNOMED}} \geq 40 %$ , the coverage was still below 90%, we intended to keep including sources until the coverage result reaches 90%.

Box 1

Equation 1 – Sensitivity (true positives) $T P R_{S o u r c e} = \frac{T P_{S o u r c e}}{T P - U T P_{S o u r c e}}$
Term used in the formula	Description
${TPR}_{Source}$	Sensitivity (true positive rate)
${TP}_{Source}$	The number of true positives (text fragments matched to concepts) in the source
$TP$	Total number of text fragments matched to at least one concept in at least one source in UMLS – total number of true positives for all UMLS sources
${UTP}_{Source}$	Text fragments uniquely matched to the concepts in the Source only
Equation 2 – False negatives ${FNR}_{Source} = \frac{TP - {TP}_{Source}}{TP}$
Term used in the formula	Description
${FNR}_{Source}$	False negative rate
$TP$	Total number of text fragments matched to at least one concept in at least one source in UMLS – total number of true positives for all UMLS sources
${TP}_{Source}$	The number of true positives (text fragments matched to concepts in the source)
Equation 3 – Coverage of distinct SNOMED CT hierarchies (axes) ${TPR}_{Axis} = \frac{{TP}_{Axis}}{{TP}_{SNOMED}}$
Term used in the formula	Description
${TPR}_{Axis}$	Sensitivity of a particular SNOMED CT hierarchy (axis)
${TP}_{Axis}$	Total number of true positives for a particular SNOMED CT hierarchy (axis)
${TP}_{SNOMED}$	The number of true positives for the entire SNOMED CT source
Equation 4 – Coverage of other Sources over SNOMED CT’s false negatives ${TPR}_{{FNR}_{SNOMED}} = \frac{{TP}_{Not in SNOMED}}{TP - {TP}_{SNOMED}}$
Term used in the formula	Description
${TPR}_{{FNR}_{SNOMED}}$	Coverage of SNOMED CT false negatives
${TP}_{Not in SNOMED}$	The number of text fragments matched to concepts in the Source and not matched to concepts in SNOMED CT
$TP$	Total number of text fragments matched to at least one concept in at least one source in UMLS – total number of true positives for all UMLS sources
${TP}_{SNOMED}$	The number of text fragments found in SNOMED CT
Equation 5 – Specificity formula ${Specificity}_{Source} = \frac{{TN}_{Source}}{{TN}_{Source} + {FP}_{Source}}$
Term used in the formula	Description
${Specificity}_{Source}$	Specificity for a Source
${TN}_{Source}$	The number of true negatives for a Source
${FP}_{Source}$	The number of false positives for a Source

A question may arise as to why we did not use Specificity as a measure in this work. Specificity takes into consideration true negatives, and in the case at hand, Specificity would have revealed the proportion between the number of text fragments that were correctly identified to have no meaning as concepts (true negatives) and the total number of text fragments that truly had no meaning as concepts (true negatives + false positives) (Box 1, equation (5)). We were able to calculate the number of true negatives for a Source as the number of text fragments that have not been recognised, neither in a particular Source nor in one of the other Sources. However, establishing a reliable number for the false positives was not possible without human agents. The question of whether a recognised match between a text fragment and a concept in a Source is a true negative or a false positive cannot be answered using automated methods at this point in time.

Results

Out of 208,513 text fragments of 1–5 words length, 9777 matched to at least one concept in at least one UMLS source. The number of UMLS sources that the matches were found in was 85. The top 5 sources were Consumer Health Vocabulary (CHV), National Cancer Institute (NCI), SNOMED CT US Edition (SNOMEDCT_US), LOINC (LNC) and MeSH (MSH).

Table 1 shows the experiment results for the top five sources. The field “Concepts” shows the number of text fragments recognised as concepts in a particular Source. The next field, named “Unique,” shows the number of text fragments recognised as concepts only in one particular source. Sensitivity Ratio is given in the next field (TPR Source). It is explained in equation (1) as the number of concepts found in the Source divided by the number of concepts found in the UMLS as if it was without that particular Source. The next field – false negatives is given in equation (2) as the number of concepts found in the UMLS but not in the tested source. And the last field in this table shows false negatives of the Source in relation to the false negatives of SNOMED CT.

Table 1.

Top 5 UMLS sources for concept matching.

Source	Concepts	Unique	TPRSource (%)	FNRSource (%)	${TPR}_{{FNR}_{SNOMED}}$ (%)
CHV	6932	916	78.23	29.10	57.91
NCI	4750	642	52.00	51.42	44.10
SNOMED CTUS	4750	261	49.92	51.42	0.00
LNC	2597	144	26.96	73.44	15.66
MSH	2500	74	25.77	74.43	20.33

CHV: Consumer Health Vocabulary; NCI: National Cancer Institute; SNOMED CT-US: SNOMED CT US Edition; LNC: LOINC; MSH: MeSH; Concepts: the number of text fragments recognised as concepts in the source; Unique: the number of text fragments only in the one particular source; TPR: true positive ratio (sensitivity); FNR: false negative ratio; ${TPR}_{{FNR}_{SNOMED}}$ : false negatives of the source in relation to the false negatives of SNOMET CT.

Table 2 shows the results of the combined sources. The first field lists the number of text fragments recognised as concepts by the relevant combination of Sources. The next field shows the number of concepts that are uniquely recognised when the particular Sources are combined. Sensitivity and false negatives for the particular combination of the sources are given as the last two fields in this table (description as for the fields in Table 1). Table 3 shows a sample of text fragments, their occurrences in the discharge summaries and the number of matching concepts in each of the top 5 Sources. Table 4 shows a sample of text fragments that have been matched to concepts in Sources other than SNOMED CT, their occurrences in the discharge summaries and the number of matching concepts in each of the top 5 Sources. Table 5 depicts the result of the coverage of distinct SNOMED CT hierarchies organised as SNOMED CT axes. It lists the SNOMED CT hierarchy axis followed by the number of concepts that belong to the particular SNOMED CT hierarchy axis and its coverage.

Table 2.

Selected sources and results of source combinations.

Source	Concepts	Unique	TPRSource (%)	FNRSource (%)
SNOMED CT + NCI	6967	903	78.51	28.74
SNOMED CT + NCI + LNC	7356	1047	84.26	24.76
SNOMED CT + NCI + LNC + MSH	7889	1121	91.13	19.31

SNOMED CT-US: SNOMED CT US Edition; NCI: National Cancer Institute; LNC: LOINC; MSH: MeSH; Concepts: the number of text fragments recognised as concepts in the source; Unique: the number of text fragments only in the one particular source; TPR: true positive ratio (sensitivity); FNR: false negative ratio.

Table 3.

Sample of matched text fragments, their occurrences in the test and the number of matching concepts in each source.

Text fragment	Occurrences	CHV	NCI	SNOMEDCT_US	LNC	MSH
Report	862	2	1	1	5	1
Status	862	1	1	1	1	0
Discharge	831	3	4	3	4	0
Patient	723	1	2	1	3	1
Summary	661	0	1	0	1	0
Pain	518	1	6	1	3	1
Stable	513	1	2	1	1	0
Admission	511	1	1	0	1	0

CHV: Consumer Health Vocabulary; NCI: National Cancer Institute; SNOMED CT-US: SNOMED CT US Edition; LNC: LOINC; MSH: MeSH.

Table 4.

Sample of text fragments matched to sources other than SNOMED CT.

Text fragment	Occurrences	CHV	NCI	LNC	MSH
Summary	661	0	1	1	0
Admission	511	1	1	1	0
Not	504	0	1	1	0
Admitted	479	1	0	2	0
Discharged	463	0	1	1	0
Treatment	431	1	3	3	2
Room	407	0	1	2	0
Service	386	1	1	2	0

CHV: Consumer Health Vocabulary; NCI: National Cancer Institute; SNOMED CT-US: SNOMED CT US Edition; LNC: LOINC; MSH: MeSH.

Table 5.

Coverage of SNOMED CT axes.

SNOMED CT Axes	Concepts	TPRAxis (%)
Qualifier value (qualifier value)	1545	32.53
Clinical finding (finding)	1087	22.88
Substance (substance)	688	14.48
Body structure (body structure)	617	12.99
Procedure (procedure)	430	9.05
Physical object (physical object)	195	4.11
Observable entity (observable entity)	187	3.94
SNOMED CT Model Component (metadata)	129	2.72
Social context (social concept)	116	2.44
Organism (organism)	109	2.29
Environment or geographical location (environment/location)	100	2.11
Physical force (physical force)	17	0.36
Event (event)	13	0.27
Pharmaceutical/biologic product (product)	10	0.21
Situation with explicit context (situation)	9	0.19
Staging and scales (staging scale)	7	0.15
Record artefact (record artefact)	4	0.08
Specimen (specimen)	3	0.06
Special concept (special concept)	3	0.06

Note. TPR_Axis is the total number of true positives for a particular SNOMED CT hierarchy (axis). It represents the number of concepts found in that hierarchy over the number of total SNOMED CT concepts recognised, expressed as percentage.

Discussion

In this study, we demonstrated a fully automated approach for selecting sources for term expansion based on their coverage of a corpus of clinical text. Similar approaches have been described in the past, some as theoretical concepts or frameworks (Alani et al., 2003; Li and Motta, 2010), some as solutions that require input of a human agent (Maiga and Ddembe, 2009) and some as complex sets of algorithms that include more than just a test of coverage in the process of source evaluation (Martínez-Romero et al., 2014, 2017). However, the method we developed differs from these other methods in several respects. Firstly, it includes consideration for preparation of text for coverage testing. As we expected that the input would be free text, we allowed for text cleaning, stop word removal and tokenisation. Secondly, we deployed a version of brute-force testing of text fragments for presence in the sources. Starting from our one-word text fragments, we extracted all multi-word text fragments up to the length of 5 and tested them all for presence in the sources. We used caching to ensure that there was no performance penalty for doing so. Thirdly, our measure for selection of the sources to extend SNOMED CT as the preferred source was the sources’ coverage of SNOMED CT’s false negatives – ${TPR}_{{FNR}_{SNOMED}}$ . This measure allowed us to find the sources that contained concepts that were the best match for the concepts missed by SNOMED CT. We also made sure that the concepts from other sources selected to supplement SNOMED CT did not overlap as we checked coverage for false negatives after adding every new ontology to the mix. We also showed that we were able to use this methodology to test coverage of SNOMED CT and to find sources whose concepts would be good contributors if included as source extension. A surprise finding in our study was the level of coverage provided by the open source Consumer Health Vocabulary (CHV). At 78.23%, this source indeed showed as the best choice for use in our study. However, our research also indicates that the open source project that has created it is not active and that raises concerns for future support and expansion. Moreover, the source was created with the purpose of helping clinicians to communicate complex health-related concepts to their patients (Mujib et al., 2018) and had no ontological properties. Due to these reasons, we decided not to use CHV as the main source in our next study.

We also considered using CHV as a source for extending the selected source. However, despite offering an impressive 916 concepts that were unique (not present in other sources), and 57.91% of coverage of SNOMED CT’s false negatives, CHV’s lack of ontological properties made us decide against using it. Although the richness of this source would certainly have increased the number of text fragments recognised as concepts, its lack of ontological properties made it unsuitable for use in semantics operations.

NCI and SNOMED CT were found to have the same number of text fragments recognised as concepts. Accordingly, their number of false negative results, represented as ${FNR}_{Source}$ was the same at 51.42%. Coincidence is the only explanation we can offer for the same number of recognised concepts in the two sources that are known to have just 17.1% overlap (NLM.govt, 2019b). This result prompted us to rerun the experiment several times and to reconsider the source code. However, results were confirmed to be correct. The number of recognised unique concepts in NCI was significantly higher than in SNOMED CT or any other sources, even in CHV if considered relative to the number of recognised concepts (13.5% vs. 13.2%). The difference in the number of unique concepts between NCI and SNOMED CT is reflected in the slight difference in the ${TPR}_{Source}$ value, where NCI has achieved a slightly better result. Considering that NCI is a terminology source highly specialised in oncology and based on the higher number of concepts found in NCI than in SNOMED CT, an assumption can be made that SNOMED CT does not have good coverage of oncology specialised concepts found in the corpus. This assumption would be in line with the reports on SNOMED CT’s low coverage of cancer-related concepts. For example, Raje and Bodenreider (2017) compared Disease Ontology (Schriml et al., 2012) and SNOMED CT and found that many of the Disease Ontology’s cancer and neoplasm concepts were not found in SNOMED CT. Similarly, Melton et al. (2006) surveyed SNOMED CT for coverage of colorectal cancer surgery concepts and found that equipment and finding concepts were insufficiently represented. However, as our test results have demonstrated, none of the text fragments that our algorithm recognised as concepts in NCI and not in SNOMED CT are concepts specific to the field of oncology. Although not sufficient to provide an indication on adequacy of SNOMED CT coverage of oncology-related concepts (unless NCI is used as a gold standard), this observation might be indicating that SNOMED CT’s coverage of oncology concepts is in line with that of NCI’s. However, further testing, using methods other than deployed in this project will be needed before any conclusions on sufficiency of SNOMED CT’s coverage of these concepts or comparison of coverages of SNOMED CT and NCI are made.

Nevertheless, some conclusions on the utility of extending SNOMED CT with the concepts from NCI can be made. NCI’s ${TPR}_{{FNR}_{SNOMED}}$ value was also substantial at 44.10%, implying that almost half of the concepts found in NCI were not present in SNOMED CT and vice versa. This indicates that these two sources would be good contributors to one another. We calculated that, if the terms from NCI are used as an extension to SNOMED CT, the resulting source would provide a total positive result of 78.51% and reduce the number of false negatives to 28.74%.

The number of concepts recognised by LOINC and MeSH was relatively small compared to the top 3 sources. However, their use as extensions of the main source is worth considering. LOINC’s overlap with SNOMED CT is small at 6.7% (NLM.govt, 2019a) and considering its specialisation in laboratory results, we saw it as a good candidate for using as an extension to the main source. MeSH’s overlap with SNOMED CT was also small at 8.1% and the prospect of the increase in coverage of close to 7% prompted us to decide that its concepts are worth inclusion. The next source with the largest number of concepts that are not contained in SNOMED CT, NCI, LOINC and MeSH was MEDCIN. The number of concepts that MEDCIN would contribute would be 145 with 87 unique. That number of concepts does not significantly improve the resulting ontology, so we decided that it was not worth including.

Although our results showing the root axes of SNOMED CT terms revealed that the greatest number of results came from the Qualifier value hierarchy, close to 60% of the concepts came from the next four hierarchies (clinical finding, substance, body structure and procedure), which contained the concepts of highest relevance for expansion of clinical concepts. This clearly shows the need for assigning to, or at least adjusting weight of, the recognised concepts based on the semantics relevance of their roots.

Study limitations

The algorithm we used for preparing text fragments for matching to concepts did not stem the text fragments before they were sent to UMLS for matching. Stemming is used to reduce variant word forms to common roots (Xu and Croft, 1998). Our tests conducted during the experiment showed that deploying the Porter Stemmer (Porter, 1980) would improve recognition of the text fragments by 8–10%. We tested just the text fragments that have been found in at least one of the sources. We believe that the utility of stemming should be further evaluated and that various stemming algorithms should be included in the testing process. Also, our focus was on expressiveness of terminology rather than higher level semantics of the text fragments retrieved. Hence, we were not concerned with recognising negation in the corpus nor any effort was put in deciphering exact semantics of the text fragments.

Conclusion

We believe that all projects that utilise ontologies for term expansion, concept recognition, coding or similar applications need to undergo an exercise similar to what we have described in our current work. The unique contribution our research offers is the description and justification of the methodology used. We believe a similar process can be employed by researchers in many other projects. We also believe that the use of UMLS as a source of truth in an automated process of discovering true positives and false negatives is novel and that, as source ontologies of the UMLS grow, the utility of UMLS for this purpose will increase.

Footnotes

Acknowledgements

Deidentified clinical records used in this research were provided by the i2b2 National Center for Biomedical Computing funded by U54LM008748 and were originally prepared for the Shared Tasks for Challenges in NLP for Clinical Data organised by Dr. Ozlem Uzuner, i2b2 and SUNY.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Aleksandar Zivaljevic, MBA, MMgt(Health)

Koray Atalag, MD, PhD, FACHI

James Warren, PhD, FACHI

References

Alani

Kim

Sanghee

Millard

, et al. (2003) Automatic ontology-based knowledge extraction from web documents. IEEE Intelligent Systems 18(1): 14–21.

Bietenbeck

Boeker

Schulz

(2018) NPU, LOINC, and SNOMED CT: a comparison of terminologies for laboratory results reveals individual advantages and a lack of possibilities to encode interpretive comments. LaboratoriumsMedizin 42(6): 267–275.

Bodenreider

(2018) The New SNOMED CT International Medicinal Product Model. In: Corvallis, Oregon, USA, 2018. International Conference on Biological Ontology 2018. Available at: http://ceur-ws.org/Vol-2285/ICBO_2018_paper_36.pdf. (accessed 21 January 2016)

Bona

Ceusters

(2018) Mismatches between major subhierarchies and semantic tags in SNOMED CT. Journal of Biomedical Informatics 81: 1–15.

Elkin

Brown

Husser

, et al. (2006) Evaluation of the content coverage of SNOMED CT: ability of SNOMED clinical terms to represent clinical problem lists. Mayo Clinic Proceedings 81(6): 741–748.

International Health Terminology Standards Development Organization (IHTSDO) (2020a) 4. SNOMED CT Basics – SNOMED CT Starter Guide - SNOMED Confluence. Available at: https://confluence.ihtsdotools.org/display/DOCSTART/4.+SNOMED+CT+Basics (accessed 10 May 2020).

International Health Terminology Standards Development Organization (IHTSDO) (2020b) SOMED international extensions practical guide. Available at: https://confluence.ihtsdotools.org/display/DOCEXTPG/Extensions+Practical+Guide (accessed 22 April 2020).

Khorrami

Ahmadi

Sheikhtaheri

(2018) Evaluation of SNOMED CT content coverage: a systematic literature review. Studies in health technology and informatics 248: 212.

Lee

Cornet

Lau

, et al. (2013) A survey of SNOMED CT implementations. Journal of Biomedical Informatics 46(1): 87–96.

10.

Motta

(2010) Evaluations of user-driven ontology summarization. In: Cimiano

Pinto

(eds) Knowledge Engineering and Management by the Masses. Berlin, Heidelberg: Lecture Notes in Computer Science. Springer, pp. 544–553.

11.

Liu

Hildebrand

Perl

, et al. (2018) Enrichment of SNOMED CT ophthalmology component to support EHR coding. In: 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), December 2018, pp. 1990–1997. DOI: 10.1109/BIBM.2018.8621272.

12.

Maiga

Ddembe

(2009) A flexible biomedical ontology selection tool. Available at: http://makir.mak.ac.ug/handle/10570/2021 (accessed 7 July 2019).

13.

Martínez-Romero

Jonquet

O’Connor

, et al. (2017) NCBO ontology recommender 2.0: an enhanced approach for biomedical ontology recommendation. Journal of Biomedical Semantics 8(1): 21.

14.

Martínez-Romero

Vázquez-Naya

Pereira

, et al. (2014) BiOSS: a system for biomedical ontology selection. Computer Methods and Programs in Biomedicine 114(1): 125–140.

15.

Melton

Morrison

Cimino

, et al. (2006) How well do electronic systems represent colorectal cancer surgery concepts? Evaluation of SNOMED-CT, ICD9-CM, and CPT-4 for content coverage. Journal of the American College of Surgeons 203(3, Supplement). Abstracts for the 61st Annual Sessions of the Forum on Fundamental Surgical Problems S69–S70. DOI: 10.1016/j.jamcollsurg.2006.05.182.

16.

Miñarro-Giménez

Martínez-Costa

Karlsson

, et al. (2018) Qualitative analysis of manual annotations of clinical text with SNOMED CT. PLOS ONE 13(12): e0209547.

17.

Mujib

Yang

Zhao

, et al. (2018) Expanding consumer health vocabularies with frequency-conserving internal context models. In: 2018 IEEE International Conference on Healthcare Informatics (ICHI), June 2018, pp. 241–246. DOI: 10.1109/ICHI.2018.00034.

18.

NLM.govt (2019a) UMLS Metathesaurus – LNC (LOINC) – Statistics. Available at: https://www.nlm.nih.gov/research/umls/sourcereleasedocs/current/LNC/stats.html (accessed 30 June 2019).

19.

NLM.govt (2019b) UMLS Metathesaurus – NCI (NCI Thesaurus) – Statistics. Available at: https://www.nlm.nih.gov/research/umls/sourcereleasedocs/current/NCI/stats.html (accessed 30 June 2019).

20.

Porter

(1980) An algorithm for suffix stripping. Program 14(3): 130–137.

21.

Raje

Bodenreider

(2017) Interoperability of disease concepts in clinical and research ontologies: contrasting coverage and structure in the disease ontology and SNOMED CT. Studies in health technology and informatics 245: 925–929.

22.

Rastegar-Mojarad

Sohn

Wang

, et al. (2017) Need of informatics in designing interoperable clinical registries. International journal of medical informatics 108: 78–84.

23.

Rodrigues

Schulz

Mizen

, et al. (2018) Is the application of SNOMED CT concept model sufficiently quality assured? AMIA Annual Symposium Proceedings 2017: 1488–1497.

24.

Schriml

Arze

Nadendla

, et al. (2012) Disease Ontology: a backbone for disease semantic integration. Nucleic Acids Research 40(D1): D940–D946.

25.

SNOMED International (2019) Compositional Grammar – Specification and Guide. Available at: https://confluence.ihtsdotools.org/display/DOCSCG (accessed 28 April 2019).

26.

Stokes

Cavedon

, et al. (2009) Exploring criteria for successful query expansion in the genomic domain. Information Retrieval 12(1): 17–50.

27.

Uzuner

Luo

Szolovits

(2007) Evaluating the state-of-the-art in automatic de-identification. Journal of the American Medical Informatics Association 14(5): 550–563.

28.

Wang

Patrick

Miller

, et al. (2008) A computational linguistics motivated mapping of ICPC-2 PLUS to SNOMED CT. BMC Medical Informatics and Decision Making 8(1): S5.

29.

Croft

(1998) Corpus-based stemming using cooccurrence of word variants. ACM Trans. Inf. Syst 16(1): 61–81.

30.

Zivaljevic

Atalag

Warren

, et al. (2015) Annotation of clinical datasets using openEHR Archetypes as a solution for data access issues faced in biomedical projects. Health Informatics New Zealand 2015. Available at: https://www.researchgate.net/profile/Aleksandar_Zivaljevic/publication/282278618_Annotation_of_clinical_datasets_using_openEHR_Archetypes_as_a_solution_for_data_access_issues_faced_in_biomedical_projects/links/560a475b08ae576ce63fbbfd.pdf (accessed 21 January 2016).