Applying Object Detection and Large Language Model to Establish a Smart Telemedicine Diagnosis System with Chatbot: A Case Study of Pressure Injuries Diagnosis System

Abstract

Background:

The scarcity of medical resources and personnel has worsened due to COVID-19. Telemedicine faces challenges in assessing wounds without physical examination. Evaluating pressure injuries is time consuming, energy intensive, and inconsistent. Most of today's telemedicine platforms utilize graphical user interfaces with complex operational procedures and limited channels for information dissemination. The study aims to establish a smart telemedicine diagnosis system based on YOLOv7 and large language model.

Methods:

The YOLOv7 model is trained using a clinical data set, with data augmentation techniques employed to enhance the data set to identify six types of pressure injury images. The established system features a front-end interface that includes responsive web design and a chatbot with ChatGPT, and it is integrated with a database for personal information management.

Results:

This research provides a practical pressure injury staging classification model with an average F1 score of 0.9238. The system remotely provides real-time accurate diagnoses and prescriptions, guiding patients to seek various medical help levels based on symptom severity.

Conclusions:

This study establishes a smart telemedicine auxiliary diagnosis system based on the YOLOv7 model, which possesses capabilities for classification and real-time detection. During teleconsultations, it provides immediate and accurate diagnostic information and prescription recommendations and seeks various medical assistance based on the severity of symptoms. Through the setup of a chatbot with ChatGPT, different users can quickly achieve their respective objectives.

Introduction

Pressure injuries commonly afflict bedridden individuals, and older patients and those with multiple underlying health conditions face an increased susceptibility to this condition because of their delicate skin and impaired blood circulation.¹ The consequences associated with pressure injuries are notably more serious, encompassing conditions such as cellulitis, sepsis, and osteomyelitis. This not only results in a deterioration of the patient's health but also places additional strain on both family members and health care providers, leading to increased medical costs.²

In recent years, the ongoing COVID-19 pandemic has worsened the scarcity of medical resources and health care professionals in various regions. Patients who are concerned about crowded health care facilities or reside at a distance from medical centers have turned to telemedicine as an alternative solution. This potential was substantiated across multiple medical domains, encompassing pediatric care,³ diabetes,⁴ and obstructive sleep apnea.⁵ However, telemedicine encounters difficulties in assessing wounds because it lacks the ability to physically examine the affected area.

The GPT model is designed for text generation and language understanding tasks in natural language processing (NLP). ChatGPT is developed based on the transformer architecture, aiming to overcome some limitations of previous sequence-to-sequence models used in NLP.⁶ ChatGPT is a significant large language model (LLM) released by OpenAI. It is trained on a vast corpus of text data, enabling the model to converse with human interlocutors and provide human-like responses to various inquiries, effectively acting as a conversational agent.⁷

The absence of chatbots on telemedicine platforms can lead to burdensome and overly complicated operational processes, which make it challenging for users to obtain the precise information they require. Despite the existence of globally accepted criteria for assessing the stages of pressure injuries, the current health care system still relies on manual approaches for evaluating these wounds. This not only consumes a significant amount of time and effort for medical professionals but also presents challenges in maintaining consistent judgment.

This research aims to integrate the YOLOv7 model with LLM to develop a smart telemedicine diagnosis system. The system is designed for the efficient identification and classification of pressure injuries, utilizing the real-time detection capabilities of the YOLOv7 model to analyze pressure injury images quickly and accurately. The system's front end displays the identification results through a responsive web interface and builds a personalized database based on medical records. Moreover, the integrated LLM mechanism enables a chatbot to provide users with intuitive and immediate consultation services through conversational interactions.

The application of this system is intended to assist medical professionals in making precise diagnoses and treatment decisions within both clinical and remote consultation settings, thereby fulfilling the objectives of smart health care and telemedicine.

The sections of this article are organized as follows: Method Section details the proposed research methodological framework, Case Study section provides a case study of the pressure injury data set provided by the hospital, Discussion section compares the results with previous studies, and Conclusion section presents the conclusion.

Methods

The proposed methodology can be divided into four phases: (1) the smart telemedicine diagnosis system platform is built, including an explanation of the system database architecture and system architecture concepts; (2) the deep learning model is built, and the YOLOv7 model is introduced; (3) the model validation is conducted, and the model recognition performance of YOLOv7 is analyzed through a confusion matrix with six common evaluation metrics, precision, recall, average precision (AP), F1 score, sensitivity, and specificity to evaluate the overall model performance; and (4) the chatbot is constructed to achieve profile editing, pressure injury classification, provision of wound stage information, and communication between doctors and patients. The research framework is shown in Figure 1.

Fig. 1.

Research framework.

SMART TELEMEDICINE DIAGNOSIS SYSTEM PLATFORM CONSTRUCTION

Model-view-controller (MVC) is an architectural design structure implemented as a point-of-concern separation structure.⁸ The MVC design allows the system to be highly scalable, easy to manage, and more conducive to the team's division of labor. In this study, the smart telemedicine diagnosis system is divided into three parts with reference to MVC structure: model (MySQL), view (Web pages), and controller (PHP). The smart telemedicine diagnosis system model part of this study, which is the MySQL database management system, has the system database architecture as shown in Figure 2.

Fig. 2.

System database architecture.

DEEP LEARNING MODEL CONSTRUCTION

In this study, YOLOv7, a deep learning model with classification and real-time judgment capability, is added to the smart telemedicine diagnosis system established. Whenever a patient uploads a picture or video of a pressure injury wound, YOLOv7 can automatically recognize the stage of the wound in the picture or video and present the wound location information to help patients record and obtain the actual situation of their own pressure injury wounds faster and more accurately. Compared with previous object detection models, YOLOv7 achieves the best performance in recognition speed and precision between 5 frames per second (FPS) and 160 FPS.⁹ The difference lies in YOLOv7's backbone and scaling sections, where it adopts the extended-ELAN architecture and composite model scaling techniques. This approach ensures effective learning and convergence while maintaining the optimal structure.

MODEL VALIDATION

The confusion matrix is used in this study to evaluate model identification performance.¹⁰ The confusion matrix shows four results: true positive (TP), true negative (TN), false positive (FP), and false negative (FN). After obtaining the confusion matrix, we can further calculate three common object detection metrics: precision in Equation (1), recall in Equation (2), AP in Equation (3), and F1 score in Equation (4), and two common measurement indicators in medicine: sensitivity [Equation (5)] and specificity [Equation (6)]. All six metrics are between 0 and 1, and the higher the value, the better the performance of the model. The six equations are as follows. $P r e c i s i o n = \frac{T P}{T P + F P}$ (1)

R e c a l l = \frac{T P}{T P + F N}

(2)

F 1 S c o r e = \frac{2 * P r e c i s i o n * R e c a l l}{P r e c i s i o n + R e c a l l}

(4)

S e n s i t i v i t y = \frac{T P}{T P + F N}

(5)

S p e c i f i c i t y = \frac{T N}{F P + T N} .

(6)

CHATBOT

We employ the GPT-3.5 model⁷ to develop a user-guided chatbot. GPT-3.5, constructed upon the transformer architecture, a self-attention neural network framework, represents an advancement over its predecessor, GPT-3. Distinctively, GPT-3.5 undergoes training through reinforcement learning from human feedback, which involves the application of reward and punishment mechanisms to fulfill designated tasks. This approach facilitates the iterative refinement of the model based on feedback outcomes, thereby enhancing its capabilities in efficiently processing textual sequences, comprehending contextual nuances, and producing responses with augmented precision.¹¹

In this study, the chatbot comes preconfigured to serve three major user categories: “health care professionals,” “patients and family members,” and “the general public.” It provides tailored information to each category, ultimately aiming to improve the overall usability of the platform.

Case Study

This case study revolves around a hospital, which has grappled with a severe shortage of medical staff in recent years, primarily due to the impact of the COVID-19 pandemic. Therefore, this research has introduced an intelligent remote medical assistance and diagnosis system underpinned by the YOLOv7 deep learning model, renowned for its classification and real-time decision-making capabilities.

SMART TELEMEDICINE DIAGNOSIS SYSTEM

Following the system database architecture shown in Figure 3, this study develops a smart telemedicine diagnosis system. The application process is as follows: (1) Patients first create an account and fill in basic personal information such as gender, age, height, and weight, which could be implanted to a part of an electronic health record (EHR) for later reference and be used as a reference for treatment and medication recommendations if needed; (2) patients upload photographs or videos of their pressure injury wounds to the front end of the smart telemedicine diagnosis system. The front end then sends the patient's relevant data and pressure injury photographs or video data to the back end; (3) based on the YOLOv7 deep learning model with classification and real-time judgment capabilities, the back end starts to recognize the category of the pressure injury wound and tracks its location information based on the uploaded photograph or video data.

Fig. 3.

Schematic diagram of the smart telemedicine diagnosis system.

The analysis results are then provided to the front end; and (4) finally, the front end provides the pressure injury identification results, the stage of the wound, the treatment method for that stage, and suggested medication information to both the doctor and patient. Doctors can also use this information to determine whether there are any omissions and provide more professional information to the patient.

MODEL FOR PRESSURE INJURIES CLASSIFICATION

This study utilizes the pressure injury data set provided by the hospital. The data set includes six categories of pressure injuries: 37 photographs of stage 1 pressure injuries, 146 photographs of stage 2 pressure injuries, 160 photographs of stage 3 pressure injuries, 60 photographs of stage 4 pressure injuries, 59 photographs of deep tissue injury (DTI) pressure injuries, and 67 photographs of unstageable pressure injuries. The Roboflow^© software was used for annotation and data augmentation of the pressure injury data set.

The data set of 561 photographs was split into training, validation, and testing sets in an 8:1:1 ratio. Data augmentation techniques, including horizontal and vertical flips, 90°, 180°, and 270° rotations, brightness increase of 0–30%, and mosaic method, were applied, resulting in a total of 2,240 images in the training set.

This study employs YOLOv7 to classify pressure injury wounds, with input images resized to 640 × 640 pixels. The model was trained using Python 3.7 in a Windows 10 environment, on a computer with an Intel Core i5-8400 2.8 GHz CPU and NVIDIA GeForce RTX 2080 Ti GPU. The training lasted for 100 epochs, with each experiment taking ∼2.5 h.

MODEL TRAINING AND RESULTS

The YOLOv7 model adopted in this study achieves optimal classification performance when trained with an Adam optimizer, a learning rate of 10⁻³, a batch size of 18, and 100 epochs. From the confusion matrix obtained from the test set, it can be seen that the recall values for all types of pressure injuries except for stage 2 are 1. The recall value for stage 2 is 0.94. The confusion matrix for the classification performance of each type of pressure injuries is given in Table 1.

Table 1.

Confusion Matrix of the Test Set

ACTUAL	STAGE 1	STAGE 2	STAGE 3	STAGE 4	DTI	UNSTAGEABLE	BACKGROUND
Predicted
Stage 1	1.00	0	0	0	0	0	0.25
Stage 2	0	0.94	0	0	0	0	0.25
Stage 3	0	0	1.00	0	0	0	0.50
Stage 4	0	0	0	1.00	0	0	0
DTI	0	0	0	0	1.00	0	0
Unstageable	0	0	0	0	0	1.00	0
Background	0	0.06	0	0	0	0	0

DTI, deep tissue injury.

From the results, there is a small probability for stage 2 pressure injuries to be ignored by the classification model, but it is not a classification error; the other types of pressure injuries are correctly classified and won't be ignored by the classification model. In conclusion, the YOLOv7 model trained in this study has a good performance with an F1 score value of 0.9238 in the task of pressure injuries classification.

OPERATION PROCEDURE OF SYSTEM

This study creates a visual representation of the operational flowchart, demonstrating the interrelations among the three group: health care professionals, patients along with their families, and the general public. The schematic diagram of the chatbot assisting users in operating the platform system is shown in Figure 4. Figure 5 presents a visual representation of the interconnected operational processes among these three user categories.

Fig. 4.

User interaction with the GPT-3.5 model-based chatbot interface.

Fig. 5.

System operation procedures for three types of users.

Discussion

We compare with previous studies that used different classification models for pressure injuries classification tasks, such as MobileNetV2,¹² Faster R-CNN,¹³ Logistic regression, artificial neural network,¹⁴ and SE-ResNext101.¹⁵ Our study outperformed previous research in terms of classification performance. The comparison results are given in Table 2.

Table 2.

Performance Comparison of Pressure Injuries Classification with Related Research

RESEARCH	CLASSIFICATION MODEL	AVERAGE F1 SCORE
Orciuoli et al. (2020)¹²	MobileNetV2	0.6–0.86
Fergus et al. (2022)¹³	Faster R-CNN	0.6956
Yilmaz et al. (2021)¹⁴	Logistic regression	0.7420
Yilmaz et al. (2021)¹⁴	Artificial neural network	0.7390
Kim et al. (2023)¹⁵	SE-ResNext101	0.7910
Our research	YOLOv7	0.9238

In addition, chatbot is based on ChatGPT to accommodate various user requirements, providing a range of workflow guidance options to streamline user operations and alleviate operational challenges. The comparative outcomes concerning previous research are summarized in Table 3.

Table 3.

Comparison Between This Study and Related Research

RESEARCH	PRESSURE INJURIES WOUND STAGE IDENTIFICATION	CONSTRUCTION OF TELEMEDICINE PLATFORM	ChatGPT ASSISTANCE
Yilmaz et al. (2021)¹⁴	V
Orciuoli et al. (2020)¹²	V	V
Fergus et al. (2022)¹³	V	V
Our research	V	V	V

V indicates content that is involved.

Conclusions

To ensure precise and consistent evaluation of pressure injury stages during medical consultations or telemedicine sessions, along with delivering timely medical assistance and care, this study develops a smart telemedicine diagnosis system founded on the YOLOv7 deep learning model equipped with classification and real-time judgment capabilities. The system's user interface integrates LLM and responsive web pages.

Academically, this research brings about two noteworthy contributions: (1) it tackles the accuracy challenges observed in prior studies concerning pressure injuries stage with an average F1 score of 0.9238 in classification and (2) compared with previous telemedicine platforms used for pressure injury recognition, the proposed telemedicine system incorporates a chatbot with ChatGPT, transforming the platform from a graphical user interface to a conversational user interface, making it more user friendly.

From a practical standpoint, this study offers four pivotal contributions: (1) it introduces a pragmatic model for classifying pressure injury stages, aiding health care professionals in making swift and accurate decisions and executing suitable medical interventions based on specific symptoms; (2) through the implementation of a chatbot with ChatGPT allows users to achieve their individual goals more quickly through conversational processes, providing them with personalized information and enabling two-way interaction for prompt responses to inquiries; (3) the system facilitates telemedicine for patients who cannot visit the hospital, delivering real-time and precise diagnostic information and prescription recommendations aligned with symptom severity; and (4) the smart telemedicine diagnosis system also monitor fluctuations in pressure injury symptoms, triggering immediate alerts to medical personnel for adjustments in treatment techniques and medications as necessary.

In the future, our system could integrate with EHR systems, which enhances the efficiency of data processing and promote the sharing of patient information. This system can further support personalized and immediate medical decision making. Moreover, the wound images collected in this study are primarily based on past medical records from domestic hospitals. We hope to collaborate with foreign hospitals to incorporate pressure injury images from foreign patients to enrich training data set. In addition to further validation, this would enhance the generalization ability and reliability of the classification model.

Availability of Data and Materials

Not applicable.

Ethics Approval and Consent to Participate

Clinical data were selected for the study. The research ethics committee of The Institutional Review Board Chung Shan Medical University Hospital (CSMUH No. CS2-20077) approved this study.

Consent for Publication

Not applicable.

Footnotes

Disclosure Statement

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the study reported in this article.

Funding Information

The authors thank the National Science and Technology Council of Taiwan for partially and financially supporting this research under contract number NSTC 112-2221-E-007-091-MY2.

References

de Bengy

, Lamartine

, Sigaudo-Roussel

, et al. Newborn and elderly skin: Two fragile skins at higher risk of pressure injury. Biol Rev Camb Philos Soc, 2022; 97(3):874–895.

Mervis

, Phillips

. Pressure ulcers: Pathophysiology, epidemiology, risk factors, and presentation. J Am Acad Dermatol, 2019; 81(4):881–890; doi: 10.1016/j.jaad.2018.12.069

Mathew

, Elia

, Penfil

, et al. Application of telemedicine technology to facilitate diagnosis of pediatric postintensive care syndrome. Telemed J E Health, 2020; 26(8):1043–1050.

Toschi

, Adam

, Atakov-Castillo

, et al. Use of telemedicine in adults with type 1 diabetes: Do age and use of diabetes-related technology matter?. Telemed J E Health. 2023; 29(9):1374–1382.

Chang

KM.

Portable obstructive sleep apnea screening system using overnight ECG and a PDA-based wireless transmission system. Telemed J E Health, 2009; 15(4):353–361.

Ray

PP.

ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet Things Cyber Phys Syst, 2023; 3:121–154.

Brown

, Mann

, Ryder

, et al. Language models are few-shot learners. Adv Neural Informat Process Syst, 2020; 33:1877–1901.

Chiu

, Chuang

. Applying transfer learning to achieve precision marketing in an omni-channel system—A case study of a sharing kitchen platform. Int J Product Res, 2021; 59(24):7594–7609; doi: 10.1080/00207543.2020.1868595

Wang

, Bochkovskiy

, Liao

HYM

. YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada; 2023; pp. 7464–7475.

10.

Chiu

, Chen

. Applying data augmentation and mask R-CNN-based instance segmentation method for mixed-type wafer maps defect patterns classification. IEEE Trans Semiconduct Manuf, 2021; 34(4):455–463; doi: 10.1109/TSM.2021.3118922

11.

Höglund

, Khedri

Comparison between RLHF and RLAIF in Fine-Tuning a Large Language Model. Bachelor Thesis, Kth Royal Institute of Technology. Available from: https://www.diva-portal.org/smash/get/diva2:1782683/FULLTEXT01.pdf 2023.

12.

Orciuoli

, Orciuoli

, Peduto

. A mobile clinical DSS based on augmented reality and deep learning for the home cares of patients afflicted by bedsores. Proc Comput Sci, 2020; 175:181–188; doi: 10.1016/j.procs.2020.07.028

13.

Fergus

, Chalmers

, Henderson

, et al. Pressure ulcer categorisation using deep learning: A clinical trial to evaluate model performance. arXiv preprint, 2022; doi: 10.48550/arXiv.2203.06248

14.

Yilmaz

, Atagün

, Demırcan

FÖ

, et al. Classification of Pressure Ulcer Images with Logistic Regression. In: 2021 International Conference on INnovations in Intelligent SysTems and Applications (INISTA). IEEE, Kocaeli, Turkey; 2021; pp. 1–6; doi: 10.1109/INISTA52262.2021.9548585

15.

Kim

, Lee

, Choi

, et al. Augmented decision-making in wound care: Evaluating the clinical utility of a deep-learning model for pressure injury staging. Int J Med Inform, 2023; 180:105266; doi: 10.1016/j.ijmedinf.2023.105266