Investigation of usability problems of electronic medical record systems in the emergency department

Abstract

BACKGROUND:

Despite the benefits of using electronic medical record (EMR) systems, existing studies show that many healthcare providers are uncertain regarding their usability. The usability issues of these systems decrease their efficiency, discourage clinicians, and cause dissatisfaction among patients, which may result in safety risks and harm.

OBJECTIVE:

The aim of this study was to collect and analyze EMR system usability problems from actual users. Practical user interface guidelines were presented based on the medical practices of these users.

METHODS:

Employing an online questionnaire with a seven-point Likert scale, usability issues of EMR systems were collected from 200 emergency department healthcare providers (103 physicians (medical doctors) and 97 nurses) from South Korea.

RESULTS:

The most common usability problem among the physicians and nurses was generating in-patient selection. This pertained to the difficulty in finding the required information on-screen because of poor visibility and a lack of distinctiveness.

CONCLUSIONS:

The major problems of EMR systems and their causes were identified. It is recommended that intensive visual enhancement of EMR system interfaces should be implemented to support user tasks. By providing a better understanding of the current usability problems among medical practitioners, the results of this study can be useful for developing EMR systems with increased effectiveness and efficiency.

Keywords

EMR user interface design guidelines clinical workflow

1 Introduction

1.1 Advantages of EMR systems

Electronic medical record (EMR) systems present a systematized collection of patient health information in a digital format for supporting the clinical and administrative processes of a single healthcare organization. An electronic health record (EHR) includes data from all clinicians and healthcare organizations involved in the care of a patient. EMRs are local and internal, whereas EHRs are interoperable. EMRs and EHRs are interchangeably used in most health informatics [1]. As electronic databases of patient medical information, they include basic medical history (demographics such as age and weight, medical history, hospital visits, medication, and laboratory test results), diagnosis, and treatment. These records also capture the information related to allergies, immunization status, and physical examinations, such as radiology images and vital signs. These records present various advantages for improving the quality of care, clinical decision making, and medical documentation. In addition, the adoption of interoperable EMR systems can achieve cost reduction and relative risk reduction related to medication errors and adverse drug events [2 –4].

1.2 Practical issues of implementing EMR systems

EMRs have various disadvantageous, such as adoption costs, acceptance by clinicians, and technical problems [1]. Unintended and unexpected consequences continue to be reported with adverse effects, including long documentation time, communication failure, information redundancy, and considerable effort for maintaining continuous information management [5]. For example, the average adverse event rate across eight countries was reported as 8.2%, ranging from 2.5%to 18.4%. Among these events, 83%were considered to be preventable, and approximately 30%resulted in death of the patient [6]. These unexpected results were not only caused by software errors but also how clinicians use EMR systems, which suggests that they suffered from ineffective user interface design (regardless of the workflow) and complex data structure [7]. Thus, to solve usability problems, there is an increasing need for testing and quality evaluation in real or simulated clinical environments as well as actively improving the awareness of EMR system usability.

EMR systems have been adopted by hospitals at a low rate because of their low efficiency and usability in the United States. Typical reasons for this are related to system design, particularly in terms of system functionality and interoperability. These aspects are intercorrelated with the poor organization and display of information, interrupted workflow practices, and increased cognitive burden on physicians [8, 9]. Thus, because of the increased workload, stress, fatigue, and burnout from the poor system design, medical staff present low levels of use and dissatisfaction with EMR systems [10, 11]. Satisfaction of clinicians has insignificantly increased despite the active improvements in the layout design based on user needs [12]. Because of this negative perception, some countries have delayed improving EMR systems, presenting a major obstacle in their adoption [7].

The leading cause pertaining to the reduced usability of EMR systems is the lack of user-friendliness, such as extraneous information on screen and poor display layout. This causes task redundancy and interruptions in relevant information [13]. Additional common reasons include typing errors (above 60%) and wrong data entry, such as adding clinical information to the chart of another patient [14]. In diagnosis or medication, unintentional selection of an erroneous item occurs frequently, particularly when a drop-down list is located above or below a desired item. Incorrect use by physicians (clicking errors, cloned, or bloated notes) also lower the satisfaction of the system use and unintentionally affect usability, suggesting that EMR systems have not been reliably proven to improve the quality of care [15, 16].

Another major cause of EMR-related problems is patient data loss due to data transfer failures. For example, standardization of data transfer in South Korea has not been appropriately established. Owing to the large volume of patient data in EMRs, systematic and interactive system requirements are still being addressed [17, 18]. In particular, it should be considered that institutional problems vary significantly with the hospital and the national environment.

1.3 Definitions of EMR usability and satisfaction

The Healthcare Information Management Systems Society defines EMR usability as

[The] effectiveness, efficiency, and satisfaction with which specific users can achieve a specific set of tasks in a particular environment. In essence, a system with good usability is easy to use and effective. It is intuitive, forgiving of mistakes and allows one to perform necessary tasks quickly, efficiently and with minimum mental effort [19].

Nielsen explained the usefulness of these systems by introducing the concept of learning efficiency. It includes the ability to remember, error prevention and protection, and system satisfaction [20]. Usability elements applicable to EMR systems include “simplicity, naturalness, consistency, minimizing cognitive load, efficient interactions, forgiveness and feedback, effective use of language, effective information presentation, and preservation of context” [19]. A convenient brief definition of usability is “the capability to be used by humans easily and effectively”, where ease and effectiveness are equivalent to specified levels of subjective assessment and (human) performance, respectively [21]. Finally, EMR usability can be defined as the capability to be used by clinicians easily and effectively for the ultimate purpose of improving quality of care and patient safety. To support the healthcare process and reduce medical errors, an EMR must support clinical workflows and have interfaces that are easy to understand and navigate [22, 23].

1.4 EMR usability evaluation requirements

Usability continues to be an important factor affecting the satisfaction of physicians [24]. However, it is very difficult to complete the overall usability evaluation of EMR systems because of the significant time and resource consumption and complexities regarding the number of processes requiring support, types of users involved, and professional work context [25]. Moreover, Theis et al. conducted context and requirement analysis, based on which they suggested the use of information technology to obtain the importance of design recommendations [26]. Therefore, choosing an appropriate usability evaluation method is extremely important for evaluating the quality of interactive software systems. Although surveys allow identifying and logically managing various problems, they may omit certain issues, which can be identified using other methods [12]. Using a multi-method approach can be effective for evaluating usability because it will benefit from the increased sample size and in-depth understanding of the issues.

1.5 Emergency department as work environment

An emergency department (ED) is an important care environment in hospitals. It specializes in the acute care of patients who arrive at a hospital without a prior appointment. Initial treatment for a wide range of diseases and injuries must be provided by this department, some of which may be life threatening and require immediate attention [27, 28]. Thus, an ED is highly susceptible to incidents and adverse events, where latent errors can be difficult to recognize by the end-users, which can lead to multiple types of active errors. Moreover, ED clinicians are responsible for numerous tasks during limited working hours, resulting in high workload and related stress [29, 30]. Thus, in this environment, accurate management of the medical records of patients can be difficult. Moreover, EMR systems may vary in terms of functionality, with each function employing a specific usability principle for connecting with a range of tasks and the overall workflow [31]. For example, medication errors in clinical decision support (CDS) systems are typically concerned only with poor usability in the process, instead of the overall workflow. The actual clinical workflow in a department should be fully considered to prevent practical deviations [22 , 33]. Therefore, both environmental factors and functional diversity can significantly impact EMR system usability issues. The most complex, dynamic, and multifactorial environment in the context of this study is an ED. In this regard, EMR system usability should be evaluated according to the particular functions of the system, instead of considering it as a single system.

1.6 EMR usability problems for ED clinicians considering workflows

Following the introduction of communicating systems in EMR systems, the medical community and governments have continued focusing on designing various health information systems to ensure efficient and error-free use [34]. However, research on EMR systems in terms of usability, workflow, and cognitive support from an ergonomic perspective remains insufficient, despite its importance. For example, although EMR systems have been in use for more than 20 years in South Korea, the changes in the functional work procedures have led to user resistance to their adoption. Strategies are needed to increase user acceptance and maintain usability in the current South Korean medical industry. These strategies should be established based on accurate usability evaluation and should match user requirements with the functional and system design elements.

ED hospital environments are generally fast, and the times of the clinicians are very important. Studies have shown that physicians are constantly challenged by time constraints due to the requirement of rapid evaluation and review of patient data to make correct clinical decisions [35]. Owing to these limitations, simulated environments and survey data are used as appropriate methods for identifying system design problems (with insufficient participants) [36, 37]. Nevertheless, usability problems remain a major concern for physicians [38]; thus, it is necessary to find a real medical scenario, as considered by clinicians, in the context of EMR system usage by maximum number of actual users (both physicians and nurses).

Therefore, this study aimed at obtaining the actual EMR user experiences of clinicians to the maximum extent to identify the most common and serious problems in the current EDs in South Korea. Moreover, clinical workflows for both physicians and nurses were considered because of their different requirements and functionalities. Finally, other external factors such as tasks and environments were considered. This qualitative study aimed to assist in identifying and understanding the usability problems in the current ED hospitals in South Korea.

2 Method

2.1 Interviews and observations

From the interviews and observations of clinicians working in the largest medical institution in South Korea, the most commonly used EMR functions were identified. The flow of EMR use in EDs was considered in developing questionnaires and selecting related EMR functions for both user groups (see Fig. 1). Although the EMRs of nurses are continuous with the time flow, those of physicians are entered once per day, except when primary care is required and there is important change in patient progression.

Fig. 1

Representative clinical workflow using EMR in ED.

2.2 Questionnaires

Because EMR systems of different manufacturers have various functions, “universal” EMR functions should be selected for identifying the representative usability problems. Therefore, the questionnaires were developed based on all results of the interviews, the observations, and a pilot survey. The pilot survey was conducted about the frequency and importance of EMR functions based on the experience of eight clinicians. Each clinician was assigned seven task items, which consisted of 2–14 functions and 3 responding options (lowest: 1 point, middle: 2 points, and highest: 3 points) for each function of each task. Consequently, functions with over 2.5 points were adopted for the final questionnaires. The results of the pilot survey for selecting the universal EMR functions are presented in Appendix A.

Table 1 summarizes the final structure of the EMR usability questionnaires including the clinical workflow. There were a total of 44 questions, which are as follows: general functions within the ED (ED Physician: General —8 items), specific functions of orders for ED physicians (ED Physician: Order —9 items), and general EMR system functions for ED nurses (ED Nurse: General —15 items). We use a seven-point Likert scale with a free-text response for each questionnaire to collect the usability problems based on the actual frequency of EMR experiences. All EMR usability questionnaires are shown in Appendix B.

Table 1
Structure of EMR usability questionnaire including clinical workflow

Classification Category Item Code

Physicians (ED) General Patient selection Patient selection ESelect

Initial assessment Care plan EPlan

Patient initial diagnosis Edx

Department notes and progress record Entering test results ELab

Communication ECom

Department notes and progress record EProg

Discharge documentation Final diagnosis and discharge summary EDisch

Future plans EFPlan

(ED) Orders Patient selection Patient selection EOSelec

Diagnosis order Pulse, weight, and EKG EOBasic

Lab order Laboratory orders EOLab

Medication order Searching for drug name or NDC code EOSearch

Dose/unit/frequency EOMed

Type and amount EOMedAMT

Mixed order EOMix

Transfusion order Blood transfusion EOTrans

Order results Inquiring order results EOResult

Nurses (ED) General Patient selection Patient selection ENSelect

Narrative nursing notes Patient triage ENTriage

Patient-related inputs (e.g., hospital route or transfer) ENPtInfo

Chief complaint, history of present illness, allergies, past medical/surgical history (PHx), medication history, and additional comments ENPtProb

Nurse (daily) notes Nurse notes ENNote

Vital sign checks ENVS

Pain assessment ENPain

Intake/output (I/O) chart ENIO

Interoperative records Medication/injection—completion of records ENMedR

Confirming and inquiring orders of physicians ENOSer

Recording after completing physician orders ENOR

Viewing imaging and lab result ENOLab

Communications Shift (hands-off) ENHF

Shift (hands-on) ENHN

Ward transfer ENCom

Classification		Category	Item	Code
Physicians	(ED) General	Patient selection	Patient selection	ESelect
		Initial assessment	Care plan	EPlan
			Patient initial diagnosis	Edx
		Department notes and progress record	Entering test results	ELab
			Communication	ECom
			Department notes and progress record	EProg
		Discharge documentation	Final diagnosis and discharge summary	EDisch
			Future plans	EFPlan
	(ED) Orders	Patient selection	Patient selection	EOSelec
		Diagnosis order	Pulse, weight, and EKG	EOBasic
		Lab order	Laboratory orders	EOLab
		Medication order	Searching for drug name or NDC code	EOSearch
			Dose/unit/frequency	EOMed
			Type and amount	EOMedAMT
			Mixed order	EOMix
		Transfusion order	Blood transfusion	EOTrans
		Order results	Inquiring order results	EOResult
Nurses	(ED) General	Patient selection	Patient selection	ENSelect
		Narrative nursing notes	Patient triage	ENTriage
			Patient-related inputs (e.g., hospital route or transfer)	ENPtInfo
			Chief complaint, history of present illness, allergies, past medical/surgical history (PHx), medication history, and additional comments	ENPtProb
		Nurse (daily) notes	Nurse notes	ENNote
			Vital sign checks	ENVS
			Pain assessment	ENPain
			Intake/output (I/O) chart	ENIO
		Interoperative records	Medication/injection—completion of records	ENMedR
			Confirming and inquiring orders of physicians	ENOSer
			Recording after completing physician orders	ENOR
			Viewing imaging and lab result	ENOLab
		Communications	Shift (hands-off)	ENHF
			Shift (hands-on)	ENHN
			Ward transfer	ENCom

2.3 Data collection

Data on the experience of using an EMR system in an ED were collected by conducting online surveys in South Korea to consider various occupational and personal characteristics. Questionnaires were developed to investigate the errors related to functions of an EMR system (e.g., patient selection, initial recording, and progress report). A total of 44 questions were included on all EMR system functions for ED clinicians, and an additional 12 questions were on the socio-demographic characteristics of the clinicians (e.g., age, gender, and career). The perceived usability with each function of the EMR system was requested using a seven-point Likert scale (from 1: None at all to 7: most frequent). The questionnaires were distributed at the Korean Society of Emergency Medical Informatics. A total of 200 subjects participated in the survey from April to May 2017. All subjects responded based on their actual experiences with EMR systems. Specifically, their usability was evaluated on the frequency of experienced EMR use problems due to the corresponding functions. Therefore, a high score indicated that they frequently encountered usability problems for each function of the EMR system.

2.4 Participant information

The sample comprised 103 physicians (medical doctors) (51.5%) and 97 nurses (48.5%) who were working in an ED. Approximately 80%physicians were from an ED. The gender distribution of the participants was 86 males and 114 females (physicians: 78 males and 25 females, nurses: 8 males and 89 females). The average age of the clinicians was 29.25 years (standard deviation (SD) = 6.87), and the average EMR system use was 6.82 years (SD = 3.96). Regarding the level of EMR education, 83.5%respondents (n = 167) had completed courses in EMR system training and 33%(n = 16.5) had not. The data showed that the respondents worked in hospitals of different sizes in various roles. For example, majority of the respondents worked in hospitals with 500–1000 beds, with shifts divided into three daily phases. Demographic information of the participants indicated the average number of EMR system experience was 2.04 per clinician and the average daily work time was 9.8 h. Finally, the number of workdays per month was 19.09 days; the number of patients reviewed per day was 35.51. More information of the participants is provided in Table 2.

Table 2
Demographics of questionnaire respondents

Demographic N(%)

Job MD 103(51.5)

Nurse 97(48.5)

Sex Male 86(43.0)

Female 114(57.0)

Age (years) 20–29 55(27.5)

30–39 105(52.5)

≥40 40(20.0)

Hospital (number of beds) 500 33(16.5)

500–1000 77(38.5)

1000–1500 17(8.5)

≥1500 73(36.5)

Department (n = 103) † Emergency medicine 82(79.6)

Others 21(20.3)

Title (n = 103) † General practitioner 95(92.2)

Specialist 8(7.8)

Work type Full-time 42(21.0)

Two-shift 55(27.5)

Three-shift 86(43.0)

Others 17(8.5)

Career (years) < 3 years 54(27)

3–5 years 81(46.5)

5–10 years 34(17)

≥10 years 31(15.5)

EMR career (years) < 3 years 54(27)

3–5 years 93(46.5)

5–10 years 27(13.5)

≥10 years 26(13)

EMR education Yes 167(83.5)

No 33(16.5)

Demographic		N(%)
Job	MD	103(51.5)
	Nurse	97(48.5)
Sex	Male	86(43.0)
	Female	114(57.0)
Age (years)	20–29	55(27.5)
	30–39	105(52.5)
	≥40	40(20.0)
Hospital (number of beds)	500	33(16.5)
	500–1000	77(38.5)
	1000–1500	17(8.5)
	≥1500	73(36.5)
Department (n = 103) †	Emergency medicine	82(79.6)
	Others	21(20.3)
Title (n = 103) †	General practitioner	95(92.2)
	Specialist	8(7.8)
Work type	Full-time	42(21.0)
	Two-shift	55(27.5)
	Three-shift	86(43.0)
	Others	17(8.5)
Career (years)	< 3 years	54(27)
	3–5 years	81(46.5)
	5–10 years	34(17)
	≥10 years	31(15.5)
EMR career (years)	< 3 years	54(27)
	3–5 years	93(46.5)
	5–10 years	27(13.5)
	≥10 years	26(13)
EMR education	Yes	167(83.5)
	No	33(16.5)

3 Results

3.1 Descriptive statistics of physicians

For the physicians, the overall average score of the 17 survey items was 2.20, and the average score for the patient selection was the highest (ESelect: Mean = 2.68, SD = 1.35), which suggests that more usability problems were collected than other issues. The lowest score was for both confirming the results of the orders (EOResult: Mean = 1.83, SD = 0.97) and managing future plans (EFPlan: Mean = 1.83, SD = 0.96) (see Table 3). All items were ranked in the following order: patient selection (ESelect); searching for a drug name or code (EOSearch); diagnosis and recording of the initial record of the patient (Edx); ordering the unit and number of doses during medication (EOMed); selecting the patient via the order screen (EOSelect); decision on formulation and dosage (EOMedAMT); orders for medical images, collecting blood samples, etc. (EOLab); communicating with other departments via the EMR (ECom); inquiry of a test result to make progress record sheet (ELab); diagnosis and recording of the final record of the patient (emission record) in the EMR (EDisch); care plan (EPlan); mixed orders of drugs (EOMix); basic orders, such as blood pressure, weight, and electrocardiogram (EOBasic); department notes and progress record (EProg); orders of blood transfusion (EOTrans); inquiry of the order results (EOResult); and managing future plans (EFPlan) (see Fig. 2).

Table 3
Descriptive statistics of physicians

Descriptive statistics —physicians (n = 103)

EMR functions (workflow) Mean Std. error Std. deviation

ESelect 2.68 0.132 1.345

EPlan 2.15 0.113 1.15

Edx 2.5 0.14 1.42

ELab 2.21 0.128 1.296

ECom 2.26 0.128 1.298

EProg 1.95 0.095 0.964

EDisch 2.2 0.129 1.309

EFPlan 1.83 0.095 0.961

EOSelect 2.35 0.12 1.218

EOBasic 1.93 0.117 1.19

EOLab 2.27 0.122 1.238

EOSearch 2.66 0.144 1.466

EOMed 2.47 0.13 1.32

EOMedAMT 2.27 0.12 1.222

EOMix 2.04 0.109 1.111

EOTrans 1.85 0.099 1.004

EOResult 1.83 0.096 0.974

Valid N (listwise)

Descriptive statistics —physicians (n = 103)
EMR functions (workflow)	Mean	Std. error	Std. deviation
ESelect	2.68	0.132	1.345
EPlan	2.15	0.113	1.15
Edx	2.5	0.14	1.42
ELab	2.21	0.128	1.296
ECom	2.26	0.128	1.298
EProg	1.95	0.095	0.964
EDisch	2.2	0.129	1.309
EFPlan	1.83	0.095	0.961
EOSelect	2.35	0.12	1.218
EOBasic	1.93	0.117	1.19
EOLab	2.27	0.122	1.238
EOSearch	2.66	0.144	1.466
EOMed	2.47	0.13	1.32
EOMedAMT	2.27	0.12	1.222
EOMix	2.04	0.109	1.111
EOTrans	1.85	0.099	1.004
EOResult	1.83	0.096	0.974
Valid N (listwise)

Note. patient selection (ESelect); searching for drug name or code (EOSearch); diagnosis and recording of initial record of patient (Edx); ordering unit and number of doses during medication (EOMed); selecting patient via order screen (EOSelect); decision on formulation and dosage (EOMedAMT); orders for medical images, collecting blood samples, etc. (EOLab); communicating with other departments via EMR (ECom); inquiry of test result to make progress record sheet (ELab); diagnosis and recording of final record of patient (emission record) in EMR (EDisch); care plan (EPlan); mixed orders of drugs (EOMix); basic orders, such as blood pressure, weight, and electrocardiogram (EOBasic); department notes and progress record (EProg); orders for blood transfusion (EOTrans); inquiry of order results (EOResult); and managing future plans (EFPlan).

Fig. 2

Average scores of usability problems in EMR system functions for physicians (high score indicates less usability). Note. patient selection (ESelect); searching for drug name or code (EOSearch); diagnosis and recording of initial record of patient (Edx); ordering unit and number of doses during medication (EOMed); selecting patient via order screen (EOSelect); decision on formulation and dosage (EOMedAMT); orders for medical images, collecting blood samples, etc. (EOLab); communicating with other departments by EMR (ECom); inquiry of test result to make progress record sheet (ELab); diagnosis and recording of final record of patient (emission record) in EMR (EDisch); care plan (EPlan); mixed orders of drugs (EOMix); basic orders, such as blood pressure, weight, and electrocardiogram (EOBasic); department notes and progress record (EProg); orders on blood transfusion (EOTrans); inquiry of order results (EOResult); and managing future plans (EFPlan).

3.2 Descriptive statistics of nurses

For the nurses, the overall average score of the 15 survey items was 2.66, and the average score for patient selection was the highest (ENSelect: Mean =3.79, SD = 1.73). The lowest score was noted for entering records about completion of the instructions of the physician (orders) (ENOR: Mean = 2.24, SD =1.30) (see Table 4). All items were ranked in the following order: patient selection (ENSelect); patient classification (ENTriage); entering patient problems, such as symptoms and medical history (ENPtProb); completing nursing notes (ENNote); communicating with other nurses (ENCom); entering patient-related information, such as visits and the hospitals involved in the EMR (ENPtInfo); searching for and confirming the orders of the physician using the ED performance record (ENOSer); checking vital signs and entering in the records (ENVS); entering level of pain (ENPain); recording performance of orders, such as medication and injection, in the ED performance sheet (ENMedR); checking test results of the lab orders (ENOLab); hands-off via takeover note (ENHF); hands-on via takeover note (ENHN); entering intake/output (I/O) (ENIO); and records about completion of the orders of the physician (ENOR) (see Fig. 3).

Table 4
Descriptive statistics of nurses

Descriptive statistics —nurses (n = 97)

EMR functions (workflow) Mean Std. error Std. deviation

ENSelect 3.79 0.176 1.732

ENTriage 3.14 0.144 1.421

ENPtInfo 2.76 0.139 1.368

ENPtProb 2.88 0.147 1.445

ENNote 2.87 0.14 1.374

ENVS 2.58 0.153 1.506

ENPain 2.54 0.149 1.465

ENIO 2.27 0.144 1.418

ENMedR 2.44 0.147 1.443

ENOSer 2.64 0.153 1.508

ENOR 2.24 0.132 1.297

ENOLab 2.37 0.135 1.333

ENHF 2.33 0.144 1.42

ENHN 2.3 0.15 1.473

ENCom 2.79 0.166 1.633

Valid N (listwise)

Descriptive statistics —nurses (n = 97)
EMR functions (workflow)	Mean	Std. error	Std. deviation
ENSelect	3.79	0.176	1.732
ENTriage	3.14	0.144	1.421
ENPtInfo	2.76	0.139	1.368
ENPtProb	2.88	0.147	1.445
ENNote	2.87	0.14	1.374
ENVS	2.58	0.153	1.506
ENPain	2.54	0.149	1.465
ENIO	2.27	0.144	1.418
ENMedR	2.44	0.147	1.443
ENOSer	2.64	0.153	1.508
ENOR	2.24	0.132	1.297
ENOLab	2.37	0.135	1.333
ENHF	2.33	0.144	1.42
ENHN	2.3	0.15	1.473
ENCom	2.79	0.166	1.633
Valid N (listwise)

Note. patient selection (ENSelect); patient classification (ENTriage); entering patient-related information, such as visits and hospitals involved, in EMR (ENPtInfo); entering patient problems, such as symptoms and medical history (ENPtProb); completing nursing notes (ENNote); checking vital signs and entering in records (ENVS); entering level of pain (ENPain); entering intake/output (I/O) (ENIO); performance of orders such as medication and injection in ED performance sheet (ENMedR); searching for and confirming orders of physician using ED performance record (ENOSer); recording after completing physician orders (ENOR); checking test results of lab orders (ENOLab); hands-off via takeover note (ENHF); hands-on via takeover note (ENHN); communicating with other nurses (ENCom).

Fig. 3

Average scores of usability problems in EMR system functions for nurses (high score indicates less usability). Note. patient selection (ENSelect); patient classification (ENTriage); entering patient-related information, such as visits and hospitals involved, in EMR (ENPtInfo); entering patient problems, such as symptoms and medical history (ENPtProb); completing nursing notes (ENNote); checking vital signs and entering records (ENVS); entering level of pain (ENPain); entering intake/output (I/O) (ENIO); performance of orders, such as medication and injection, in ED performance sheet (ENMedR); searching for and confirming orders of physician using ED performance record (ENOSer); recording after completing physician orders (ENOR); checking test results of lab orders (ENOLab); hands-off via takeover note (ENHF); hands-on via takeover note (ENHN); communicating with other nurses (ENCom).

3.3 Analysis of quantitative data: frequency of EMR usability problems

Independent samples t-tests were conducted in detail to determine the differences in the frequency of the EMR usability problems with 17 EMR functions and 15 EMR functions for the ED physicians and nurses, respectively. Compared to the 103 physicians, the 97 nurses demonstrated significantly more experience on a four-scale (occasionally), t(198) = –2.24, p = .03; five-scale (frequently), t(198) = –2.39, p = .02; and six-scale (very frequently), t(198) = –3.03, p = .00. Similarly, compared to the 97 nurses, the 103 physicians demonstrated significantly less experience on a one-scale (never), t(198) = 2.29, p = .02 and a two-scale (very rarely), t(198) = 3.13, p = .00. In comparison, there was no significant effect on a three-scale (rarely) and a seven-scale (always). All analysis results are provided in Table 5.

Table 5
Difference in frequencies of EMR usability problems of physicians and nurses in terms of EMR functions (Independent samples T-test results)

Frequency of EMR usability problems Mean Std. deviation t-value p-value

Physicians (n = 103) Nurses (n = 97) Physicians Nurses

1 (never) 5.50 3.87 5.69 4.40 2.29 0.02

2 (very rarely) 6.50 4.56 4.89 3.88 3.13 0.00

3 (rarely) 2.58 2.45 2.75 2.29 0.36 0.72

4 (occasionally) 1.29 1.98 2.06 2.30 –2.24 0.03

5 (frequently) 0.79 1.35 1.45 1.87 –2.39 0.02

6 (very frequently) 0.20 0.63 0.57 1.27 –3.03 0.00

7 (always) 0.13 0.16 0.55 0.47 –0.53 0.60

Frequency of EMR usability problems	Mean	Std. deviation	t-value	p-value
1 (never)	5.50	3.87	5.69	4.40	2.29	0.02
2 (very rarely)	6.50	4.56	4.89	3.88	3.13	0.00
3 (rarely)	2.58	2.45	2.75	2.29	0.36	0.72
4 (occasionally)	1.29	1.98	2.06	2.30	–2.24	0.03
5 (frequently)	0.79	1.35	1.45	1.87	–2.39	0.02
6 (very frequently)	0.20	0.63	0.57	1.27	–3.03	0.00
7 (always)	0.13	0.16	0.55	0.47	–0.53	0.60

Note. 17 EMR functions for 103 physicians; 15 EMR functions for 97 nurses.

3.4 Analysis of qualitative data: subjective response to EMR usability problems

A total of 1,525 responses were collected and analyzed by counting and classifying the responses into three categories: design/usability, individual elements, and psychosocial working conditions. In terms of design/usability, eight major system usability problems were identified: not capturing information at a glance, confusing and nonintuitive methods of delivering information, irrelevant opening of additional windows, complex search structures when effecting additional queries, ambiguous classification schemes and a lack of options, strict template with mandatory records, incomplete or erroneous records, and overall lack of functionality. All problems indicated that a poorly designed EMR system can influence the performance of clinicians. From the 1,525 responses, 821 (53.84%) belonged to the first category, ranking first among the three categories. In addition, 24 detailed problems were summarized in the first level (see Table 6). Regarding human-related errors noted by clinicians (individual factors), three major problems were identified: various clicking, scrolling, and typing errors; in particular, these errors were be related to a temporary decrease in the cognitive or physical capabilities of the clinicians. From the 1,525 responses, 218 (13.97%) belonged to the second category, ranking third among the three categories. In terms of psychosocial working conditions, five detailed problems were identified: restricted ED working conditions, disconnected organizations, poor communication, teamwork and decision-making structure, system management, and other supports. In particular, the interactions among these problems showed that psychosocial working conditions can adversely affect the quality of patient care. From the 1,525 responses, 486 (31.77%) belonged to the third category, ranking second among the three categories. Moreover, 14 detailed problems were summarized from the third to fifth levels (see Table 6).

Table 6
Structure representing EMR usability problems of clinicians based on subjective evaluation

Category Level Comments Physician Frequency (%) Nurse Frequency (%) Total Frequency (%)

Design/usability; poorly designed EMRs influence performance of clinicians 1st Not capturing information at glance 101 (11.88) 53 (7.56) 821 (53.84)

•screen extremely busy with additional windows

•small size of fonts and narrow column/row

•similar names of patients, drugs, and orders

Confusing and nonintuitive methods of delivering information 26 (3.06) 37 (5.28)

•mixed information in list

•difficulty in separating left/right or positive/negative

Irrelevant opening of additional windows 16 (1.88) 22 (3.14)

•frequent mouse controls due to screen truncation and multiple windows

•wrong records in window of another patient or user

Complex searching structures with additional queries 71 (8.35) 66 (9.42)

•clicks on numerous tabs and checkboxes

•difficult to convert Korean to English characters

•repetitive retyping from beginning

•inconvenient application for views

Ambiguous classification scheme and lack of options 89 (10.47) 65 (9.27)

•lack of options (no search result)

•excessive standardization in terms of options

•ambiguous matching problems (replacement with other similar options)

•no helpful tool for providing subjective impression about symptoms of patient

Stringent template with mandatory records 22 (2.59) 27 (3.85)

•limited number of characteristics (leading to unclear description)

•stringent against incomplete records with every popup

•increased time for re-charting

•numerous mandatory records based on every timeline

Incomplete or erroneous records 113 (13.29) 77 (10.98)

•no alarm when wrong pasting or shortcut

•incorrect or omitted inputs in required (or defaulted) items

•including common input errors, such as usage and volume

Overall lack of functionality 12 (1.41) 24 (3.42)

•no filtering, select-all, and temporary saving features

•no navigation and customization

Individual factors; degree of freedom from time pressure influence on working capabilities of clinicians 2nd Human-related errors with high task burdens and low perception 167 (19.65) 51 (7.28) 218 (13.97)

•various clicking errors (failing on-target and double vs. single)

•scrolling errors

•typing errors

Psychosocial work conditions; psychosocial work conditions of clinicians adversely influence patient care quality 3rd Limited work conditions of ED 33 (3.88) 38 (5.42) 71 (4.55)

•crowded physical environment

•severe time pressure of outsources

•uncooperative patients

•independent system of EMR (poor interoperability)

4th Disconnected organizations 12 (1.41) 18 (2.57) 202 (13.25)

•nonsearchable hospital names in system

Communication, teamwork, and decision-making structure 82 (9.65) 90 (12.84)

•different abbreviations and interfaces by ward

•overlapping records (concurrency)

•time conflict between ordering and charting

5th System management 60 (7.06) 82 (11.70) 213 (13.97)

•unintended saving errors

•low speed of system with buffering

•abrupt shutdown

Other supports (maintenance) 20 (2.35) 51 (7.28)

•inappropriate synchronization

•strong desire for real-time takeover

•log in and log out of another user

^*number of participants: physicians (103), nurses (97). ^**number of responses: physicians (824), nurses (701)

3.5 Comparison of subjective responses of physicians and nurses

When comparing the two user groups, it was observed that the nurses provided 123 fewer subject responses than the physicians. However, the opinions of the nurses offered greater diversity and clearer directions for improvement than those of the physicians. Only four subcategories were found in which the percent of physicians was higher than that of nurses, i.e., difficulty capturing information at a glance (physicians: 11.88% > nurses: 7.56), inconvenience due to ambiguous classifications and a lack of options (physicians: 10.47% > nurses: 9.27), frequent incomplete or incorrect entries (physicians: 13.29% > nurses: 10.98), and simple human error due to high workload and low perception (physicians: 19.65% > nurses: 7.28). Simple human errors accounted for the highest percentage for the physicians, particularly those related to clicking, scrolling, and typing. These errors were frequent among the physicians. For the remaining ten items, the nurses indicated a higher frequency of subjective responses than the physicians. Moreover, their criticisms and directions for improvement were primarily identified as being related to communication, teamwork, and the decision-making structure at the fourth level. This was primarily due to the different medical abbreviations, different system interfaces, time conflicts between ordering and charting, and consequently duplicated records.

4 Discussion

4.1 Major findings of usability evaluation of EMR systems

The most common EMR system error in both groups was caused by incorrect selection. For example, the clinicians reported difficulty in distinguishing between same or similar names and similar ages and when navigating between patients because of the small font size and narrow columns/rows. This suggests that the current EMR systems ineffectively deliver information on screen, which interrupts the workflow of the clinicians. Based on cognitive load theory, the cognitive load of clinicians on mental resources increases with the amount of information provided, both on screen and off screen. Because healthcare providers in an ED examine numerous patients through processes that include complex tasks, their working memory is probable to be affected by prioritizing and multitasking. Concerned and unsatisfied clinicians can potentially make numerous errors (defined as psychosocial problems), which occur because the available memory for solving problems is reduced by the intensive workload in the ED [39]. Furthermore, the clinicians claimed the inadvertent selection of patients primarily occurred because of careless clicks resulting from a poor user interface. Elements that are extremely closely paired on screen can decrease the mental processing rate of information when new cognitive resources, such as telephone, smartphone calls, and other noises, are present [40, 41]. Consequently, extended working hours from the inadvertent selection can lead to interruptions in the workflow and even accidental errors.

EMR systems must have an interface that is easy to understand and use to support multitasking. In this study, the survey assigned the highest score to the worst function in patient selection (ESelect and ENSelect) and the lowest to the best function in confirming results and completion on orders (EOResult for the physicians and ENOLab for the nurses). This suggests that both physicians and nurses considered the retrieval of test results as the most useful function of the EMR system. Specifically, the following user-unfriendly interfaces for medications were reported in this study. First, careful selection (e.g., of a patient or drug list) on a busy screen was noted as the most difficult aspect because focused attention was frequently required. Second, both groups criticized the unsophisticated filtering by the systems. The reasons for this included disorganized lists and ambiguous options (unclear names and codes of drugs), which increases the interruptions and workload. Third, a strong demand was indicated for rapid navigation and real-time notifications. Accordingly, an appropriate support must be added to each individual system function to dramatically reduce task burdens. This reveals that the overall user satisfaction for EMR systems is highly correlated with the clinical workflow [42, 43]. In addition, realizing these functions can enable interactive communication when submitting new orders, appropriate responses to new data related to laboratory results, and caring plan reminders [44]. Finally, they criticized the frequent control of windows (e.g., overlapping windows and screen flips). This can be prevented by a user interface minimizing the number of windows (clicks) or installing at least dual monitors at the PC station.

CDS is the most important aspect of EMR systems. It primarily involves decision-making support for electronic orders and prescribing medication [45]. One of the most commonly used data entry tools is a collection of orders, performed at once, to manage a series of orders, disease conditions, or specific procedures. In the survey conducted for the group of physicians, five items pertaining to medication prescribing were ranked as relatively significant usability problems. These were searching for a drug name or code medication (EOSearch); ordering the unit and number of doses during medication (EOMed); selecting the patient via the order scree (EOSelect); ordering for medical images, blood samples, etc. (EOLab); and decision on formulation and dosage (EOMedAMT). Accordingly, improved innovative CDS interventions should be performed for understanding and evaluating information in the areas of data search and data entry. These interventions should be triggered by user tasks to help clinicians select the correct medications appropriately [7 , 46]. Interventions include suggestions for drug alternatives based on patient characteristics and potential allergy–food interactions. In addition, functions for duplicating previous care or formulary adherence are suggested for saving the time and effort of clinicians. Similar to order sets, more advanced ordering tools can be developed to collect orders for procedures, radiology, laboratories, and medications using built-in calculators and guided dosing. In addition, parameter guidance and immediate alerts for data out of range can be added. All these interventions can help to reduce medication errors, enhance workflow, and develop a highly safe healthcare environment.

This study found that both physicians and nurses had continuous interactions with EMR systems based on the following five categories: data selection (e.g., patient selection, ESelect); data input (e.g., medication; EOMed); data retrieval (e.g., searching for and confirming the orders of the physician using the ED performance record; ENOSer and completing nursing notes; ENNote); data review (e.g., entering records about completion of the orders of the physician; ENOR); data inquiry (e.g., managing future plans; EFPlan and inquiry of order results; EOResult). In each interaction, the clinicians expected appropriate and frequently patient-specific information to be displayed throughout the interaction, indicating a certain point within the course of care. The representative decision support tools for continuous interactions in the EMR systems included “electronic guidelines, order sets, calculators, reports and dashboards, documentation templates, and diagnostic or clinical workflow support tools” [46]. However, some decision support tools, such as alerts and reminders (push and pull), within each interaction may require additional attention because of their increased complexity, which can cause inappropriate overrides [47]. Therefore, stepwise directions with increased salient feedback should be provided to clinicians throughout the working process as additional opportunities for validating decisions and/or to detect errors. Furthermore, predefined templates for documentation and custom features (e.g., font size) are suggested to allow for more rapid interactions, thereby providing more flexibility in EMR systems.

4.2 In-depth analysis of subjective responses

Comparing the two user groups, the opinions of the nurses offered greater diversity and clearer directions for improvements. The reason for this is that physicians are more probable to be replaced by nurses in an ED. Existing studies have indicated similar health outcomes when comparing the care provided by physicians and nurses. An appropriately trained nurse practitioner can provide the same high-quality care and achieve the same positive health results for patients as physicians [48]. More specifically, patients tend to be satisfied with their care at first contact with both types of healthcare practitioners, which suggests that short-term health outcomes are similar between the two groups. Therefore, the role of a nurse is becoming highly prominent in EDs where a short-term contact with patients is frequent. In addition, nurses use a range of EMR functions when providing various cares. Thus, they appeared to exhibit a high level of response to the subjective questionnaire aspects about these systems and presented useful data for application in various improvement plans. Accordingly, the involvement of nurses as well as physicians is urgently needed in the development of EMR systems.

The results of the qualitative evaluation regarding the usability of EMR systems in an ED were classified into three categories: user interface of the system, individual factors, and psychosocial working conditions. These categories also represent 64%of all 211 safety issues associated with HIT events in Deep Dive analysis of ECRI Institute PSO: system interface issues, wrong input, software issue –system configuration, wrong record retrieved, software issue –functionality [49]. This suggests that the user interface of the system and individuals were responsible for the medical errors or accidents although the error determinants in complex healthcare systems are multifactorial. Part et al. (2015) emphasized the need for research on how clinicians perform their daily work processes and which parts of the system prove to be the most challenging for their uses [50]. Therefore, the present study focused on macro-ergonomic aspects, such as supporting efficient workflow and minimizing time constraint by scheduling appointments with sufficient time allocated to undertake the required treatment [51], which are as important as designing an improved system interface. In addition to micro-ergonomic factors, which directly affect usability and safety, environmental and organizational factors have a significant impact on the quality of care. In the EDs, most clinicians found it difficult to manage patient throughput owing to patient admission and repeated hands-off documents, suggesting that using an increased systematic approach to job design and workload planning is important for the physical and mental health of clinicians [52, 53].

In summary, EMR fundamental problems can be summarized as follows. First, usability problems are caused by low system reliability, failure in data transfer, and poor system performance (management, integration, and other technology bases) [17, 54]. Second, usability problems arise from human errors by systems that do not present well interfaces or clinical workflows [31, 55]. Third, the behaviors of clinicians in most care processes have been changed after EMR adoption, which constantly introduces new EMR-related problems. Lanier et al. [56] explained that after the introduction of an EMR system, all records (in relation to treatment) are available on the screen of the EMR without examining the patient, thus reducing physician–patient visual interaction and overall consultation time (increasing quiet time). The sociotechnical approach to system-related problems is improving system interoperability and performance, whereas efforts to change user interface designs, clinical workflows, and user behaviors of the medical staff remain unresolved. Despite their importance, few studies have been conducted in the real world.

4.3 Practical methods of improving EMR usability

The most common usability issues in current EMR systems are summarized in Table 7. Based on these results, the following EMR function-based guidelines are proposed.

Table 7
Most common usability problems in current EMR systems

Two types of most common EMR usability problems

Human-related Clinicians could not identify patient correctly and entered all clinical information in wrong record.

Clinicians were confused by opening of multiple records and entered incorrect data in record.

Data were not entered into system although clinicians thought that they completed it.

Clinicians entered incomplete and missing data entries.

Computer-related Data were not displaying suitably in system.

Data were lost.

Software was not working properly.

Network was slow or down.

Two types of most common EMR usability problems
Human-related	Clinicians could not identify patient correctly and entered all clinical information in wrong record.
	Clinicians were confused by opening of multiple records and entered incorrect data in record.
	Data were not entered into system although clinicians thought that they completed it.
	Clinicians entered incomplete and missing data entries.
Computer-related	Data were not displaying suitably in system.
	Data were lost.
	Software was not working properly.
	Network was slow or down.

For identifying and selecting data, improving user visibility and information distinctiveness is paramount. Because the physical structure of the layout guides a user to the required information [57], visual elements must be organized into logical groups defined by space (proximity) and alignment, highlighting important elements by appropriate distinct levels. For example, patient names on the screen should be separated by a sufficient number of spaces to build relationships with other information (e.g., sex, age, and visiting time). Furthermore, the ability to highlight the selected patient with a larger font, liberal use of color, and enhanced contrast should be provided. This will enable clinicians to easily identify patient names and additional information. Inadvertent clicking on an incorrect option can be prevented by adjusting the proximity of user interface items [22, 58]. For example, visual feedback on a selected item (highlighted selection) and relevant information (patient photo) can be presented.

In data review and entry, minimizing unnecessary window popups and improving screen recognition are recommended. Additionally, single scrolling should be implemented when observing a complex arrangement of visual elements (e.g., assessment) and for increased understanding when reviewing patient summaries, notes, and medications. To learn about the condition of a patient, clicking a button directly linked to an information source presenting detailed, evidence-based knowledge regarding the disease and its treatment will be helpful. Furthermore, functions EOSearch, Edx, and EOMed for physicians and functions ENTriage, ENNote, and ENPtProb for nurses should be simplified because early diagnosis of patients is extremely time-consuming, particularly in an ED. In addition, more innovative (smart) entry formats and document templates should be provided with liberal letter-writing features, such as predefined letter templates, given-condition automation, and rapid duplication. For data entry, timely interventions such as a “temporary saving mode” and “synchronizing mode” can promote uninterrupted records without missing information and ease the concern of clinicians about records.

When searching and filtering, easy retrieval of names and drug codes in the ordering process can improve the efficiency of subsequent tasks. One effective filtering method involves adding decision logic along with general drug–drug interaction rules and additional data from the individual EMR, rendering more patient-specific rules. Owing to the difficulty in prioritizing tasks in busy healthcare environments, evidence-based clinical practices (based on external knowledge), instead of individual clinical expertise, can help upgrade the foundation of data-driven care, which separates clinicians from their responsibilities for implementation of new knowledge [59]. Finally, seamless care is important in terms of system management. This lack can result in lagging, shut down, and even collaborative care [52]. Thus, it has been recently recommended to exchange messages between clinicians and access by smartphones [60]. Consequently, continuous system management will reduce response times, accelerate the work speed of clinicians, and minimize clicking errors, thereby increasing their overall system satisfaction.

In EMR systems, better support for decision making and actions can help to optimize the processes and outcomes related to delivering, analyzing, and improving care. This requires receiving the correct information about the correct person in the correct intervention format, using the correct channel and at the right workflow time [61]. This study indicates that implementing the correct intervention is crucial in human–computer interactions, which can be classified into four categories. In data entry tasks, various smart documentation formats should be provided, such as order sets, care planning, parameter instructions, and immediate alerts. In data review tasks, well-organized single-patient data summaries should be provided and rapid multi-patient monitoring should also be enabled. Simple analysis should be possible using predictable and backdated data. When evaluating tasks, filtered reference information, knowledge resources, and professional and technical advisors should be available to increase understanding. Finally, supports not triggered by user tasks should be driven by alerts and reminders that are dependent on relevant data and time, respectively. All examples of these EMR usability enhancements are illustrated in the guidelines from the Safety Enhanced Design Briefs by SHARPC in 2016 (See Fig. 4) [62].

Fig. 4

Visual enhancements of EMRs from Safety Enhanced Design Briefs (SEDB-G03; SEDB-G04; SEDB-MU01; SEDB-MU02): (a) patient list, (b) medication list, (c) decision-making support (drug–drug interaction), (d) order results [62].

Discussions on whether better methods are available for investigating the usability problems of EMR systems are important. Previous studies have attempted to compare four different methods: surveys, observations, interviews, and investigating individual inherent usability issues with respect to EMR systems. A survey and think-aloud approach were applied with results indicating that “system response time” and “layout/screen organization” were the most significant usability issues, respectively [12]. However, no single best method has been established to describe usability issues. Moreover, different methods are best suited for evaluation at different stages of the design and layout process and offer various usability characteristics [25, 63]. However, problems may vary based on the specific usability assessment; thus, it is important to perform additional assessment to increase the usefulness of systems.

In conclusion, this study proposes a scenario-based evaluation for future research to evaluate a specific task and provide narrative responses in real-time about interactive systems. Based on usability problems, the use of surveys as a usability test can be supported to determine the actual behavior of clinicians, how pain points can be effectively improved, and preferably how new interfaces can be retested by actual users. In terms of usability, safety is a more important and urgent aspect than ease of use of an EMR system, which differs from other information technology systems. Therefore, various user groups should be considered in all usability assessments to achieve the best possible flexibility.

5 Conclusion

This study investigated user-related problems of EMR systems in an ED environment and identified key requirements for the practical improvement of their usability. The results indicated the primary issues, and their causes were summarized. The most important finding was the requirement for significant improvement in the interface design of user tasks, such as patient selection, order medication, and order search. Survey data were combined with the qualitative feedback of clinicians to describe how users respond to the interface for each system function. Consequently, an effective solution for user-friendly information presentation in EMR systems included a more intuitive design, particularly related to common clinical tasks such as screen recognition and document and order entry. In addition, system designers must provide task-based action items to clinical tasks, for which the users indicated the strongest need, with a wide range of options and supportive decision-making tools. System capabilities in terms of organizational management can also improve user satisfaction for the system. All findings are expected to improve the user interface of EMR systems and provide reference data for formulating visual style guidelines. This also applies to usability and design consistency in other clinical applications.

This study was based on real-world user experience of an EMR in an ED with 200 clinicians. In addition, a questionnaire was developed considering the clinical workflow (applicable to general ED context), which can be used in other medical environments. Finally, structurally summarized subjective responses provided insights for analyzing EMR usability problems in integrated approaches. We expect that the framework can be very useful for understanding complex EMR usability problems, not only interface design but also work environment design. Continuous improvement of the usability of EMR systems will provide users with better working conditions and increase their satisfaction.

Footnotes

Acknowledgments

This study was supported by a grant of the Korean Health Technology R&D Project, Ministry of Health & Welfare, Republic of Korea (no. HI15C1101).

Conflict of interest

None to report.

Supplementary data

The supplementary files are available from .

References

Ajami

, Bagheri-Tadi

. Barriers for Adopting Electronic Health Records (EHRs) by Physicians. Acta Inform Med. 2013:21(2):129–34.

Chandak

, Shinde

. The effectiveness of use of electronic medical record system at university health centre. Indian Journal of Public Health Research and Development. 2019:10(7):1526–31.

Buntin

, Burke

, Hoaglin

, Blumenthal

. The Benefits Of Health Information Technology: A Review Of The Recent Literature Shows Predominantly Positive Results. Health Affairs. 2011:30(3):464–71.

Weeger

, Gewald

. Acceptance and use of electronic medical records: An exploratory study of hospital physicians’ salient beliefs about HIT systems. Health Systems. 2015:4(1):64–81.

McDonald

, Callaghan

, Weissman

, Goodwin

, Mundkur

, Kuhn

. Use of internist’s free time by ambulatory care electronic medical record systems. JAMA Internal Medicine. 2014:174(11):1860–3.

Wilson

, Michel

, Olsen

, Gibberd

, Vincent

, El-Assady

, et al. Patient safety in developing countries: retrospective estimation of scale and nature of harm to patients in hospital. BMJ. 2012:344, e832.

Farzandipour

, Meidani

, Riazi

, Sadeqi Jabali

. Task-specific usability requirements of electronic medical records systems: Lessons learned from a national survey of end-users. Inform Health Soc Care. 2018:43(3):280–99.

Schooley

, Barber

, Birtwhistle

, Hall-Barber

, Ellis

, Flavin

, et al. An electronic asthma performance indicator reporting system: Use of standardized electronic medical record data at the point of care. Canadian Journal of Respiratory Critical Care and Sleep Medicine. 2017:1(2):110–1.

Johnson

, Johnston

, Crowle

. EHR usability toolkit: A background report on usability and electronic health records. Rockville, MD: Agency for Healthcare Research and Quality. 2011.

10.

Bhesania

, Mehta

, Moore

, Dinhofer

. 10 recommendations to improve electronic health record satisfaction and reduce burnoutamong physicians. Journal of General Internal Medicine. 2018:33(2):757.

11.

Sutton

, Ash

, Al Makki

, Kalakeche

. A Daily Hospital Progress Note that Increases Physician Usability of the Electronic Health Record by Facilitating a Problem Oriented Approach to the Patient and Reducing Physician Clerical Burden. The Permanente Journal. 2019;23.

12.

Hudson

, Kushniruk

, Borycki

, Zuege

. Physician satisfaction with a critical care clinical information system using a multimethod evaluation of usability. International Journal of Medical Informatics. 2018:112:131–6.

13.

Ratwani

, Savage

, Will

, Arnold

, Khairat

, Miller

, et al. A usability and safety analysis of electronic health records: a multi-center study. J Am Med Inform Assoc. 2018:25(9):1197–201.

14.

Walley

, Peden

, May

. Laboratory Order Errors Before and After Implementation of Electronic Health Record. Clinical Laboratory Science. 2016:29(3):158–62.

15.

Khajouei

, Abbasi

, Mirzaee

. Errors and causes of communication failures from hospital information systems to electronic health record: A record-review study. International Journal of Medical Informatics. 2018:119:47–53.

16.

Qian

, Munyisia

, Reid

, Hailey

, Pados

, Yu

. Trend in data errors after the implementation of an electronic medical record system: A longitudinal study in an Australian regional Drug and Alcohol Service. International Journal of Medical Informatics. 2020:144:7.

17.

Lee

, Kim

, Lee

. Developing Transnational Health Record Framework with Level-Specific Interoperability Guidelines Based on a Related Literature Review. Healthcare. 2021:9(1):67.

18.

Jung

B-Y

, Kim

S-Y

, Sung

H-K

, Park

J-S

, Go

H-Y

, et al. Statues and improvement of electronic medical record system in traditional Korean medicine. Journal of pharmacopuncture. 2018:21(3):195.

19.

Belden

, Grayson

, Barnes

. Healthcare Information and Management Systems Society (HIMSS). 2009. Defining and testing EMR usability: principles and proposed methods of EMR usability evaluation and rating URL: https://www.himssorg/file/1268216/download.2019.

20.

Salvendy

. Handbook of human factors and ergonomics: John Wiley & Sons; 2012.

21.

Shackel

. Usability–context, framework, definition, design and evaluation. Interacting with computers. 2009:21(5-6):339–46.

22.

Taieb-Maimon

, Plaisant

, Hettinger

, Shneiderman

. Increasing Recognition of Wrong-Patient Errors through Improved Interface Design of a Computerized Provider Order Entry System. International Journal of Human-Computer Interaction. 2018:34(5):383–98.

23.

Coleman

, Gotz

, Eaker

, James

, Bice

, Carson

, et al. Analysing EHR navigation patterns and digital work flows among physicians during ICU pre-rounds. Health Information Management Journal. 2020:1833358320920589.

24.

Makam

, Lanham

, Batchelor

, Samal

, Moran

, Howell-Stampley

, et al. Use and satisfaction with key functions of a common commercial electronic health record: a survey of primary care providers. Bmc Medical Informatics and Decision Making. 2013:13(1):86.

25.

Brown

, Balatsoukas

, Williams

, Sperrin

, Buchan

. Multi-method laboratory user evaluation of an actionable clinical performance information system: Implications for usability and patient safety. J Biomed Inform.. 2018:77:62–80.

26.

Theis

, Schäfer

, Bröhl

, Schäfer

, Rasche

, Wille

, et al. Predicting technology usage by health information need of older adults: Implications for eHealth technology. Work. 2019:62(3):443–57.

27.

Denton

, Soni

, Kannampallil

, Serrichio

, Shapiro

, Traub

, et al. Emergency Physicians’ Perceived Influence of EHR Use on Clinical Workflow and Performance Metrics. Applied Clinical Informatics. 2018:9(3):725–33.

28.

Van den Heede

, Van de Voorde

. Interventions to reduce emergency department utilisation: A review of reviews. Health Policy. 2016:120(12):1337–49.

29.

Vaidotas

, Yokota

PKO

, Negrini

NMM

, Leiderman

DBD

, Souza

, Santos

, et al. Medication errors in emergency departments: is electronic medical record an effective barrier? Einstein (Sao Paulo). 2019:17(4):eGS4282.

30.

Yuwanich

, Sandmark

, Akhavan

. Emergency department nurses’ experiences of occupational stress: A qualitative study from a public hospital in Bangkok, Thailand. Work. 2016:53(4):885–97.

31.

Lee

, Jung

, Shin

, Bahn

, Park

, Cho

, et al. Safety and Usability Guidelines of Clinical Information Systems Integrating Clinical Workflow: A Systematic Review. Healthc Inform Res. 2018:24(3):157–69.

32.

Brown

, Mulcaster

, Triffitt

, Sittig

, Ash

, Reygate

, et al. A systematic review of the types and causes of prescribing errors generated from using computerized provider order entry systems in primary and secondary care. Journal of the American Medical Informatics Association. 2017:24(2):432–40.

33.

Yoo

, Hwang

, Jheon

. Hospital information systems: experience at the fully digitized Seoul National University Bundang Hospital. Journal of Thoracic Disease. 2016:8(Suppl 8):S637–S41.

34.

Mills

. Electronic Health Records and Use of Clinical Decision Support. Crit Care Nurs Clin North Am. 2019:31(2):125–31.

35.

Sultanum

, Brudno

, Wigdor

, Chevalier

, editors. More text please! understanding and supporting the use of visualization for clinical text overview. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems; 2018.

36.

Dobrzykowski

, Tarafdar

. Linking electronic medical records use to physicians’ performance: A contextual analysis. Decision Sciences. 2017:48(1):7–38.

37.

Mosaly

, Mazur

, Yu

, Guo

, Derek

, Laidlaw

, et al. Relating task demand, mental effort and task difficulty with physicians’ performance during interactions with electronic health records (EHRs). International Journal of Human–Computer Interaction. 2018:34(5):467–75.

38.

Kaipio

, Lääveri

, Hyppönen

, Vainiomäki

, Reponen

, Kushniruk

, et al. Usability problems do not heal by themselves: National survey on physicians’ experiences with EHRs in Finland. International Journal of Medical Informatics. 2017:97:266–81.

39.

Colicchio

, Cimino

. Clinicians’ reasoning as reflected in electronic clinical note-entry and reading/retrieval: a systematic review and qualitative synthesis. Journal of the American Medical Informatics Association.. 2019:26(2):172–84.

40.

Senathirajah

, Borycki

, Kushniruk

, Cato

, Jinglu

. Use of Eye-Tracking in Studies of EHR Usability - The Current State: A Scoping Review...17th World Congress of Medical and Health Informatics, 25-30 August 2019, Lyon, France. Studies in Health Technology & Informatics. 2019:264:1976–7.

41.

Pfaff

, Eris

, Weir

, Anganes

, Crotty

, Rahman

, et al. Analysis of the Cognitive Demands of Electronic Health Record Use. Journal of Biomedical Informatics. 2020:103633.

42.

Momah

, Stanley

. The Role of the Electronic Medical Record in Workflow Design and Provider Satisfaction. Journal of the National Medical Association. 2020:112(5):S32.

43.

Tajirian

, Stergiopoulos

, Strudwick

, Sequeira

, Sanches

, Kemp

, et al. The influence of electronic health record use on physician burnout: Cross-sectional survey. Journal of Medical Internet Research. 2020;22(7).

44.

Vos

JFJ

, Boonstra

, Kooistra

, Seelen

, van Offenbeek

. The influence of electronic health record use on collaboration among medical specialties. BMC Health Services Research. 2020:20(1):676.

45.

Prgomet

, Li

, Niazkhani

, Georgiou

, Westbrook

. Impact of commercial computerized provider order entry (CPOE) and clinical decision support systems (CDSSs) on medication errors, length of stay, and mortality in intensive care units: a systematic review and meta-analysis. J Am Med Inform Assoc. 2017:24(2):413–22.

46.

Horsky

, Schiff

, Johnston

, Mercincavage

, Bell

, Middleton

. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform. 2012:45(6):1202–16.

47.

Marcilly

, Ammenwerth

, Roehrer

, Niès

, Beuscart-Zéphir

M-C

. Evidence-based usability design principles for medication alerting systems. BMC Medical Informatics and Decision Making. 2018:18(1):1–17.

48.

Laurant

, van der Biezen

, Wijers

, Watananirun

, Kontopantelis

, van Vught

. Nurses as substitutes for doctors in primary care. Cochrane Database of Systematic Reviews. 2018(7).

49.

ECRI Institute PSO. ECRI Institute PSO Deep Dive: Health Information Technology. 2012: Plymouth Meeting, PA.

50.

Park

, Chen

, Rudkin

. Technological and organizational adaptation of EMR implementation in an emergency department. ACM Transactions on Computer-Human Interaction (TOCHI). 2015:22(1):1–24.

51.

Stone

, Oakman

. Oral health professionals: An exploration of the physical and psychosocial working environment. Work. 2020(Preprint)):1–9.

52.

Arceneaux

. Identifying Physicians’ User Experience (Ux) Pain Points in Using Electronic Health Record (EHR) Systems: Nova Southeastern University; 2020.

53.

Kim

, Howe

, Franklin

, Krevat

, Jones

, Adams

, et al. Health Information Technology–Related Wrong-Patient Errors: Context is Critical. Patient Safety. 2020:2(4):40–57.

54.

Dutta

, Hwang

H-G

. The adoption of electronic medical record by physicians: A PRISMA-compliant systematic review. Medicine. 2020;99(8).

55.

Zheng

, Ratwani

, Adler-Milstein

. Studying workflow and workarounds in electronic health record–supported work to improve health system performance. Annals of Internal Medicine. 2020:172(11_Supplement):S116–S22.

56.

Lanier

, Dao

, Baer

, Haller

, Sommer

, Perron

. How Do Patients Want Us to Use the Computer During Medical Encounters?—A Discrete Choice Experiment Study. Journal of General Internal Medicine. 2021:1-8.

57.

Galitz

. The essential guide to user interface design: an introduction to GUI design principles and techniques: John Wiley & Sons; 2007.

58.

King

, Cooper

, Clermont

, Hochheiser

, Hauskrecht

, Sittig

, et al. Leveraging Eye Tracking to Prioritize Relevant Medical Record Data: Comparative Machine Learning Study. J Med Internet Res. 2020:22(4):e15876.

59.

Honoré

, Pallesen

, Ellegård

, Feldbæk Nielsen

. Mind the gap. How job task distributions of health professional developers constitute evidence-based practice. Work. (Preprint):1–11.

60.

do Carmo Alonso

, de Lima

, Oggioni

BdMP

, Teixeira

, Oliveira

, Couto

MCV

, et al. Contributions of activity ergonomics to the design of an electronic health record to support collaborative mental care of children and youth: Preliminary results. Work. 2020:65(1):187–94.

61.

Osheroff

, Teich

, Levick

, Saldana

, Velasco

, Sittig

, et al. Improving outcomes with clinical decision support: an implementer’s guide: HIMSS Publishing; 2012.

62.

SHARPC, National Center for Cognitive Informatics & Decision Making in Healthcare. Safety Enhanced Design Briefs. 2016. Available from: https://sbmi.uth.edu/nccd/SED/Briefs/index.htm.

63.

Horsky

, McColgan

, Pang

, Melnikas

, Linder

, Schnipper

, et al. Complementary methods of system usability evaluation: surveys and observations during software design and development cycles. J Biomed Inform. 2010:43(5):782–90.

Category	Level	Comments	Physician Frequency (%)	Nurse Frequency (%)	Total Frequency (%)
Design/usability; poorly designed EMRs influence performance of clinicians	1st	Not capturing information at glance	101 (11.88)	53 (7.56)	821 (53.84)
		•screen extremely busy with additional windows
		•small size of fonts and narrow column/row
		•similar names of patients, drugs, and orders
		Confusing and nonintuitive methods of delivering information	26 (3.06)	37 (5.28)
		•mixed information in list
		•difficulty in separating left/right or positive/negative
		Irrelevant opening of additional windows	16 (1.88)	22 (3.14)
		•frequent mouse controls due to screen truncation and multiple windows
		•wrong records in window of another patient or user
		Complex searching structures with additional queries	71 (8.35)	66 (9.42)
		•clicks on numerous tabs and checkboxes
		•difficult to convert Korean to English characters
		•repetitive retyping from beginning
		•inconvenient application for views
		Ambiguous classification scheme and lack of options	89 (10.47)	65 (9.27)
		•lack of options (no search result)
		•excessive standardization in terms of options
		•ambiguous matching problems (replacement with other similar options)
		•no helpful tool for providing subjective impression about symptoms of patient
		Stringent template with mandatory records	22 (2.59)	27 (3.85)
		•limited number of characteristics (leading to unclear description)
		•stringent against incomplete records with every popup
		•increased time for re-charting
		•numerous mandatory records based on every timeline
		Incomplete or erroneous records	113 (13.29)	77 (10.98)
		•no alarm when wrong pasting or shortcut
		•incorrect or omitted inputs in required (or defaulted) items
		•including common input errors, such as usage and volume
		Overall lack of functionality	12 (1.41)	24 (3.42)
		•no filtering, select-all, and temporary saving features
		•no navigation and customization
Individual factors; degree of freedom from time pressure influence on working capabilities of clinicians	2nd	Human-related errors with high task burdens and low perception	167 (19.65)	51 (7.28)	218 (13.97)
		•various clicking errors (failing on-target and double vs. single)
		•scrolling errors
		•typing errors
Psychosocial work conditions; psychosocial work conditions of clinicians adversely influence patient care quality	3rd	Limited work conditions of ED	33 (3.88)	38 (5.42)	71 (4.55)
		•crowded physical environment
		•severe time pressure of outsources
		•uncooperative patients
		•independent system of EMR (poor interoperability)
	4th	Disconnected organizations	12 (1.41)	18 (2.57)	202 (13.25)
		•nonsearchable hospital names in system
		Communication, teamwork, and decision-making structure	82 (9.65)	90 (12.84)
		•different abbreviations and interfaces by ward
		•overlapping records (concurrency)
		•time conflict between ordering and charting
	5th	System management	60 (7.06)	82 (11.70)	213 (13.97)
		•unintended saving errors
		•low speed of system with buffering
		•abrupt shutdown
		Other supports (maintenance)	20 (2.35)	51 (7.28)
		•inappropriate synchronization
		•strong desire for real-time takeover
		•log in and log out of another user