Mathematical Model for Computer-Assisted Modification of Medication Dosing Rules

Abstract

Objective:

Medication dosing in pediatrics is complex and prone to errors that may lead to patient harm. To improve computer-assisted dosing, a mathematical model and algorithm were developed to optimize clinical decision support dosing rules and reduce spurious alerts. The objective was to evaluate the feasibility of using this algorithm to adjust dosing rules.

Materials and methods:

Incorporating historical ordering data, a mathematical model and algorithm were developed to automatically determine optimal dosing rule parameters. The algorithm optimizes the dosing rules by balancing the number of alerts generated for a medication with a minimal length dose interval. In all, 5 candidate medications were tested. An analysis was performed to compare the number of alerts generated by the new model with the current dosing rules.

Results:

For the 5 medications, the algorithm generated multiple clinically relevant rule possibilities and the rules returned performed as well as current dosing rule or matched historical prescriber behavior. The rules were comparable to or better than the existing system rules in reducing the total alert burden.

Discussion:

The mathematical model and algorithm are an accurate and scalable solution to adjusting medication dosing rules. They can be implemented to change suboptimal rules more quickly than current manual methods and can be used to help identify and correct poor quality rules.

Conclusions:

Mathematical modeling using historic prescribing data can generate clinically appropriate electronic dosing rule parameters. This approach represents an automatable and scalable solution that could help reduce alert fatigue and decrease medication dosing errors.

Keywords

electronic health record models theoretical medical order entry systems pediatrics decision support systems clinical

Introduction and Background

Computerized Provider Order Entry (CPOE) is a core functionality of electronic health records (EHRs) and is a system where health care providers enter medication orders for use in patient care. Clinical decision support (CDS), which provides assistance in clinical decision-making, is often implemented within CPOE. Electronic medication rules (eRules) help the prescriber determine whether a medication order is within reasonable and safe parameters based on factors such as the weight and age of a patient.¹ If an order does not satisfy the conditions of the eRule, it will produce an alert or some other form of notification to the prescriber.

Prescribers override CDS alerts for a variety of reasons. As clinicians are familiar with traditional dosing patterns (often through textbooks and other sources of truth), they will often ignore electronic alerts because they feel the system is overwhelming or that the alerts are inaccurate, clinically inappropriate, or simply not helpful.² All these factors contribute to alert fatigue, which increases overrides of medication order-related CDS alerts.³ This raises safety concerns because clinicians are then more likely to ignore a correct alert (true positive) that may be lost among a sea of false-positive alerts.

In pediatrics, it is important that dosing eRules are accurate because medication dosing is extremely complex due to weight-based factors and is prone to error.^4-8 Weight-based prescribing errors are often due to single-dose overdoses that lead to adverse drug events (ADEs). Unfortunately, most of the eRules in modern CDS systems are not tailored for pediatrics and orders entered into CPOE produce high amounts of alerts. Improving the eRules to make the CDS more accurate for pediatrics will theoretically reduce alert burden (thereby improving alert and CDS salience) and dosing errors, improving medication safety. Efforts have been made to change some of these rules and changes have been shown to reduce alerts on a small scale.⁹ However, medication-related CDS on the whole is still immature in many health care organizations; specificity of the alerts themselves is frequently cited as a main cause.¹⁰ This phenomenon of poor attention to alerts is seen in both inpatient¹¹ and outpatient environments.¹² Very little evidence exists regarding the effectiveness of medication-related CDS interruptive ordering alerts (especially in pediatrics), and what does exist shows mixed results of effectiveness.¹³ Attempts to change the specificity and performance of alerts on a large scale is challenging as common vendor eRule databases contain hundreds of thousands of records. The modification of medication rules is labor-intensive and minor changes in dosing parameters of one medication formulation may require the modification of dozens of eRules due to the granularity of the databases. Proper configurations of CPOE and CDS systems have the potential to mitigate this issue.^14,15 Currently, there is no automatable, scalable, and efficient solution to this problem. The application of computational and mathematical modeling to historic prescribing data presents an opportunity to overcome these issues.

Objective

The objectives of this study were to (1) develop a mathematical model that can optimize eRule dosing parameters based on historical order data, (2) use the model to develop an algorithm that determines optimal medication dosing rule parameters, and (3) perform an offline simulation test and evaluate the process, comparing the output against our institution’s existing operationally active eRules. Our hypothesis is that the model and algorithm will be able to empirically adjust the dosing rule parameters and will perform favorably when compared with the performance of current eRules.

Materials and Methods

Setting

Cincinnati Children’s Hospital Medical Center is a level-1 trauma center with 628 licensed beds. It has 1.2 million patient encounters annually, including 30 000 admissions, 33 000 surgeries, 900 000 ambulatory encounters, and 125 000 emergency department visits. There are approximately 200 000 medication orders a month, which generate 75 000 dosing alerts. The institution implemented an enterprise EHR (EpicCare^®, Verona, WI, USA) in 2007. The EHR is configured to use a combination of the Medi-Span (Wolters Kluwer Health, Philadelphia, PA, USA) drug dosing decision support rules and supplemented with custom dosing rules created and maintained by the pharmacy.

Dosing rules description

There are 2 sets of dosing rules used in this study: actual dosing rules and the model-generated dosing rules. The actual dosing rules are the electronic rules (eRules) that are used in the study site’s production CPOE system rules engine and clinical care. The actual dosing rules were purchased from a third-party vendor. All the dosing parameters for the candidate medications selected in this study were already customized and overrode the vendor’s default rule parameters. The model-generated dosing rules are the dosing rules output by the mathematical model created to optimize the rule parameters. The 2 sets of dosing rules represent different CPOE configurations for weight-based dosing CDS. The theoretical performances of these sets of dosing rules were compared with demonstrate the viability and feasibility of the model-generated dosing rules (refer sections “Assessing the performance of the Model-generated Dosing Rules” and “Results”).

Clinical dataset for dosing rules model development

Historical medication order data from 2011 to 2015 were retrieved from a previously described decision support analytic data warehouse.^9,16 A total of 5 medications were selected to use in developing the model: Acetaminophen, Ibuprofen, Diphenhydramine, Amoxicillin, and Ursodiol. The order counts and the Actual eRules in the CPOE of these medications are listed in Table 1. These medications were selected based on criteria that included frequency of ordering and frequency of alerting. Ursodiol was selected because it was previously known to have a poor match between the formal dosing eRules and how it is ordered in practice. A pediatrician (ESK) aggregated different formulations for the selected medications based on ordering behavior (Table 2). For example, in our dataset, Ibuprofen has formulations of suspension, tablets, chewable tablets, solution, and capsule. These were grouped together for analysis as those formulations are typically prescribed with the same dosing guidelines. The data were filtered for orders placed using weight-based dosing, eg, in milligram/kilogram. Orders placed using absolute dosing, eg, in milligrams, were excluded as the model calculated weight-based dosing parameters only. The data include all medication dosing orders that were attempted as well as orders canceled and removed by prescribers after they attempted to sign the orders and received an alert.

Table 1.

Dosing rule parameters (actual eRule) and medication data counts.

Medication	Actual eRule	Order count
Acetaminophen	5-15 mg/kg	376 794
Diphenhydramine	0.1-1.5 mg/kg	163 225
Ibuprofen	4-11 mg/kg	42 884
Amoxicillin	8-45 mg/kg	61 323
Ursodiol 0-12^a	10-15 mg/kg	1191
Ursodiol 12-99^a	2-5 mg/kg*	39
Total		645 456

Ursodiol has a different eRule for different age groups: the split was based on the construct of the current eRules.

Table 2.

Medication formulation groupings.

Medication	Cap	Tab	Chew Tab	Solution	Liquid	Elixir	Suspension	Recon suspension	Gel	Syrup	Tablet dispersible
Acetaminophen	X	X	X	X	X	X	X		X	X	X
Diphenhydramine	X	X	X	X	X	X
Ibuprofen	X	X	X	X			X
Amoxicillin	X	X	X					X
Ursodiol 0-12	X						X
Ursodiol 12-99	X						X

Generation of the artificial dataset

The mathematical model and optimization algorithms require dosing data to generate meaningful dosing rules. Using the assumption that future dosing behavior can be approximated using historical dosing data, we could generate dosing rules directly using historical data. A disadvantage of the direct use of historical data is that it neglects the potential for the appearance of unseen doses in the future. For example, typographical mistakes where the user hits the wrong/multiple keys or unit errors where the user inputs the incorrect units could be reasonably expected to occur eventually. For frequently ordered medications, the probability of an unseen dose appearing in the future would be small, but for infrequently ordered medications, using only historical data may not be sufficient. To incorporate the knowledge that doses unseen in historical datasets could appear in the future, we develop a model of future historical dosing behavior that is statistically consistent with historical dosing data and incorporates unseen doses.

The clinical dataset described in section “Clinical dataset for dosing rules model development” was used to generate an artificial dataset. Our artificial data model is based on 2 assumptions: that future ordering behavior will match historical ordering behavior and that there exists doses that were unseen in the historical data but could appear in the future. Here, we use a frequentist statistical model based on these assumptions. Other approaches to generate meaningful artificial data such as Bayesian models can be used in lieu of this approach as the mathematical model and optimization algorithm are indifferent to the origin of the data.

For large historical dosing datasets from commonly prescribed medications such as Acetaminophen or Ibuprofen, the probability of a specific dose can be estimated using the maximum likelihood estimate (MLE), where the probability is the frequency of the specific dose divided by the total number of orders. A limitation in this approach is that doses that do not appear in the historical dataset have a zero probability of occurring in the artificial dataset. Various data entry errors that might have not occurred historically but have a small but non-zero probability of occurring in the future would be neglected from our study. It is important that the probability of these orders is not zero.

To address this limitation, we use an alternative statistical model to the MLE that assigns certain unseen dose orders a non-zero probability. Our statistical model uses Good-Turing frequency estimation (GTE)^17-19 to generate the probability of set of seen and unseen dose orders.

From the historical dataset, we have the set, $D_{observed}$ , that contains all the observed dose orders and we can compute the frequency each dose is prescribed. We define the frequency of dose frequency, $N_{r}$ , as the number of doses in $D_{observed}$ ordered with the frequency $r$ . For example, $N_{1}$ is the number of doses that were ordered only once in the dataset. The total number of orders can be computed from the frequency of frequency data by the relation

N = \sum_{r = 1}^{\infty} r N_{r} .

The MLE estimate of the probability a specific dose is ordered $r$ times is $p_{r} = r / N$ and the probability of a dose not in $D_{observed}$ is $p_{0} = 0$ . The GTE of the total probability of all unseen doses is estimated by

p_{0} = \frac{N_{1}}{N} .

Note that this is the probability that any unseen dose will appear in the future, not the probability of a single, specific unseen dose. The probability of individual doses in $D_{observed}$ that appear with frequency $r$ is estimated by

p_{r} = \frac{(r + 1) S (N_{r + 1})}{N S (N_{r})},

where $S (N_{r})$ denotes a smoothed frequency estimate generated by linear regression.^17,20

The GTE estimates the total probability of all unseen dose orders but we need to generate specific dose values that are unseen. As the set of possible unseen orders is uncountably infinite, we will restrict it to a finite set of more probable unseen doses, $D_{unseen}$ containing $N_{0}$ elements. For each dose in $D_{unseen}$ , we assign an equal probability

{\tilde{p}}_{0} = \frac{p_{0}}{N_{0}} = \frac{N_{1}}{N_{0} N} .

The set of all possible doses from the artificial data generator will be the union of the 2 sets $D = D_{observed} \cup^{} D_{unseen}$ . To generate artificial data from the GTE of the probabilities, we begin by computing the discrete cumulative distribution function (CDF) for the dose set $D$ : for dose $d \in D$

C D F (d) = \sum_{d_{j} < d} p (d_{j}),

where $d_{0} < d_{1} < \dots < d_{j} < d_{M}$ are the sorted doses in $D$ , $M$ is the total number of possible dose orders, and $p (d_{j})$ is the GTE of the probability of dose. As the CDF is a monotonically increasing function and $0 ″ C D F (d) ″ 1$ , we can use the CDF to map a uniform random variable, $0 ″ X ″ 1$ , to a specific dose through the relation

d = \max_{d_{j} \in D} C D F (d_{j}) \leq X .

This relation is used to generate the artificial data by mapping a set of values from a uniform random distribution to a set of dose orders. Python (version 3.5.1, 2015; Python Software Foundation, Delaware, Beaverton, USA) was used to generate the historical frequency of frequency data, implement the GTE, and the artificial dose data and the respective histograms and to implement the following model and optimization algorithm. A total of 100 000 order instances were generated for each medication.

Development of the dosing rule model

Based on our goal of decreasing alert burden to improve alert salience, we created a model to optimize CDS dosing eRules with the express intent of reducing alerts. The model evaluates a discrete number of possible rule range intervals, based on the number of alerts each range would produce when the artificial dataset is applied, ie, it assesses the number of theoretical alerts that would be generated with the model dosing limits. The multiple discrete combinations of dosing rule intervals considered are based on the values of where most of the medication dose orders occur, eg, as most dosing orders for Acetaminophen occur between 0 and 20 mg/kg, multiple combinations of rule intervals between 0 and 20 are considered. To find the optimal rule, we must assign a score to each one being considered. We developed a measure of the quality of any given rule. As it is mathematically easier to study the inverse of the quality of the dosing rules, we propose this measure as the inferiority score of a dosing rule. Our inferiority measure assigns a score to each rule based on the number of theoretical alerts produced. We assume that our quality measure of a given dosing rule depends on 2 factors:

The number of alerts produced by a set of simulated dosing data: our goal is to empirically create dosing rules that are both clinically accurate (increase true positive alerts and true negatives) and decrease alert burden (through reduction of false-positive alerts). A dosing rule that generates a large number of alerts will, in general, have a poor quality measure.

Length of the rule interval: This term penalizes the quality measure if the dosing interval size is larger than the length of the actual eRule, which is included to prevent the degenerative case where an arbitrary large interval that produces no alerts is acceptable.

The function will return an inferiority score for each dosing rule. A high inferiority score for a dosing rule range means that this rule is producing high amounts of alerts or that the rule interval is too large. A low inferiority score means that a dosing rule has few alerts and an acceptable interval size. Given a rule interval $R = (x, y)$ , where $x$ is the lower dosing limit and $y$ is the upper dosing limit and a series of dose orders, $E$ , the measure of the inferiority, $I$ , of a rule is defined as

I (R, E) = w_{1} \frac{A}{N} + w_{2} \frac{| y - x |}{L_{0}},

where $A$ is the number of alerts generated from the set of orders, $E$ ; $N$ is the total number of orders in the dataset; $L_{0}$ is the length of the actual dosing rule interval; and $w_{1}$ and $w_{2}$ are arbitrary weights. Each term is scaled so that the alert term is less than 1 and the other term has a value close to unity. The weights are arbitrary and affect the relative importance between the 2 terms. For example, having $w_{1} > w_{2}$ implies that reducing the number of alerts is more important than reducing the interval length when improving the quality of the rule. Note that these 2 terms are proxies for false positives (total number of alerts) and false negatives (interval length) and the weights can be viewed as their relative importance. While these proxies are crude, they could be an improvement compared with the status quo until more accurate proxies are developed.

As multiple discrete combinations of dosing rules are considered, multiple inferiority scores are returned. The goal is to find the lowest inferiority score, which will be associated with a dosing rule interval.

Implementation of the dosing interval optimization algorithm

An algorithm was implemented to find the lowest inferiority score. The algorithm computed the inferiority scores for each dosing interval and stored the values in a grid where the position in the grid corresponds to the lower and upper dosing rule limit. For optimization, the algorithm found the lowest score in the grid and returned the associated lower and upper rule limits of the dosing interval. Surface plots of the resulting grids were visualized using MATLAB (Release 2012b; The MathWorks, Inc., Natick, MA, USA).

Applying a weights simulation algorithm

In lieu of choosing the weights a priori, we developed an algorithm that assesses the impact of various weights on the model output. We prioritize reducing the number of alerts, so we chose to restrict the weights where $w_{1} > w_{2}$ . As it is the relative sizes of the weights to each other, we set $w_{2} = 1$ and scan the results for various values of $w_{1}$ . We use a modified form of the bisection method to search through the range of weight values, $w_{1}$ , looking for the critical weight values where changes in the model’s dosing rule output occur. Once each critical weight value was found, the associated dosing rules returned were recorded and considered as part of the model’s output. This provides a complete set of optimal dosing rules based on the relative importance of reducing the number of alerts to the interval length.

Assessing the performance of the model-generated dosing rules

Finally, we evaluated the theoretical performance of the model-generated dosing rules compared with the actual dosing rules currently applied in the EHR. We computed the alert rates using our artificial dataset on each of the model dosing rules returned and the actual eRule. The alert rate is the percent of dosing orders that generate an alert for a given rule. We then used the alert rates to calculate the percent improvement of the returned dosing rules from the alert rate of the actual eRule as well as the number of alerts saved per 100 000 dosing orders. The integration and flow of all of these processes are shown in Figure 1.

Figure 1.

Flowchart for generating and assessing empirically-derived dosing rules. Process diagram for the study, showing the methods of the study integrated to produce data-driven dosing thresholds.

Results

Artificial datasets were visualized using histograms (Figure 2). The insets allow the user to better understand the distribution of the lesser frequent doses.

Figure 2.

Artificial dataset for (A) Acetaminophen and (B) Ursodiol 0-12 years. The large histograms show 2 examples of the distribution of the artificial dataset doses. The histograms in the insets show the less frequent doses; visualization of these less-frequent doses is difficult otherwise.

After the model computation used the artificial data and a set of weight values to generate inferiority scores, the optimization algorithm found the minimum value for this score and the output was visualized as shown in the example medications (Figure 3). This visualization permits the capability to understand how prospective changes in the dosing rule parameters would affect the score. In the case of acetaminophen, the optimal dosing range is 10-15 mg/kg. Decreasing the upper limit (eg to 13 mg/kg) would drastically impact the inferiority score, while increasing the lower limit to 13 mg/kg would not have such a profound affect.

Figure 3.

Surface plots of dosing rule interval limits; grid output from the optimization algorithm. Each dosing rule limits considered between 0 and 20 mg/kg was assigned an inferiority score. The lowest inferiority score denotes the optimal dosing range as defined by a priori criteria. In both A and B, the lowest inferiority score returned was 10-15 mg/kg in this instance of weight values, thus the optimal dosing interval for both (A) Acetaminophen and (B) Ursodiol (0-12 years) is 10-15 mg/kg.

Table 3 represents the results of the model output and weight simulation experiments; the different rules are the result of the model computation with the weights selected by the weight simulation algorithm. Varying the weights on the model terms output differing optimal rule ranges, with varying affects on alert rates and alerts saved. These tables allow users to quickly scan the model output and select most clinically appropriate rule thresholds that have the largest reductions in alert burden. Figure 4 demonstrates these gains graphically. Gains to commonly and uniformly prescribed medications such as ibuprofen are modest as the threshold limits are widened (Figure 4A; alert reduction gains of 2%-3%), while modest changes to threshold limits impart bigger gains in more heterogeneously prescribed medications such as Ursodiol (Figure 4B; 50%-100% alert reduction).

Table 3.

Weight value simulation dosing rule results.

Weight ratio (r = w1/w2)	eRule	Rule range (mg/kg)		Alert rate	Percent improvement vs actual eRule	Alerts saved per 100 000 orders vs actual eRule
Weight ratio (r = w1/w2)	eRule	Lower	Upper	Alert rate	Percent improvement vs actual eRule	Alerts saved per 100 000 orders vs actual eRule
Acetaminophen
	Actual eRule	5	15	1.20
(2, 54)	Model Rule 1	10	15	1.73	–44.19%	–529
(55, 2734)	Model Rule 2	7.5	15	1.21	–1.00%	–12
(2735, 16 124)	Model Rule 3	5	15	1.20	0.00%	0
(16 125, 24 000)	Model Rule 4	3.5	15	1.14	0.05%	6
Ibuprofen
	Actual eRule	4	11	0.75
(16, 3571)	Model Rule 1	5	10	0.76	–0.80%	–6
(3572, 7117)	Model Rule 2	5	15	0.75	–0.27%	–2
(7118, 20 918)	Model Rule 3	5	15	0.73	2.40%	18
(20 919, 21 938)	Model Rule 4	2.5	15	0.73	2.66%	20
Diphenhydramine
	Actual eRule	0.1	1.5	1.25
(1, 38)	Model Rule 1	0.5	1	3.28	–166.64%	–2133
(39, 1453)	Model Rule 2	0.5	1.5	1.34	–6.25%	–80
(1454, 56 578)	Model Rule 3	0	1.5	1.25	–0.16%	–2
>56 579	Model Rule 4	0	5	1.26	1.64%	21
Amoxicillin
	Actual eRule	8	45	2.06
(1, 4)	Model Rule 1	40	45	13.27	–544.03%	–11 207
(5, 9)	Model Rule 2	25	45	7.03	–241.41%	–4973
(10, 11)	Model Rule 3	20	45	5.26	–155.53%	–3204
(11, 37)	Model Rule 4	12.5	45	2.33	–13.25%	–273
(37, 61)	Model Rule 5	12.5	50	1.70	17.52%	361
(61, 1887)	Model Rule 6	10	50	1.43	30.58%	630
(1888, 3250)	Model Rule 7	7.5	50	1.42	31.07%	640
(3251, 3360)	Model Rule 8	4.5	50	1.41	31.36%	646
Ursodiol 0-12
	Actual eRule	10	15	10.25
(3, 7)	Model Rule 1	10	15	10.25	0.00%	0
(8, 43)	Model Rule 2	7	15	5.18	49.43%	5068
(44, 320)	Model Rule 3	5	15	1.98	80.68%	8271
(321, 802)	Model Rule 4	2	15	1.16	88.73%	9097
Ursodiol 12-99
	Actual eRule	2	5	50.62
(8, 24)	Model Rule 1	5	10	16.08	68.24%	34 538
(25, 119)	Model Rule 2	4	10	11.05	78.17%	39 564
(120, 237)	Model Rule 3	4	14.5	4.08	91.94%	46 535
(238, 498)	Model Rule 4	4	15	3.56	92.96%	47 052

Figure 4.

Visual representation of rule % improvement vs actual eRule. “Actual eRule” label is directly beneath the bar representing the parameters or the actual dosing rule in clinical use. Other bars represent the comparison to different algorithmically derived dosing rule parameter choices. (A) Ibuprofen shows about 3% improvement in alert savings if the dosing rule is 5-15 mg/kg vs the actual eRule of 4-11 mg/kg. (B) Ursodiol shows significant improvement in alert reduction if the rule range is increased.

Discussion

Current methods for adjusting medication dosing rules have many challenges and are largely predicated on time-consuming procedures that are dependent on expert opinion and knowledge. The approach outlined in this study represents an alternative set of procedures for setting and adjusting electronic rule (eRule) dosing parameters, one that is empiric, scalable, and scientific. It uses data generated by the EHR through previous activities by prescribers (empiric), is automatable (thus scalable and addressing the current state of entirely manual adjustment), and can serve as a basis for a CDS learning system.²¹ This secondary use of data means that resources used to input data into the model are minimal—there is no requirement for gathering primary data as substrate. The mathematical model and algorithm are a feasible alternative solution to adjusting medication dosing rules.

The candidate medications for this study were carefully selected. Their commonality allows for plentiful data to serve as the foundation for our test data. They represent one end of the prescribing spectrum. Unfortunately, these medications are also commonly used for a multitude of clinical indications, and the eRules for these drugs have been heavily customized with great scrutiny. The results (Table 2) reflect as such. The dosing intervals are generally very liberal, which limits our ability to demonstrate a large alert savings. Heavy customization of the eRules has also led to a pre-study adjustment that limits the alert savings results, especially with Acetaminophen, Ibuprofen, and Diphenhydramine, all of which are dosed heterogeneously in our clinical environment.

The real value of the model and algorithm will be demonstrated in less mainstream circumstances; in situations where dosing is more heterogeneous, rule customization has not been performed or customization thresholds have been configured that do not match clinical practice. The comparison of the model-generated dosing rules to the customized eRules for Amoxicillin and Ursodiol demonstrates a greater alert savings. The next step for this work should seek to validate the approach and demonstrate realized alert savings in a wider range and number of medication formulations.

Automating the process of dosing rule adjustment through this method does not alleviate the need for clinical judgment. Because the system generates several optimized possibilities for dosing rule thresholds, clinicians with pharmacy expertise need to provide the context to (1) select reasonable parameters presented to them by the system and (2) ensure clinical validity before integrating into practice. The rule that decreases alert burden the most may not be clinically appropriate. Until the model has matured and can receive input regarding the validity of the rules, a human validator will continue to play a key role in the process and this role must understand the balance between alert burden from false positives and safety risks from false negatives. The goal of this work is to reduce that burden in creating better dosing rules, not eliminate the clinical expert entirely or fall prey to automation bias.^22,23 This tool has the capability to rapidly assess all drug rules in a corpus in an automated fashion, identify and prioritize candidate rules for editing based on the potential decrease in alert burden or other factors, and present these rules to a human expert for validation and implementation. Like almost all informatics tools, the goal is to leverage the strengths and capabilities of both humans and machines. The system can become more intelligent and efficient with additional feedback, helping to identify and correct poor-quality rules. The efficient identification and correction of poor quality rules will lead to many attractive gains including decreases in alert burden through fewer false-positive alerts, improved EHR efficiency through a decrease in alert fatigue, and most importantly, potential decreases in medication errors as providers are able to pay more attention to the remaining alerts.⁹

This study has several limitations. First, we assumed that historical ordering patterns are representative of future ordering patterns. The inclusion of data across a long time span mitigates this concern somewhat, but methods to determine clinically accurate changing prescribing patterns (new indications and/or dosing for a medication outside of the normal dosing range) should be considered to overcome this limitation. Another limitation is that the system is based on the assumption that historic prescribing patterns represent appropriate and safe dosing. It is possible that aspects of EHR configurations (such as order sets) could introduce frequent unsafe orders, which could feed into the model and affect its output. At our institution, we have only witnessed this phenomenon on a small scale, but this limitation speaks to the importance of the role of the clinical validator in optimal dosing rule selection process and in evaluating for “false negative” instances, which could lead to a dosing error because no alert was activated.

The generated rules are not patient-specific and the algorithm represents a population-based approach. While this could potentially lessen the clinical impact, with this method we could aggregate data across different populations or disease indication to come up with more specific rules for those patients. Unfortunately, most dosing rules in production systems are not currently patient-specific, including at our institution. As this becomes more common, our method will need to be expanded to account for this. Finally, as demonstrated with the results of this study, the largest gains in decreasing alert burden are minimized by previously implemented, well-constructed operational dosing rules. However, given the hundreds of thousands of dosing rules in vendor products, further study of this automated empiric approach will demonstrate which types of dosing rules (frequently used, most alerted, etc) will return the most value.

Future work includes augmenting the model with additional parameters to increase its accuracy. Incorporating factors such as the type of alerts produced and the override rate of alerts from clinicians’ behavior will be undertaken. Large-scale studies across more types of medications and prescribing environments, as well as for different indications for the same medications, should be performed to further add validity to this approach. The algorithm will be implemented in user-facing software as a tool for pharmacists to use in improving CDS systems. It could then be applied to both pediatric and adult populations.

Conclusions

Mathematical modeling using historic prescribing data can generate more clinically appropriate electronic dosing rule parameters. This approach represents an automatable and scalable solution that could help address alert fatigue and decrease medication dosing errors.

Footnotes

Acknowledgements

The authors would like to thank our research group, the Decision Support Analytics Workgroup (DSAW), for their invaluable assistance and help on this project. The views expressed in this article are authors’ own and are not an official position of the University of Cincinnati, the James M Anderson Center for Health Systems Excellence, or Cincinnati Children’s Hospital Medical Center.

Funding:

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Declaration of conflicting interests:

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Author Contributions

MG was a key member in the design and execution of the project, including responsibility for writing the software, analysis, and drafting and editing of the manuscript. BLV was the primary mentor for MG, overseeing the mathematical sciences aspect of the project, helping design and execute the project. JWD was a key contributor in all aspects of the study, including conceptualization, design, implementation, evaluation, and dissemination. ESK originally created the concept and oversaw the entire study in all phases. All members of the team contributed significantly in all phases of creating the manuscript.

References

Stultz

Nahata

MC.

Computerized clinical decision support for medication prescribing and utilization in pediatrics. J Am Med Inform Assoc. 2012;19:942-953.

van der Sijs

van Gelder

Vulto

Berg

Aarts

. Understanding handling of drug safety alerts: a simulation study. Int J Med Inform. 2010;79:361-369.

Ash

Sittig

Campbell

Guappone

Dykstra

, eds. Some unintended consequences of clinical decision support systems. Paper presented at: AMIA Annual Symposium Proceedings; November 10-14, 2007; Chicago, IL. Bethesda, MD: American Medical Informatics Association.

Koren

Barzilay

Greenwald

Tenfold errors in administration of drug doses: a neglected latrogenic disease in pediatrics. Pediatrics. 1986;77:848-849.

Kozer

Scolnik

Macpherson

et al . Variables associated with medication errors in pediatric emergency medicine. Pediatrics. 2002;110:737-742.

Lesar

TS.

Tenfold medication dose prescribing errors. Ann Pharmacother. 2002;36:1833-1839.

McPhillips

Stille

Smith

et al . Methodological challenges in describing medication dosing errors in children; 2005. https://www.ncbi.nlm.nih.gov/books/NBK20505/.

Wong

Ghaleb

Franklin

Barber

Incidence and nature of dosing errors in paediatric medications: a systematic review. Drug Saf. 2004;27:661-670.

Kirkendall

Kouril

Dexheimer

et al . Automated identification of antibiotic overdoses and adverse drug events via analysis of prescribing alerts and medication administration records. J Am Med Inform Assoc. 2017;24:295-302.

10.

Tolley

Slight

Husband

Watson

Bates

DW.

Improving medication-related clinical decision support. Am J Health Syst Pharm. 2018;75:239-246.

11.

Nanji

Seger

Slight

et al . Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc. 2018;25:476-481.

12.

Nanji

Slight

Seger

et al . Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21:487-491.

13.

Page

Baysari

Westbrook

A systematic review of the effectiveness of interruptive medication prescribing alerts in hospital CPOE systems to change prescriber behavior and improve patient safety. Int J Med Inform. 2017;105:22-30.

14.

Garg

Adhikari

McDonald

et al . Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293:1223-1238.

15.

Kawamoto

Houlihan

Balas

Lobach

DF.

Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330:765.

16.

Kirkendall

Kouril

Minich

Spooner

Analysis of electronic medication orders with large overdoses. Appl Clin Inform. 2014;5:25-45.

17.

Gale

Sampson

Good-turing frequency estimation without tears. J Quant Linguist. 1995;2:217-237.

18.

Good

IJ.

The population frequencies of species and the estimation of population parameters. Biometrika. 1953;40:237-264.

19.

Orlitsky

Suresh

, eds. Competitive distribution estimation: why is good-turing good. Paper presented at: 28th International Conference on Neural Information Processing Systems; December 7-12, 2015; Montreal, QC, Canada. Cambridge, MA: MIT Press.

20.

Church

Gale

WA.

A comparison of the enhanced Good-Turing and deleted estimation methods for estimating probabilities of English bigrams. Comput Speech Lang. 1991;5:19-54.

21.

Schneeweiss

Learning from big health care data. N Engl J Med. 2014;370:2161-2163.

22.

Goddard

Roudsari

Wyatt

JC.

Automation bias: a systematic review of frequency, effect mediators, and mitigators. J Am Med Inform Assoc. 2011;19:121-127.

23.

Lyell

Magrabi

Raban

et al . Automation bias in electronic prescribing. BMC Med Inform Decis Mak. 2017;17:28.