Utilizing Machine Learning and Automated Performance Metrics to Evaluate Robot-Assisted Radical Prostatectomy Performance and Predict Outcomes

Abstract

Purpose:

Surgical performance is critical for clinical outcomes. We present a novel machine learning (ML) method of processing automated performance metrics (APMs) to evaluate surgical performance and predict clinical outcomes after robot-assisted radical prostatectomy (RARP).

Materials and Methods:

We trained three ML algorithms utilizing APMs directly from robot system data (training material) and hospital length of stay (LOS; training label) (≤2 days and >2 days) from 78 RARP cases, and selected the algorithm with the best performance. The selected algorithm categorized the cases as “Predicted as expected LOS (pExp-LOS)” and “Predicted as extended LOS (pExt-LOS).” We compared postoperative outcomes of the two groups (Kruskal–Wallis/Fisher's exact tests). The algorithm then predicted individual clinical outcomes, which we compared with actual outcomes (Spearman's correlation/Fisher's exact tests). Finally, we identified five most relevant APMs adopted by the algorithm during predicting.

Results:

The “Random Forest-50” (RF-50) algorithm had the best performance, reaching 87.2% accuracy in predicting LOS (73 cases as “pExp-LOS” and 5 cases as “pExt-LOS”). The “pExp-LOS” cases outperformed the “pExt-LOS” cases in surgery time (3.7 hours vs 4.6 hours, p = 0.007), LOS (2 days vs 4 days, p = 0.02), and Foley duration (9 days vs 14 days, p = 0.02). Patient outcomes predicted by the algorithm had significant association with the “ground truth” in surgery time (p < 0.001, r = 0.73), LOS (p = 0.05, r = 0.52), and Foley duration (p < 0.001, r = 0.45). The five most relevant APMs, adopted by the RF-50 algorithm in predicting, were largely related to camera manipulation.

Conclusion:

To our knowledge, ours is the first study to show that APMs and ML algorithms may help assess surgical RARP performance and predict clinical outcomes. With further accrual of clinical data (oncologic and functional data), this process will become increasingly relevant and valuable in surgical assessment and training.

Introduction

Surgical skill assessment has increasingly become a research focus of surgical education and performance improvement. Surgical performance directly impacts patient outcome.^1,2 Good surgical performance is associated with optimal clinical outcome.¹ Therefore, objective and effective surgical performance evaluation may not only determine a surgeon's eligibility and proficiency to perform surgery but may also determine the clinical outcome of patients undergoing surgery.

Efforts have been devoted to achieve effective yet practical surgical performance assessment. The number of cases previously performed by a surgeon (caseload) is a commonly utilized marker of surgeon experience. Learning curve studies of radical prostatectomy show that patient outcome improves with greater caseload.³ However, evaluating surgical performance by caseload alone may be inaccurate or inconsistent as it often is self-reported by the surgeon.³ To overcome this limitation, several surgical assessment tools have been developed and validated to measure surgical performance.^4
–6 With the widely validated Global Evaluative Assessment of Robotic Surgery (GEARS), evaluation of the vesicourethral anastomosis in the robot-assisted radical prostatectomy (RARP) demonstrated association with select patient outcomes.⁷ A recent study showed that GEARS scores from select steps of RARP were predictive of early continence after surgery.⁸ Prostatectomy Assessment and Competency Evaluation⁵ and Robotic Anastomosis Competency Evaluation⁶ are procedure-specific and step-specific evaluative tools and provide deconstructed task evaluation with cognitive surgical skill feedback. Although designed with standardized, objective evaluation criteria, these tools nonetheless require manual review of surgical videos, introducing inherent variance by expert or crowd-sourced evaluators.⁹

For the last two decades, artificial intelligence (AI) has been broadly applied in healthcare to process big data to deliver more efficient care.^10,11 Machine learning (ML), a form of AI, automates analytical algorithm building. Using algorithms that iteratively learn from data, ML allows computers to find hidden insights and detect underlying patterns of a given data set without explicit instruction. Thus, this approach avoids preassumptions regarding model types and variable interactions and may offer additional knowledge that has eluded detection by standard statistical methods.¹¹

With a novel data-recording device, the “dVLogger” (Intuitive Surgical), automated performance metrics (APMs) (instrument motion tracking metrics and system events recorded in Cartesian coordinates) and synchronized surgical footage can be captured directly from the da Vinci robotic system in real time during the actual live surgical procedure. In our original pilot study, we identified and validated certain APMs during RARP that can distinguish surgeon expertise.¹² In the current study, we evaluate the utility of ML algorithms to evaluate surgical performance during whole procedure RARP and predict patient outcome. We also evaluate the relative importance of individual APM in learning and predicting outcomes.

Materials and Methods

APM collection

Under institutional review board-approved protocol for collection and analysis of surgical performance and clinical patient data, patients with localized prostatic adenocarcinoma who underwent primary RARP from August 2016 to March 2017 at our institution were included in this study. To control for surgical complexity, patients with prior treatment for prostate cancer (i.e., radiotherapy and cryotherapy) were excluded from the study. Also, any cases with lysis of adhesion from prior abdominal or pelvic surgeries were excluded from analyses. Synchronized surgical endoscopic video footage and 25 APMs from the studied cases were recorded by the “dVLogger” (Intuitive Surgical) in a process previously described.¹² The “dVLogger” captures synchronized video as endoscope view at 30 frames per second. The APMs, derived from robot system data (kinematic and events), are recorded at 50 Hz. Kinematic data included characteristics of movement such as instrument travel time, path length, and velocity. System events included frequency of master controller clutch, camera movements, third arm swap, and energy usage (Table 1).

Table 1.

The 25 Automated Performance Metrics Captured by the “dVLogger” During Robot-Assisted Radical Prostatectomy and Utilized as “Input” to Train the Machine Learning Algorithms

APM summary
Overall kinematic metrics	Camera metrics
Total completion time (seconds)	Position adjustments (No.)
Total idle time (seconds)	Adjustment frequency (No./s)
Path length of all instruments (meters)	Path length (meters)
Path length ratio of dominant and nondominant instrument	Moving time (seconds)
Dominant instrument metrics	Mean velocity (m/s)
Path length (meters)	Idle time (seconds)
Moving time (seconds)	Other event metrics
Mean velocity (m/s)	Energy usages (No.)
Idle time (seconds)	Energy usage frequency (No./s)
Nondominant instrument metrics	Third arm swaps (No.)
Path length (meters)	Third arm swap frequency (No./s)
Moving time (seconds)	Master clutch usages (No.)
Mean velocity (m/s)	Master clutch usage frequency (No./s)
Idle time (seconds)	Head out of console (No.)

APM = automated performance metric; m/s = meter per second; No./s = numbers/second.

Patient outcome data collection

Patient demographics, surgical data, such as surgery time, estimated blood loss (EBL), and early postoperative outcomes, such as length of stay (LOS), pelvic drainage volume, drainage tube duration, Foley catheter duration, and readmission, were prospectively collected under the same institutional review board-approved protocol. At our institution, patients undergoing RARP are taken care by the same midlevel care team, following a standardized enhanced recovery after surgery protocol, and unified discharge criteria. Surgical drainage tube is removed if the daily drainage output is less than 200 mL. Patients are followed up in the clinic 1 week after the surgery where the Foley catheter is removed. If there is anastomosis leakage, Foley catheter will be left in place for a longer time. If no anastomosis leakage is detected, Foley catheter will be removed in the clinic.

ML algorithm selection

We trained three distinct ML algorithms: “Random Forest-50” (RF-50), Support Vector Machine-Radial Basis Function (SVM-RBF), and L2 Regularization (LR-L2), using Python^® with scikit-learn 0.19.0 package (http://scikit-learn.org/stable). We then selected the algorithm with the highest prediction accuracy and proceeded with further analysis.

ML case categorization

All 78 RARPs were first categorized into two groups based on postoperative LOS. The routine LOS of RARP at our institution is 1 to 2 days.¹³ Thus, in this study, we defined cases with LOS ≤2 days as “Expected LOS (Exp-LOS),” and those with LOS >2 days were determined as “Extended LOS (Ext-LOS).” We chose LOS as the label in this study because in our initial analysis, this perioperative data point was a sensitive and stable differentiator between cases of experienced and less experienced attending surgeons. We used APMs as “training input” and LOS as the “label” to train ML algorithms (Fig. 1). We also trained the algorithms with additional patient demographic data (age, body mass index [BMI], American Society of Anesthesiologists [ASA] scores) to determine any improvement in the prediction accuracy.

FIG. 1.

The process of ML model training and prediction.

We used “Stratified k-fold (k = 10)” cross-validation to predict all cases and estimate the model's performance. For example, the “training input” (APMs, with or without patient demographics) and “label” (LOS) of 70 cases were utilized to train the ML algorithms. We then fed the trained algorithms with the “training input” from the remaining eight separate cases to evaluate and predict surgical performance of those cases. This was the first round of cross-validation. For each around of validation, by using “Stratification,” we made sure that the predicting set contains approximately the same percentage of each label as the training set. During the next round, another combination of 70 cases was used to train, and the outcome of another 8 cases was predicted. After 10 rounds of training and prediction (the last two rounds each contained 71 training cases and 7 predicting cases, to make sure that each of the 78 cases had only been predicted once), all surgeries were classified by the algorithms as “Predicted as expected LOS (pExp-LOS)” and “Predicted as extended LOS (pExt-LOS)” groups based on the APMs of each case (Fig. 1). We then compared the clinical outcomes (continuous variables) among the two predicted groups using the Kruskal–Wallis test. Readmission was compared using Fisher's exact test.

ML model predicting outcomes

We further utilized the same “training input” and distinct postoperative outcomes as “labels” to separately train the selected ML algorithm. For continuous outcome variables (surgery time, EBL, LOS, pelvic drainage volume, drainage tube duration, Foley duration), we trained the algorithm to perform regression analysis. For categorical variable (readmission), we trained the algorithm to perform classification. These distinctly trained algorithms (specific to each clinical outcome variable) were then applied to directly predict the corresponding clinical outcome of each case. We then determined the accuracy of ML prediction by evaluating the association between the predicted results and the actual (“ground truth”) patient outcomes using Spearman's correlation test (continuous variables) and Fisher's exact test (categorical variables).

Metric ranking

A total of 25 APMs were utilized as “input” to train the ML algorithms (Table 1). After the algorithms were trained to categorize cases and predict patient outcome, the importance of each metric (instrument motion tracking [kinematic] metrics or system events [frequency] metrics) was calculated using the algorithms' learned parameters. For the RF-50 algorithm, the importance score for each metric was calculated as the normalized total reduction of Gini impurity associated with splitting(s) on that metric in each tree, averaged over all the trees. For the SVM-RBF algorithm, the importance scores cannot be straightforwardly calculated due to its inherent complexity. For the LR-L2 algorithm, the importance score for each metric was taken from the multiplicative weight associated with that metric. A higher importance score for a metric indicated a greater predictive performance of that metric in the algorithm. We then ranked these APMs and determined the top five metrics most predictive of patient outcome.

Results

Algorithm selection

Seventy-eight RARP cases by nine faculty surgeons (median 500 console cases, range 100–2000) were included in this study. Of the 78 studied RARP cases by nine faculty surgeons, 67 cases had LOS 1 to 2 days and 11 cases had Ext-LOS (>2 days). From this same cohort of patients, the RF-50 algorithm predicted 73 cases as “pExp-LOS” and 5 as “pExt-LOS.” The predictive accuracy was 87.2%. The SVM-RBF algorithm predicted 66 cases as “pExp-LOS” and 12 as “pExt-LOS,” demonstrating an accuracy of 83.3%. The LR-L2 algorithm predicted 71 cases as “pExp-LOS” and 7 as “pExt-LOS,” demonstrating the accuracy of 82.1%. We chose RF-50 for further analysis. If trained with APMs and patient demographics, the predictive accuracy of the RF-50 algorithm improved to 88.5%. To focus on how surgeon performance influences patient outcome, we performed all subsequent analyses using only APMs as “training material.”

Case categorization

No significant difference was identified in patient age, BMI, ASA scores, prostate-specific antigen, Gleason score, prior transurethral resection of prostate, or presence of median lobe between the two groups predicted and categorized by RF-50 algorithm (p > 0.05) (Table 2). Surgeries in the “pExp-LOS” group outperformed the “pExt-LOS” group in surgery time (3.7 hours vs 4.6 hours, p = 0.007), LOS (2 days vs 4 days, p = 0.02), and Foley catheter duration (9 days vs 14 days, p = 0.02).

Table 2.

Patient Demographics and Postoperative Outcomes of the Two Groups Categorized by the Random Forest-50 Algorithm Utilizing Automated Performance Metrics

	Categorization of predicted performance by LOS
	“pExp-LOS” (N = 73)	“pExt-LOS” (N = 5)	p
Patient demographics, median (IQR)
Age, years	65 (61–72)	64 (59–69)	0.92
BMI, kg/m²	28.0 (26.1–30.4)	30.3 (26.9–32.9)	0.49
ASA score	3 (2–3)	3 (2–3)	0.85
PSA, ng/mL	8.7 (6.4–12.5)	10.3 (5.1–19)	0.26
Prostate volume, mL	45 (36–62)	43 (30–61)	0.46
Gleason score	7 (7–7)	7 (6–8)	0.36
Prior TURP, n (%)	1 (1.4)	0	0.94
Median lobe, n (%)	7 (9.6)	0	0.49
Operation data, median (IQR)
EBL, mL	100 (100–150)	100 (88–200)	0.72
Surgery time, hours	3.7 (3.4–4.3)	4.6 (3.5–7)	0.007
Surgical outcomes, median (IQR)
Positive surgical margin, n (%)	20 (27.4)	2 (40)	0.62
LOS, day	2 (1–2)	4 (2–6)	0.02
Drainage volume (POD1), mL	319 (191–379)	203 (155–401)	0.98
Drainage duration, day	4 (1–5)	6 (2–18)	0.06
Foley duration, day	9 (7–9)	14 (9–20)	0.02
Readmission, n (%)	2 (2.7)	0 (0)	0.88

In “pExp-LOS” group, patients were readmitted for small bowel obstruction (one patient) and vesicourethral anastomosis leakage (one patient).

ASA = American Society of Anesthesiologists; BMI = body mass index; EBL = estimated blood loss; IQR = interquartile range; LOS = length of stay; pExp-LOS = predicted as expected length of stay; pExt-LOS = predicted as extended length of stay; POD1 = postoperative day 1; PSA = prostate-specific antigen; TURP = transurethral resection of prostate.

Bold type means p ≤ 0.05.

Outcome prediction

Patient outcomes predicted by the RF-50 algorithm had significant association with actual “ground truth” value in surgery time (r = 0.73, p < 0.001), LOS (r = 0.52, p = 0.05), and Foley catheter duration (r = 0.45, p < 0.001) (Table 3).

Table 3.

Association Between the Actual “Ground Truth” Patient Outcomes and the Random Forest-50 Algorithm-Predicted Results

	Ground truth outcome	RF-50 predicted outcome	p	r
Operation data, median (IQR)
EBL, mL	100 (100–150)	125 (108–157)	0.35
Surgery time, hours	3.9 (3.4–4.4)	3.9 (3.5–4.3)	<0.001	0.73
Surgical outcomes, median (IQR)
Positive surgical margin, n (%)	28 (36)	7 (9)	0.09
LOS, day	2 (1–2)	1.7 (1.5–2.1)	0.05	0.52
Drainage volume (POD1), mL	296 (191–379)	211 (170–276)	0.42
Drainage duration, day	2 (1–7)	3 (2–4)	0.81
Foley duration, day	8 (7–10)	9 (8–10)	<0.001	0.45
Readmission, n (%)	2 (2.6)	0 (0)	0.69

RF-50 = Random Forest-50.

Bold type means p ≤ 0.05.

Metric ranking

We calculated the APMs' “importance score” from the RF-50 algorithm, and identified the top five metrics adopted by RF-50 algorithm in predicting surgery time, LOS, and Foley duration (Table 4). Metrics related to camera manipulation were top predictors for each of our selected perioperative outcomes.

Table 4.

The top Five Automated Performance Metrics Adopted by Each Model During Learning and Prediction of Surgery Time, Length of Hospital Stay, and Foley Duration

	Surgery time		Length of hospital stay		Foley duration
Rank	Metrics	Weight (%)	Metrics	Weight (%)	Metrics	Weight (%)
1	Camera idle time (seconds)	15	Camera adjustment frequency (No./s)	15	Camera position adjustment (No.)	14
2	Dominant instrument moving time (seconds)	11	Path length ratio of dominant and nondominant instruments	9	Third arm swap frequency (No./s)	13
3	Nondominant instrument moving time (seconds)	11	Camera position adjustment (No.)	8	Nondominant instrument mean velocity (m/s)	10
4	Camera moving time (seconds)	7	Energy usage (No.)	7	Dominant instrument mean velocity (m/s)	10
5	Total completion time (seconds)	7	Third arm swaps (No.)	6	Camera adjustment frequency (No./s)	9

m/s = meter/second; No./s = numbers/second.

Discussion

In this study, we trained an RF-50 ML algorithm to evaluate APMs during full-length RARP and to predict select postoperative outcomes. ML algorithms, while now commonplace outside of medicine (i.e., weather forecasting, DNA sequence classification), have not previously been harnessed to process surgeon APMs taken directly from da Vinci robot systems data.

Common wisdom indicates that one can predict surgical performance by knowing that particular surgeon's prior case volumes, that is, a more experienced surgeon is likely to perform better than someone less experienced. Assessment of surgical footage by peer evaluators also grades a surgeon's expertise. Yet, by having the ability to perform detailed analysis of every single movement of the robot across an entire procedure, we may be able to have a fuller picture of surgical performance and may be able to better predict patient outcomes. In the present study, APMs processed by an ML model showed the ability to effectively categorize surgical performance by LOS and identify significant differences with other patient outcomes by this categorization. Furthermore, select outcome data (surgery time, LOS, Foley catheter duration) predicted by the ML algorithms were also significantly associated with the ground truth. This pilot study showed the significant potential of APMs and ML in improving surgery evaluation, augmenting surgeon training, and predicting patient safety and wellness.

The “training inputs” used in this study were the APMs, captured directly from the robotic system by the “dVLogger” in real time during the actual performance of RARP. The “label” was the hospital LOS. Although there were other patient outcome variables that could evaluate surgical performance, we chose LOS as the “label” in this initial pilot study due to its ability to differentiate cases by surgeon experience (data not presented). All RARPs performed at our institution are under the same postoperative pathway, and the discharge-to-home criteria for each patient are nearly uniform. Patients are discharged after meeting a constellation of criteria.¹³ Thus, LOS is a comprehensive and standardized short-term postoperative outcome measurement. Using LOS as a “label” of clinical outcome, the algorithm was able to evaluate and classify surgeries that were not only significantly different in LOS but also in surgery time and Foley catheter duration. We believe that after effective training, the ML algorithm learns through the APMs how to differentiate surgical performance with several optimal patient outcomes.

ML data processing is different from the conventional statistical analysis. For the conventional statistical approach, researchers need to choose a predesigned model that is most suitable for the data. One major limitation of the conventional statistical model is that only theoretically relevant variables based on previous studies and experience, or significant variables identified by the univariate analysis, are to be tested. The number of variables to put in the model is limited, for irrelevant variables will compromise the power of the model. This greatly constricts the ability to explore the influencing factors.

In contrast, ML is not built on a prestructured model; rather, herein, the data create the model by detecting the underlying patterns. Naturally, the more variables (input) used to train the algorithm, the more accurate the algorithm becomes. Also, this approach avoids preassumptions regarding model types and variable interactions. For the above reason, ML may unveil hidden knowledge that may remain undetected by traditional statistical methods. Furthermore, by using dVLogger data, subtle differences in robotic manipulations that are difficult to detect by human review can be recorded and examined. When looking across the entire procedure, these accumulated technical differences may dramatically affect patient outcome. Increasing evidence has suggested that ML can be more accurate than conventional logistic regression across a wide variety of subject areas.^14,15 Thus, when the input data volume is high, it is important to consider techniques beyond standard regression to optimize accuracy.¹⁴

In our previously published robotic APM validation study, experts showed more efficient camera manipulation in select steps of RARP.¹² Another study on robotic virtual reality simulation exercises also suggested that, expert surgeons had significantly less total camera moving time but higher frequency of camera movement than new robotic surgeons.¹⁶ In this study, the metric importance ranking also showed that camera manipulation-related metrics were important during ML learning and prediction. Thus, we believe that skillful (brief, efficient, and frequent) camera manipulation is emerging as an important indicator of robotic surgical expertise. In reality, this discovery perhaps represents a sensitive measure of surgeon performance as opposed to a specific technical skill to be trained. Further work in this area will associate APMs to surgeon intent and behavior in relation to the surgical context (i.e., fluctuation of metrics during lower vs higher complexity tasks).

There are limitations to the present study. This is a preliminary study of applying APMs and ML to surgical evaluation and patient outcome prediction. We are accruing oncologic and functional data, which will be presented in future work. We have studied only one procedure, the RARP. However, the lessons learned in this effort may be applied to other urologic and nonurologic procedures. Given that the study took place at a major teaching institution, several cases had resident or fellow participation, although under careful faculty surgeon supervision. It is therefore noteworthy that the aggregate movements and events that occurred in each full-length RARP may have stemmed from more than one operator. In the whole cohort analysis, most statistically significant associations between ground truth and predicted clinical outcomes were not strong; it is possible that with a larger cohort of cases training the algorithm in the near future, more accurate predictions can be made and thus result in stronger associations with actual outcomes. Finally, while the performance metric analyzed reflect movement and efficiency, it does not yet describe a skills domain that is deficient or provide step-specific feedback. Further work that associates APMs to surgeon cognition and intent may allow for meaningful surgeon feedback.

Conclusion

The marriage of APMs derived from the surgical robot and ML algorithms may help objectively assess surgical performance on a large scale and predict clinical outcomes. With further accrual of clinical data (oncologic and functional data), this process will become more relevant and valuable in surgical assessment and training.

Footnotes

Acknowledgment

The authors thank Omid Mohareri for dVLogger support.

Author Disclosure Statement

A.J.H. is a consultant for Ethicon, Inc. A.J. is an employee of Intuitive Surgical.

Abbreviations Used

References

Birkmeyer

, Finks

, O'Reilly

, et al. Surgical skill and complication rates after bariatric surgery. N Engl J Med, 2013; 369:1434–1442.

Reznick

, MacRae

. Teaching surgical skills—Changes in the wind. N Engl J Med, 2006; 355:2664–2669.

Abboudi

, Khan

, Guru

, et al. Learning curves for urological procedures: A systematic review. BJU Int, 2014; 114:617–629.

Goh

, Goldfarb

, Sander

, et al. Global evaluative assessment of robotic skills: Validation of a clinical assessment tool to measure robotic surgical skills. J Urol, 2011; 187:247–252.

Hussein

, Ghani

, Peabody

, et al. Development and validation of an objective scoring tool for robot-assisted radical prostatectomy: Prostatectomy Assessment and Competency Evaluation. J Urol, 2017; 197:1237–1244.

Raza

, Field

, Jay

, et al. Surgical competency for urethrovesical anastomosis during robot-assisted radical prostatectomy: Development and validation of the robotic anastomosis competency evaluation. Urology, 2015; 85:27–32.

Ghani

, Arbor

, Comstock

, et al. Technical skill assessment of surgeons performing robotic-assisted radical prostatectomy: Relationship between crowdsourced review and patient outcomes. J Urol, 2017; 197:No. 4S, Suppl: PNFBA-02.

Goldenberg

, Goldenberg

, Grantcharov

. Surgeon performance predicts early continence after robot-assisted radical prostatectomy. J Endourol, 2017; 31:858–863.

Ghani

, Miller

, Linsell

, et al. Measuring to improve: Peer and crowd-sourced assessments of technical skill with robot-assisted radical prostatectomy. Eur Urol, 2016; 69:547–550.

10.

Murdoch

, Detsky

. The inevitable application of big data to health care. JAMA, 2013; 309:1351–1352.

11.

Shouval

, Hadanny

, Shlomo

, et al. Machine learning for prediction of 30-day mortality after ST elevation myocardial infraction: An Acute Coronary Syndrome Israeli Survey data mining study. Int J Cardiol, 2017; 246:7–13.

12.

Hung

, Chen

, Jarc

, Hatcher

, Djaladat

, Gill

. Development and validation of objective performance metrics for robot-assisted radical prostatectomy—A pilot study. J Urol, 2018; 199:296–304.

13.

Berger

, Chopra

, Desai

, et al. Outpatient robotic radical prostatectomy: Matched-pair comparison with inpatient surgery. J Endourol, 2016; 30 Suppl 1:S52–S56.

14.

Fernandez-Delgado

, Cernadas

, Barro

, et al. Do we need hundreds of classifiers to solve real world classification problems?. J Mach Learn Res, 2014; 15:3133–3181.

15.

Churpek

, Yuen

, Winslow

, et al. Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards. Crit Care Med, 2016; 44:368–374.

16.

Jarc

, Curet

. Viewpoint matters: Objective performance metrics for surgeon endoscope control during robot-assisted surgery. Surg Endosc, 2017; 31:1192–1202.