Abstract
According to Bolsin and Colson (2), CUSUM analysis is a technique that has been transferred from use in industrial and managerial processes to monitoring performance in the practice of medicine. They point out that the CUSUM technique is one of a series of statistical tests developed during World War II as quality control assessments for munitions production lines. The first detailed description of the CUSUM technique appeared in 1954 (2). The need for sequential analyses arose from extensions of statistical techniques that included the well-recognized shortcomings of repeated statistical tests and the difficulties associated with tests in which the sample number was unknown and had the potential to expand exponentially. Bolsin and Colson (2) point out that, to meet the requirement for sequential testing, it was necessary to develop a mathematical model that allowed the observer to decide whether a production process was “in control.” Thus, part of the application of the CUSUM technique is to identify the need for and definition of a “stopping rule” to suspend a process that is out of control and to suspend an unacceptable performance.
In short, this is a well-described statistical test that has utility in assessing physician performance. As a part of this process, in 2003 Grigg et al. (3) described a chart that could visually depict the performance of a surgeon. This chart graphically shows the process over time, which is a crucial part of the analysis that can define the “out of control” state. As shown by the charts in the Connolly and Watters study (1), failed attempts at the procedure are indicated by upward deviations of the plot and competence of the operator is demonstrated by a downward direction, which at first glance might seem to be heading in the wrong direction, but it is a chart that is easily interpreted over time.
The CUSUM analysis has been shown to be successful in a variety of settings. For example, Lim et al. (4) found that CUSUM charting was useful in the assessment of a doctor's performance of endoscopic retrograde pancreatography, renal and breast biopsies, and instrument delivery. For the competence of a trainee in new procedures, it has been shown to be helpful (2), and, except for the variation in the expertise of practicing surgeons, the CUSUM technique functions well in the assessment of practicing physicians and surgeons (5). Still, outcome indicators are a key part of the process, describing the limits of unacceptable outcomes, which may be a difficult problem. In the Connolly and Watters study (1), a literature search found the key performance indicators for thyroid surgery (hematoma, hypocalcemia, and vocal cord palsy), and CUSUM charting was only performed after agreeing by consensus to the boundaries of acceptable and unacceptable performance, including complications and surgical outcomes and key performance indicators, which is a major part of the process.
After the records of 216 patients who underwent thyroidectomy were assessed and charted, the incidence of temporary and permanent hypocalcemia and vocal cord palsy were 24% and 2.6% and 3.2% and 0.65%, respectively, and 1.39% of patients required evacuation of a hematoma. In all, this was a very good performance by the surgeons being assessed.
What can we learn about this process? First, this is a statistical analysis that is customized for the process under assessment. Second, the surgeons participated in the definitions of performance standards. Third, this is a process that provides surgeons with an opportunity to evaluate their performance and to improve it if necessary. The process is also able to identity outliers that require further training or, in a worst-case scenario, removal of hospital privileges. This seems to be an accurate and fair analysis of a surgeon's performance. From my standpoint the CUSUM evaluation provides a clear assessment of the surgeon's skills providing testing is performed with same high level of expertise shown by Connolly and Watters (1) and the endpoints can be clearly measured. Still, it is difficult to track parameters such as the completeness of thyroidectomy or tumor resection, the extent of cosmetic issues, and the subtle disturbance of tissues that may occur adjacent to the thyroid.
How would this work for an endocrinologist or for an internist who refers patients for thyroidectomy to surgeons in practice? One of my favorite memories is a comment made by a first-class surgeon shortly after we began working together when he seemed surprised to learn that the management of patients with thyroid cancer is a team effort in which each member must have a high level of proficiency to ensure a favorable outcome. This immediately translates into the question of performing an assessment of internists and others who participate in the care of patients undergoing thyroidectomy. Yet whether we can evaluate physicians who do not have a hospital practice may pose major challenges using CUSUM. This may be the Achilles heel of CUSUM.
Connolly and Watters (1) have focused a bright light on the assessment of quality of care for patients undergoing thyroidectomy, especially total thyroidectomy, which is being done with greater frequency now than ever before (6,7). Wherever we practice, we should all note this study and consider its ramifications for the care of our patients.