Reliability of Chinese Medicine Diagnostic Variables in the Examination of Patients with Osteoarthritis of the Knee

Practitioners and participants

Two (2) CM practitioners participated in this study. Both are registered CM practitioners in the state of Victoria, Australia, with bachelor degree–level training in CM and at least 10 years clinical experience. One (1) practitioner was trained and was practicing in China, and the other practitioner had completed training and is in clinical practice in Australia.

Forty (40) eligible patients from the CHM study participated in the inter-rater reliability study. The inclusion criteria for the main clinical study included unilateral or bilateral OA of the knee and fulfillment of the criteria provided by the American College of Rheumatology (ACR) 1995, ¹⁵ mainly based on knee pain and presence of radiographic osteophytes. Radiographic evidence of OA was based on the Kellgren-Lawrence radiographic system, ¹⁶ either grade II or grade III severity primary tibio-femoral OA as a condition of inclusion. The exclusion criteria of the study included secondary OA or rheumatoid inflammatory or any other type of arthritis, accompanying OA of hip of sufficient severity to interfere with the functional assessment of the knee, having received intra-articular treatment of the involved joint or joint lavage in the previous 6 months (e.g., corticosteroids or hyaluronic acid) or knee surgery during the previous 3 months, and any significant systemic illnesses or medical conditions that could lead to difficulty with complying with the protocol.

There was no CM assessment made for any of the participants prior to the study.

Materials

A CM assessment form was utilized to record systematically CM diagnostic information during a CM examination. This form was developed based on standard questions and observations using the four diagnostic methods: Inquiry; Inspection; Auscultation; and Palpation, as set out in the State Administration of Traditional Chinese Medicine's Advanced Textbook on Traditional Chinese Medicine and Pharmacology, vol. 1, ¹⁷ a standard textbook used in CM curricula in China. The form was originally developed by consensus of a group of Australian researchers, and applied in a study of the reliability of CM diagnosis in hypercholesterolemic Australians, and, thus, had face validity and content validity. ¹³ This assessment form was modified to include specific questions related to the knee joint.

The clinical information in the form recorded clinical data in a manner that allowed it to be readily analyzed. The first section (Inquiry, case history) consisted of a series of questions related to bodily functioning, with extra questions about the knee joint added for this study. The other sections recorded information relating to Inspection, Auscultation, and Palpation. For most questions, information was recorded as categorical variables, and practitioners were required to choose one answer from a limited range of answers.

With respect to pulse diagnosis, the basic characteristics recorded in this study were pulse speed, force, and depth. Pulse speed was measured as the number of beats of beats per breath of the patient. A normal speed is considered to be 4–5 beats per breath. A slow pulse is defined as ≤3 beats per breath, and a fast pulse is described as ≥6 beats per breath. The CM assessment form used in this study is considered to have face validity, construct validity, and criterion validity in terms of the theoretical structure and the real practice of CM diagnosis. Diagnostic endpoints utilised in the CM assessment form are set out in Table 1.

Study procedures

The reliability study was conducted at the first study visit of the CHM clinical trial. Both practitioners conducted the CM assessment separately on the same day, with a maximum of 30 minutes between the two CM assessments (each session lasted approximately 30 minutes). There was no training or discussion between the 2 practitioners prior to or after the CM assessment and the CM assessment forms were kept separate until data analysis. Data entry and analysis were conducted by the first author of this article.

Statistical analysis

The level of agreement (%) for the key diagnostic endpoints was calculated, along with the (Cohen's) κ coefficient. If data were missing from at least one tabulation cell (the responses of an endpoint were concentrated on a particular choice) the κ coefficient could not be calculated. The interpretation of κ values used in this study ¹⁸ is shown in Table 2.

Inquiry variables

There were 84 investigated endpoints in the Inquiry section. Insufficient data were collected for 9 variables, and data for a further 23 variables that could not be analyzed using the κ coefficient because there were missing data in at least one of the tabulation cells (mainly because of the consistent recording of one particular response for one of the practitioners). Inter-rater reliability was assessed for 52 key Inquiry variables and ranged from “poor” to “almost perfect.” Results are shown in Table 3.

Inspection variables

A total of 24 Inspection variables were assessed. Nine variables could not be analyzed using the κ coefficient because 1 rater consistently chose the one response. Agreement for the 15 Inspection diagnostic variables ranged from “poor” to “moderate.” Results are shown Table 4.

Auscultation variables

Agreement between 2 practitioners for the two auscultation variables was 82.5% for voice strength and 97.5% for characteristics of breath sounds. Given that one practitioner consistently chose the same responses for each participant for these variables, the κ coefficient could not be applied. Results are shown in Table 4.

Palpation variables

A total of eight Palpation variables were assessed. One variable (moisture of hands) could not be assessed using the κ coefficient because one rater consistently chose one response. Agreement ranged from “poor” to “substantial” for the eight palpation endpoints investigated. “Fair” agreement was found for the variable of location of right pulse. “Slight” agreement was found for four variables of pulse diagnosis (location and force of left pulse, force and speed of right pulse) A “poor” level of agreement was found for speed of the left pulse (level of agreement 76.9%). Results are shown in Table 4.

Reproducibility of Inquiry variables

Given that most of the information elicited from a case history comprise, by nature, subjective symptoms, it is common to find inconsistency in diagnostic assessments among different doctors—even in conventional medicine. ^19,20 Zhang and colleagues argued that Inquiry as a diagnostic method is not a valid instrument, because of the low level agreement of diagnosis found among 3 practitioners working with patients who had rheumatoid arthritis (RA). ^4,10 The use of a CM Assessment form used in this study attempted to standardize the Inquiry process.

In the current study, a relatively good level of agreement for most Inquiry variables was found. For almost 50% of the Inquiry endpoints there was a “substantial” to “almost perfect” level of agreement.

The limitations of the κ statistics need to be considered when interpreting some items. A “fair” level of agreement was found for some symptoms, such as characteristics of abdominal pain; there was only a small number of valid cases for the κ analysis and therefore there were insufficient data from which to judge reliability. There were also responses to 23 questions that could not be analyzed. Reasons included missing data for κ tabulation cells, or small numbers of valid cases. In the case of some variables when the κ coefficient interpretation indicated only “moderate” to “fair” levels of agreement, the actual percentage agreement may have been quite high. Caution should be exercised in the interpretation of the level of agreement using the κ coefficient when there is a skewing of a high number of responses for one particular response option. This is one of the limitations of the κ coefficient. ²¹

Given the nature of the response options (fixed, categorical) and the use of a fixed form of words, it is not surprising that there was a reasonably high level of agreement on Inquiry variables. This contrasts with clinical practice in which asking an open-ended question may elicit more varied responses. Other research found the use of objective questionnaires instead of conventional case notes for Inquiry can increase agreement on CM diagnosis among practitioners for RA. ²²

Reproducibility of Inspection variables

Inspection in CM is based on the concept of correspondence between the Internal Organs and their external manifestations. A “slight” to “fair” level of agreement was found for the majority of the eight Inspection variables assessed, indicating a relatively low level of agreement overall for Inspection variables in general. The results are similar to those found in other studies that found variable levels of agreement for Inspection endpoints. ^12,13 For example, although not directly comparable to the current study, because their study used three practitioners, O'Brien and colleagues' found that the level of agreement between 3 practitioners for color and size of tongue body was “slight,” and for thickness of tongue coating it was “fair” (level of agreement was higher when level of agreement was measured for at least 2 practitioners). ¹³ The low level of agreement is indicative of the subjective nature of Inspection, which can require quite subtle observations. Clear definitions of these characteristics in CM are needed. Work is underway in China to develop objective methods for tongue diagnosis, including color detection instruments and computerized image analysis systems based on different color models. ^{23 –26} These models are an attempt to quantify characteristics of the tongue body and tongue coating color. Although studies have found that tongue color identification systems can reflect the characteristics of tongue color and record similar judgments to those of CM practitioners, there are also several factors that could contribute to errors in measurement, such as the shape of tongue, the structure of its curved surface, and the sampling area. ^{24 –26} In addition, studies in Western medicine have also indicated considerable variation among observers in physical examinations. ^{27 –31}

Reproducibility of Auscultation variables

The level of agreement between 2 practitioners on the strength of voice and character of breathing sounds was reasonably high in the current study; they were 82.5% and 97.5%, respectively (although κ could not be calculated). This was a similar result to that found by O'Brien and colleagues who found a “moderate” to “almost perfect” level of agreement. ¹³ However in CM clinical practice, not much significance is attached to Auscultation because the hearing ability of practitioners varies from person to person. Some research has attempted to quantify and interpret the voice using a spectrogram. ^32,33 However, Auscultation in CM also includes many other sounds such as eructation, groaning, and crying which have not been studied with respect to reliability. More studies are needed to establish the reproducibility of Auscultation variables.

Reproducibility of Palpation variables

Palpation in CM includes palpation of the chest, abdomen, other body parts as indicated, meridians, and acupoints, but the most important aspect in clinical practice is pulse diagnosis. The results of this study indicated “poor” agreement for left pulse speed, “slight” agreement for four aspects (left pulse location and force, and right pulse speed and force), and “fair” for one aspect (right pulse location). This is not dissimilar to O'Brien and colleagues' study that found the level of agreement among 3 practitioners was “slight” for location and “fair” for force for the right pulse. ¹³ The results support the notion that pulse diagnosis is the most difficult part of the art of CM diagnosis, which requires extensive clinical experience to master. ³

Despite its crucial role in the diagnostic process in CM, it is difficult to objectify and standardize pulse diagnosis. Some attempts have been made in China to try to objectify pulse diagnosis through development of pulse-measuring apparatuses. ^34,35 However, it has been argued that those apparatusus are simply pulse-tracing devices based on anatomy and physiology as described in conventional medicine rather than in CM theory. ³⁶ Current research indicates that unambiguous definitions of pulse characteristics are critical in pulse research. ^37,38 Standardizing the pressure applied (i.e., measurement of finger strength) in pulse detection is the most difficult part in pulse research. ³⁶ A review of reliability studies of pulse diagnosis ³⁸ showed that level of agreement varies from “low” to “very good” agreement.

Study limitations

There were a number of factors that may have affected the results of this study. First, the use of the CM assessment form may have interrupted or sidetracked the clinical thinking process by requiring the practitioner to ask about symptoms/signs that were not necessarily relevant. This may be likened to breaking a whole picture into too many (diagnostic) pieces. Some of the knee-related questions were more general, and it emerged that they may not have captured the experience of the condition of knee OA sufficiently. For example, some patients reported that they did not have pain or stiffness, but had “discomfort.”

Second, the difference in CM training and experience between the 2 practitioners may have contributed to the variability in observations and diagnosis. Some signs are open to interpretation, which is likely to be influenced by clinical training and experience.

Third, there was no training prior to the study. Other studies have found that prior training improved the level of agreement between practitioners. ²²

Finally, there are limitations of the κ coefficient, which have already been discussed. κ can become unstable or even inappropriate as a statistic. ³⁹

To improve the reliability of CM data collected using the four diagnostic methods, clear definitions for all outcome variables should be established and prior training of the examiners (as some other studies have done) should be incorporated into future study designs.