The better way to determine the validity,reliability,objectivity and accuracy of measuring devices

Abstract

BACKGROUND: In relation to statistical analysis, studies to determine the validity, reliability, objectivity and precision of new measuring devices are usually incomplete, due in part to using only correlation coefficient and ignoring the data dispersion.

OBJECTIVE: The aim of this study was to demonstrate the best way to determine the validity, reliability, objectivity and accuracy of an electro-inclinometer or other measuring devices. Another purpose of this study is to answer the question of whether reliability and objectivity represent accuracy of measuring devices.

METHODS: The validity of an electro-inclinometer was examined by mechanical and geometric methods. The objectivity and reliability of the device was assessed by calculating Cronbach’s alpha for repeated measurements by three raters and by measurements on the same person by mechanical goniometer and the electro-inclinometer. Measurements were performed on “hip flexion with the extended knee” and “shoulder abduction with the extended elbow.” The raters measured every angle three times within an interval of two hours. The three-way ANOVA was used to determine accuracy.

RESULTS: The results of mechanical and geometric analysis showed that validity of the electro-inclinometer was 1.00 and level of error was less than one degree. Objectivity and reliability of electro-inclinometer was 0.999, while objectivity of mechanical goniometer was in the range of 0.802 to 0.966 and the reliability was 0.760 to 0.961. For hip flexion, the difference between raters in joints angle measurement by electro-inclinometer and mechanical goniometer was 1.74 and 16.33 degree (P<0.05), respectively. The differences for shoulder abduction measurement by electro-inclinometer and goniometer were 0.35 and 4.40 degree (P<0.05).

CONCLUSION: Although both the objectivity and reliability are acceptable, the results showed that measurement error was very high in the mechanical goniometer. Therefore, it can be concluded that objectivity and reliability alone cannot determine the accuracy of a device and it is preferable to use other statistical methods to compare and evaluate the accuracy of these two devices.

Keywords

Electro-goniometer goniometry psychometric properties

1 Introduction

Detecting normal angles for each joint is important to correctly diagnose and prescribes movement therapy in physiotherapy, corrective exercises, and exercise rehabilitation programs [1]. Therefore, it is important to apply tools that are the most accurate, valid and reliable and at the same time are accessible, objective, cost effective and easy to use. The mechanical goniometer is one of the conventional tools to measure joint angles. More advanced tools include electric goniometers, electro-inclinometers. Other methods include motion analyzers and primary techniques for measuring joint angles during motion such as filming and photographing. Applying these techniques requires sophisticated tools, high expense, and highly trained and experienced raters [2]. Goniometers are the simplest tools for measuring joint angles [3] with one degree of precision. However, the content validity of goniometers is open to question. Mechanical goniometer limitations include the following:

Setting the mechanical goniometer at the proper point of joints; it can be difficult to find specific points of the body due to skin changes in different perspectives.

Incorrect reading of degrees; for example, a non-perpendicular viewing angle can lead to incorrect reading of degrees

Patient’s awareness of measurement results.

The probable effect on the motivation of the examinee to exert extra or less effort.

Difficulty of instructions for some of the points like “Q angle 1 ” (4).

Both the electrical goniometer and inclinometer can measure angles, but the electro-inclinometer can also measure the slope. Previous studies have shown that electrical goniometers provide more precision in measurement than mechanical goniometers; as they do not need mobile arms and marked points, and the instructions for the use of electrical goniometers are simpler. With this reduced possibility of error, they are perceived to be more accurate than mechanical goniometers. Moreover, the sensors of electrical goniometers are more sensitive, there is no error in the reading of digital outputs, and the outputs can be stored in a memory card. Previous studies were focused on assessing the precision, reliability and validity of electrical goniometers and inclinometers. In most of these studies, data was analyzed according to common statistical methods including standard deviation and interclass correlation, or linear regression [3, 5 -15].

When a researcher wants to introduce a new technique for clinical assessment of a new tool, it is very important to propose justifications to replace the old method with the new one. But the statistical procedures applied in many studies [3, 5 -13] are often inappropriate. In those studies, a correlation coefficient is only used to analyze two sets of data, the results of such analysis can be misleading and the results obtained are not always correct. It is better that agreement levels be assessed in following way:

The first step involves drawing a scatter diagram (draw the line through the points). This approach can help a person assess the amount of agreement between measurements at a glance. Also a diagram drawn based on the mean difference of data can be helpful. The diagram shows the relationship between errors of measurement and the true score of measured variables. As the true data is not known, the mean of two measurements is the most appropriate evaluation method.

The second step is to calculate the correlation coefficient. In this approach, the null hypothesis states that the two measurements are not linearly correlated. Although a high correlation does not assure that the two measures are completely agreement with each other, since the index of correlation shows only strength of association but not agreement. Complete agreement is attained when the points on a scatter diagram lie on a line, or even curved line. In a complete correlation, points lie on a straight line only. The change in scale has no effect on correlation but has a definite affect on agreement. Correlation is related to the range of correct data collected in samples. If the data is widely varied or heterogeneous, the correlation is higher than when it is homogenous and less variant. Researchers usually apply the correlation procedure for two methods and often results indicate that correlation coefficient is high. It is possible that two methods show a high correlation coefficient, but this does not assure that they are reasonably related. The correlation coefficient is not related to the amount of agreement.

The most appropriate method to estimate reliability is repeated measurements on multiple samples. If differences are proportional to the average value, it is possible to apply logarithm form. If the repeated measures by both methods are similar for the individuals, the average values for both methods on every individual can be calculated. Every pair can be used to compare the two methods and employ the analysis for describing the agreement [14].

These facts show that applying the correlation coefficient may not be the most appropriate way to measure the relationship between two groups of data obtained by two different methods. Employing descriptive statistics and interclass correlation methods to show agreement and validity, and repeated measurements to estimate objectivity and reliability of two methods can be more appropriate. Moreover objectivity and reliability cannot show the accuracy of the device.

2 Methodology

We have designed and created an electro-inclinometer. The main purpose of making the device was to create a portable device that is accurate and inexpensive.

2.1 Making process of electro-inclinometer

The initial version of our device had an ADXL202 sensor that enabled data to be directly transmitted to the processor by one wire. Research partners proposed that the device should include a display screen and memory for recording data. In addition, calibration ability and functionality to define samples and reread data was proposed. These changes were performed and measurement accuracy of the device was assessed by the primary mechanical goniometer. The results indicated that the device had an acceptable precision for the range of zero to 50 degrees. But the precision was not acceptable out of this range. To solve the problem and improve device accuracy, we added another sensor perpendicular to previous one.

One of sensors was set to measure angles of 0-45 degrees (and related symmetrical angles) and the other one was set to 45-135 degrees (and related symmetrical angles). Validation of the electro-inclinometer by primary mechanical goniometer indicated that there was a discrepancy between two sensors signals in the range of 40-50 degrees. Therefore, some modifications were performed in both the software and hardware sections of electro-inclinometer and it was retested. But this time, displayed angles were not stable and device was very sensitive to slight hand vibrations. More modifications were performed on the software section of the device. When the electro-inclinometer passed its tests with the primary mechanical goniometer successfully, validation of the device was verified by trigonometric scale and results showed that the device had an acceptable precision in the whole range of angles. Finally, the objectivity, reliability, and validity of the device were assessed to measure joint angles of humans.

Our constructed electro-inclinometer contains three basic components: sensors, microprocessor and display Screen (Fig. 1).

2.2 Sensors

The ADXL202 accelerometer sensor was used in this inclinometer. The ADXL202 is a low-cost, low-power, complete dual-axis iMEMS^® accelerometer with a measurement range of ±2g. The ADXL202 can measure both dynamic acceleration (e.g., vibration) and static acceleration (e.g., gravity). The outputs are Duty Cycle Modulated (DCM) signals whose duty cycles (ratio of pulse width to period) are proportional to the acceleration in each of the 2 sensitive axes. This type of sensor is used in the Wii and smart cell phones.

2.3 Microprocessor

This component is considered the brain of the electro-inclinometer and its principle function is the processing of the input data. The data entering the apparatus is translated into applicable figures.

2.4 Data processing algorithm

Sensor output pulses are read by the microprocessor and the ratio of high to low width pulse is calculated. The first time it is necessary to initialize the pulse width value related to zero acceleration by calibrating the inclinometer. Following this step, the electro-inclinometer is ready to work. Once calibrated, the ratio of pulse width to the total wave length is equal to the cosine of the device to the horizon. After that the angle can be calculated by employing an “arc cos” function on the cosine value. In practical application, the pulse measured and pulse width may be distorted by noises, vibrations or interferences. To avoid such fluctuations in the final result, a mean calculator function has been considered in the algorithm. This derives mean values (1000 times for each angle) from sensor outputs then performs calculations to compute angle degree. In addition, to avoid affects from hand (or body) vibration on calculated angles, the algorithm will recalculate the values if any unexpected change happened in the measured pulse width.

2.5 Creativity applied in electro-inclinometer construction

ADXL sensors and accelerometers lose their precision and effectiveness in angles higher than 50°. In order to solve that problem in this device two sensors perpendicular to each other have been applied to measure the whole range of angles with reasonable accuracy. Due to this arrangement, angles below 45° are measured by a horizontal sensor and angles above 45° are measured by a vertical sensor. The same operation will be performed for related symmetric angles. Thus angles in the range of 315°-45° and 135°-225° can be measured by the horizontal sensor and angles in range of 45°-135° and 225°-315° can be measured by the vertical sensor, all with reasonable precision.

2.6 Validation by the primary mechanical goniometer

For the purpose of validating the performance of electro-inclinometer in measuring physical angles, a primary mechanical goniometer was designed by the research advisor and constructed by the researchers (Fig. 2).

Validation of the device activity was examined by measuring 36 different angles (from 0° to 180° with intervals of 5°) by the primary mechanical goniometer. Descriptive statistical methods were used for the data analysis. Each angle was measured 10 times.

2.7 Validation by geometrical method

In this stage, the mechanical goniometer was used as a compasses and it was placed on the wall. We put a sign on both of the goniometer arms at 20 cm from axis. The electro-inclinometer was placed on the mobile arm of the mechanical goniometer (at 20 cm from axis). Then we opened the goniometer arm until the inclinometer screen showed 5 degrees and the length of chord angle was recorded (Fig. 3). This was repeated from 5° to 140° in intervals of 5°. The observed angles were compared with the degrees shown by the electro-inclinometer through a Pythagorean trigonometric formula 2 .

2.8 Objectivity and reliability

The reliability and objectivity of the device was examined at hip flexion angles with extended knee (Fig. 4), and shoulder abduction with extended elbow (Fig. 5). Three raters performed the measurement three times with 2-hour intervals between the measurements. The measurements were performed once with the electro-inclinometer and once with the mechanical goniometer. A body location for the limb was prepared to prevent subject fatigue and also to minimize errors related to variation within subjects. For this purpose, a body limb support surface was placed on an adjustable tripod camera (Fig. 6).

Each 20 mm section of the center column of the camera was marked at with intervals of 2 cm. The marks were randomly numbered from 1 to 19. Our tripods have two section legs that nest inside each other to make it adjustable. For this reason thigh flexion angle was measured for 15 repetitions over the bottom section and 15 repetitions over the top section. At this stage of the test, the ankle was placed on a body limb support surface on the camera tripod. In the shoulder abduction with extended elbow test, measurements were performed for 10 repetitions over the bottom section and 10 repetitions over the top section. In this test, the wrist was placed on the body limb support surface on camera tripod. In both tests, height was randomly changed by one of the researchers and the testers were unaware of the change in the height of camera tripod from one time to the next. In order to avoid bias effects, every angle initially was measured by a mechanical goniometer and then by the electro-inclinometer. The data form was completed by one of the researchers in a blinded fashion and raters were unaware of each other’s measurements. The raters were trained in one session and all the measurements were performed on one person. Data was analyzed using the interclass correlation coefficient by calculating Cronbach’s alpha.

2.9 Measurement accuracy

Objectivity and reliability cannot provide information about accuracy and differences in measurements between two devices. Confounding factors were intra-class (or between groups). This analysis aimed to determine and compare the maximum measurement error in the mechanical goniometer and the electro-inclinometer. Statistical analysis was applied to the data obtained in the previous step. The data was analyzed using three-way ANOVA with the measurement devices (mechanical goniometer and electro-inclinometer), the raters (three people with different ability levels), and the number of tests (three repetitions with 2-hour intervals between the measurements) as factors.

3 Results

Table 1 shows the mean values of errors in 10 repetitions comparing results obtained using the electro-inclinometer and the primary mechanical goniometer. The correlation coefficient calculated for every 10 repetitive measurements was 1.000. The mean of the standard deviation error was 1.2 ± 0.2 degrees. The results indicated that the electro-inclinometer has a high degree of precision.

Following the completion of the hardware and software preparation, precision of measurement should be examined by an appropriate criterion. The most precise method for measuring an angle is geometric calculation. The mean and standard deviation of the differences between real and calculated angle was 0.86 ± 0.88. In other words, the mean of error was less than 0.9 degree and rarely reached to 2.5 degrees.

Some of the raw data documented in Excel has been shown in Table 2. The correlation scores between the angle observed by the electro-inclinometer and the geometric method was 1.000. The mean of standard deviation error was 0.86 ± 0.88.

Firstly, the results of this study (Table 3) indicate that the reliability coefficient of angles reported by the three raters using the mechanical goniometer to measure the hip flexion was good and relatively high (0.92 to 0.94). Secondly, the electro-inclinometer has increased the reliability coefficient to ideal levels (0.98 to 0.99). In addition, the results showed that for rater 1 and 2 the reliability coefficients for the mechanical goniometer in shoulder abduction were good and relatively high (0.95 to 0.96). However, the reliability coefficient for the third rater was moderate (0.76). But results of this study showed that electro-inclinometer has increased the reliability coefficient to ideal levels for all raters (0.97 to 0.99).

The results of this study (Table 4) indicate that objectivity coefficients (variations among the raters) of the mechanical goniometer in hip flexion and shoulder abduction was relatively high (0.80 to 0.96), but use of the electro-inclinometer increased the objectivity coefficient to ideal levels (0.99).

These results demonstrated that a repeatedmeasurements obtained by the electro-inclinometer are more objective and reliable than mechanical goniometer.

Figure 7 shows the interaction between raters and devices in hip flexion. The results indicate that the interaction was statistically significant (F2, 58 = 107.2, P = 0.001). The maximum difference for using the electro-inclinometer and the mechanical-goniometer was 0.23 and 8.23 degrees, respectively.

The results of this study (Fig. 8) show that the interaction of “rater by repetition by device” in hip flexion was statistically significant (F4, 116 = 6.1, P = 0.001). The maximum difference for the electro-inclinometer and mechanical-goniometer was 1.80 and 11.03 degrees, respectively.

Figure 9 shows the interaction between raters and devices in shoulder abduction. The results indicate that this interaction was statistically significant (F1.2, 24.6 = 4.1, P = 0.044). Maximum difference for using the electro-inclinometer and the mechanical-goniometer was 0.55 and 2.82 degrees, respectively.

Results of this study (Fig. 10) indicate that the interaction of “rater by repetitions by device” in shoulder abduction is not significant (F2, 39.8 = 0.2, P = 0.425). Maximum difference for theelectro-inclinometer and mechanical-goniometer was 1.40 and 5.30 degrees, respectively.

4 Discussion and conclusion

The aim of this study was to provide a better way to determine the validity, reliability, objectivity and accuracy of measuring devices and assess them in an electro-inclinometer. The results of the assessment the electro-inclinometer compared to the primary mechanical goniometer (Table 1) indicate that the mean error of measurement in 47 percent of the cases was less than 0.5 degree, in 27 percent of the cases it was between 0.5 to 0.9 degree and in 26 percent of the cases it was 1 to 1.8 degree. The results also show that the mean standard deviation of error was 1.2 ± 0.2 degrees. It should be pointed out that using mechanical criterion was not the only source of error. Naked eye measurements evaluating degree values on mechanical goniometer and effect of angle of vision on data reading should be considered as main sources of error. In this stage, it cannot be concluded whether the observed error can be attributed to electro-inclinometer or the primary mechanical goniometer or both devices. However, a reasonable supposition is that the source of error could be related to the primary mechanical goniometer.

The common method of assessing the precision of measurement for a device is that several angles are measured at one time by new device and at one time by the older device. In this study we used trigonometric methods. To calculate the desired angle, if the length of sides of a triangle is measured by highly sensitive tools, we can expect a maximum 1.0 degree error. It was shown that the mean value of the difference (± standard deviation) was0.88 ± 0.86 degrees. In other words, the average value of measurement error was less than 0.9 degree and occasionally reached to 2.5 degrees. In this step, measurement error was decreased about 30% compared to the primary mechanical goniometer. It should be remembered that the decrease is related to removal of errors attributed to the primary mechanical goniometer. The measurement of the precise distance between the two arms of the mechanical goniometer can be the source of possible error in the assessment. In general, the data obtained in this step indicated that the precision and efficiency of the hardware and software for the electro-inclinometer were acceptable.

In previous studies, it is common that reliability and objectivity of goniometer were assessed on a particular joint and on 30 subjects (Sanchew, 2009). This method suffers from two basic problems. First, the precision of the goniometer is examined in limited angles. For instance, in SLR 3 test, the range of motion in healthy people is limited to 80 to 100 degrees. Second, variation within subjects is one of the main variability sources (referred to here as “error”) from one time to the next time of testing. For instance, in the SLR test, the subject may extend his thigh angle about 75, 80 or 85 degrees three times. In addition, traditional researches usually have two repetitions of a test by one rater, avoiding sophisticated statistical analysis [14]. In this research, we performed measurements on one subject at 30 different angles trying to resolve the first two defects. If every subject could produce similar measurements on different tests by using the same device, this suggested that the device had acceptable reliability. Also, if different raters could produce similar measures using the same device, this suggested that the device had acceptable objectivity. Despite the fact that two joints were examined in this research, these joints were measured through a wide number of angles ranging from 5 to 180 degrees. For resolution of the third problem, we employed three raters and asked them to measure every angle three times. This permitted the researchers to differentiate the errors attributable to the devices from those committed by the raters.

In relation to statistical analysis for comparing the accuracy of measuring devices, only correlation coefficient was usually employed in the previous studies. Bland et al. [14], and Counter et al. (2011) have shown the weaknesses of this method [14, 15]. In order to overcome these weaknesses, several methods have been proposed such as comparing the mean differences at two test times [14]. In our study, 18 measurements were performed on a sample. Therefore, the analysis techniques used in the previous studies were not appropriate for this study. Instead of applying the correlation coefficient between the groups, we used the interclass correlation coefficient method. Reliability and objectivity were calculated by Cronbach’s alpha. In addition, the three-way ANOVA for repeated measures was used to compare changes between measurements.

In terms of reliability, the results of this study indicated that the reliability for the electro-inclinometer was higher than the mechanical goniometer in measuring the angles of “hip flexion with extended knee” and “abduction of shoulder with extended elbow”. For all the three raters, the electro-inclinometer resulted in moderate and good reliability (0.76 to 0.98) to ideal reliability (0.98 to 0.99). Regarding objectivity, the results of this analysis indicate that the electro-inclinometer was also more acceptable than the mechanical goniometer. These results showed that for raters less skilled in evaluating angles in human joints, the electro-inclinometer can be used more readily and easily. In other words, the use of electro-inclinometer is much easier than mechanical goniometer.

The purpose of analyzing the performance difference between the mechanical goniometer and the electro-inclinometer was to determine and compare the measurement error. The appropriate analysis to compare error between two devices was to use a three-way ANOVA with repeated measures. There were three independent variables including tools, raters and repetitions of measurement.

In regards to hip flexion, the results showed that the interaction between raters and devices was significant. The difference between three rater’s measurements by electro-inclinometer was lower than by goniometer. Also the result indicated that the interaction of “rater by repetitions by device” was significant. There was little difference between raters measuring by electro-inclinometer for three measuring repetitions, whereas there was more difference between raters measuring by goniometer. This result clearly shows the advantage of electro-inclinometer. The maximum of difference as large as 11 degree between measurement repetitions or raters usually is significant and this error could be wrongly deduced as a significant difference.

In related to shoulder abduction, the result indicated that interaction between raters and devices was significant. The difference between three rater’s measurements by electro-inclinometer was lower than by goniometer. Also the results of this study indicate that the interaction of “rater by repetitions by device” in shoulder abduction was not significant. However, there was little difference between raters measuring by electro-inclinometer at three measuring repetitions, whereas there was more difference between raters measuring by goniometer. This result clearly shows the advantage of electro-inclinometer over mechanical goniometer. The maximum of difference of large as 5 degree between measurement repetitions or raters usually is significant and this error could be wrongly deduced as a significant difference.

In related to both flexion and abduction tests, the results of statistical analysis indicate that the measurement errors of inter- and intra- raters were 1 to 2 degrees using the electro-inclinometer, compared to about 5 to 11 degrees which using the mechanical goniometer. According to the results, the use of electro-inclinometer compared to with a mechanical goniometer is better and provides better results. Objectivity and reliability alone aren’t an appropriate scale for determining accuracy assessment of a new device. It is better to use performance analysis.

Conflict of interest

The authors have no conflict of interest to report.

Footnotes

The angle of incidence of the quadriceps muscle relative to the patella. The Q angle determines the tracking of the patella through the trochlea of the femur.

²c²= a^2 + b²+2ab×cosα.

Straight leg raise.

Acknowledgment

I wish to thank Dr. Nader Rahnama and Dr. Linda Kelly for editing this paper and Shirin Davarpanah, Mahsa Jafari and Atiyeh Karimzadeh that help us to do this study.

References

Goniometer. 2010. Wikipedia. Available at: http://en.wikipedia.org/wiki/Goniometer. Accessed Aug 31, 2010.

Joint range of motion data using a goniometer. 2011. Aokhealth, Available at: http://aokhealth.securestand.com/xq/ASP/ProductID.614/qx/PDF/UsingaGoniometerEffectively.pdf. Accessed Feb 15, 2011.

Elveru

, Rothstein

, Lamb

. Goniometric reliability in a clinical setting: Subtalar and ankle joint measurement. Phys Ther 1998;(68):672–77.

Rajabi

, Samadi

Laboratory guide to corrective movements for graduate students, Tehran. Tehran University Publishers; 2008.

Rowe

, Myles

, Hillmann

, Hazlewood

. Validation of flexible electrogoniometry as a measure of joint kinematics. Physiotherapy 2001; 87(9):479–88.

Hayes

, Walton

, Szomor

, and Murrell

. Reliability of five methods for assessing shoulder range of motion. Aust J Physiother 2001; 47(4):289–94.

Assal

, Shofer

, Roher

, Price

, Czerniecki

, Sangeorzan

. Assessment of an electronic goniometer designed to measure equinus contracture. J Rehabil Res Dev 2003; 40(3):235–39.

Pringle

. Intera-instrument reliability of 4 goniometers. J Chiropr Med 2003; 2(3):91–5.

Sanjeev

. Introduction validation of “bodygraph” for measurement of cervical lateral flexion. Indi J of Physiother & Occupat Ther 2009; 3(3):71–4.

10.

Perriman

, Scarvell

, Hughes

, Ashman

, Lueck

, Smith

. Validation of the flexible electrogoniometer for measuring thoracic kyphosis. Spine 2010; 35(14):E633–40.

11.

Bronner

, Agraharasamakulam

, Ojofeitimi

. Reliability and validity of electrogoniometry measurement of lower extremity movement. J Med Eng Technol 2010; 34(3):232–42.

12.

Herrero

, Carrera

, García

, Gómez-Trullén

, Oliván-Blázquez

. Reliability of goniometric measurements in children with cerebral palsy: A comparative analysis of universal goniometer and electronic inclinometer. BMC Musculoskelet Disord 2011; 12(1):155.

13.

Chapleau

, Canet

, Petit

, Laflamme

, Rouleau

. Validity of goniometric elbow measurements: Comparative study with a radiographic method. Clin Orthop Relat Res 2011; 469(11):3134–40.

14.

Bland

, Altman

. Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet 1986;327(8476::307–10.

15.

Kottner

, Audige

, Brorson

, Donner

, Gajewski

, Hrobjartsson

, et al . Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. J Clin Epidemiol 2011; 64(1):96–106.