Validation of Electronic Activity Monitor Devices During Treadmill Walking

Abstract

Introduction:

The purpose of this study was to assess the validity of the step count measurement of commercial electronic activity monitor devices. Two popular models, Fitbit Charge HR and Mi Band 2, were selected for treadmill walking in a single session.

Materials and Methods:

Thirty healthy volunteers walked at five predetermined speeds (0.90, 1.12, 1.33, 1.54, and 1.78 m/s) on a treadmill with both Fitbit Charge HR and Mi Band 2 worn on their dominant hand's wrist. Observers counted the steps, with the aid of taped video, which was taken as the criterion measure for steps. The validity of the electronic activity devices was assessed by (1) Paired sample t test with the criterion measures and (2) Pearson's correlation coefficients and the corresponding p-values were calculated to compare the output of devices with manual step count. In addition, Bland-Altman plots were constructed to visually inspect the data and to assess agreement with the criterion measures.

Results:

There were no significant differences in step measurement between Fitbit Charge HR and Mi Band 2 with the criterion measures. Besides, there was a very strong agreement between step count measurements obtained using the Fitbit Charge HR (r = 0.99) and the Mi Band 2 (r = 0.99), at five predetermined speeds while comparing with the observed step counts.

Conclusion:

Both Fitbit Charge HR and Mi Band 2 provided accurate step count measurement in the treadmill walking test.

Introduction

Ancient physician Hippocrates believed that walking was one of the best medicines for human beings. Sedentary adults, who accumulate 30 min of brisk walking per day (or activities that require equivalent energy expenditure), could receive significant health benefits.¹ Although human movement is not limited to bipedal locomotion, it is a fundamental part of daily life and is the prominent focus of public health physical activity guidelines.²

Steps can be accumulated throughout the day during chores, occupational requirements, child care, errands, and transportation.³ As such, the step-based recommendations for physical activity might be more appropriate and better received by the large segment of the population who do not regularly engage in any sport or exercise apart from walking.⁴

Technology facilitates our healthy practices.^5,6 In the recent years, commercially available electronic activity monitor system (EAMS) is growing in popularity. The EAMS is equipped with several fundamental behavior change techniques related to physical activity behavior.⁷ These techniques include goal setting, review of behavioral goals, and social support.⁸ These lightweight, portable EAMS contains triaxial accelerometer claim to track multiple measures of physical activity, including step count and distance traveled. Systematic reviews have been carried out to check validation of different functions, including step, distance, sleeping quality, and energy expenditures on these EAMS.^9,10 Regarding the step measurement, previous studies showed that models of Fitbit were high in correlations with different criteria, namely manually step counting and pedometers or accelerometers.^11
–13 However, there are several studies that evaluated the accuracy of Fitbit models, with mixed results. Fitbit Zip and Fitbit Flex overestimated the number of steps during free-living physical activity.^14,15 In the meanwhile, Fitbit One was proved as an accurate device for measuring steps in treadmill walking¹⁶ and 2-min walking test.¹³

To our knowledge, the validation of popular new devices, Fitbit Charge HR (Fitbit, Inc., San Francisco, CA) and Mi Band 2 (Xiaomi, Corp., China) have not yet been covered in a laboratory-based study. Therefore, the purpose of this pilot study is to assess the validity and reliability of the step count of these two popular models of new electronic activity monitor devices in a population of healthy adults during treadmill walking at multiple speeds.

Materials and Methods

INSTRUMENTS

A single session of treadmill walking was used for assessing the validity of the Fitbit Charge HR and Mi Band 2 step counter in August 2016. The EAMS, Fitbit Charge HR (launched in 2015), and Mi Band 2 (launched in 2016) are small light-weighted devices. The Fitbit Charge HR has a microelectromechanical triaxial accelerometer and Mi Band 2 has a military-grade accelerometer that converts acceleration to step counts using proprietary algorithms. Both designs of Fitbit Charge HR and Mi Band 2 are fit for wearing as wristband.

The testing protocol is based on previous activity monitor validation study.¹² Previous studies of the correlation of activity monitor output to observed step counts range from 0.5 to 1.0. Based on the most conservative findings (correlation of 0.5), α = 0.05, and a power of 0.80, a priori sample size was calculated as 28 participants. In this study, 30 interested volunteers with the ability to walk continuously on a treadmill for 30 min without aid were recruited from the university community. Participants walked at five predetermined speeds (0.90, 1.12, 1.33, 1.54, and 1.78 m/s) on a calibrated treadmill. Step count, measured by trained observers by using tally and with the aid of videotaping, was taken as the criterion measure for steps.

The tests were carried out by using a calibrated treadmill in the laboratory. Informed consent was obtained from the volunteers after the explanation of possible risks and benefits associated with the experimental procedure. Participants' demographic data (age, height, mass, gender, and dominant hand) were measured and collected during the single testing session. Height, mass, and gender data were entered into the Fitbit Charge HR and Mi Band 2 account for each participant before the treadmill walking.

The test administrator delivered instruction on how to use the treadmill to all participants before the start of the treadmill walking. Participants practiced for 5 min to familiarize the treadmill walking. Fitbit Charge HR and Mi Band 2 were then placed on the participant's dominant wrist. Participants were given a 5 s warning before the end of each walking trial and stopped walking after 5 min at each speed. Adequate rest between each trial was given and the step count on each Fitbit Charge HR and Mi Band 2 was recorded at the meanwhile. Two observers, who manually counted step number during the videotaping, independently analyzed step count data. Step count was reanalyzed if observers were not in agreement.

Paired sample t tests were used to evaluate speed-specific mean difference in step counts between the Fitbit Charge HR, Mi Band 2, and the criterion measures. In addition, Bland-Altman plots were used as the standard method for assessing agreement between these two devices, both visual and statistical interpretation. The difference in the step count, measured by the two devices, is plotted against the averages. Moreover, Pearson's correlation coefficients and the corresponding p-values were calculated to provide an indication of the relationship between the recorded step counts from the Fitbit Charge HR and Mi Band 2. Finally, to facilitate the comparison with previous activity monitors studies, the percent relative error of the Fitbit Charge HR and Mi Band 2 were also calculated. Percent relative error was calculated by: \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\usepackage{upgreek}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document} \begin{align*}{ \rm Percent \ relative \ error} = \left( {{ \rm Devices \ Output - Observer \ Count}} \right) / \\\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad{ \rm Observer \ Count} \ \times 100 \%\end{align*} \end{document}

All statistical analyses were conducted using SPSS v23.0.0.

Results

Thirty (15 males) volunteers, mean (standard deviation [SD]) age 32.1 (8.66) years, participated. All were right-hand dominant. Table 1 summarizes the mean SD of step count measurements using the Fitbit Charge HR and Mi Band 2 activity monitor, together with the step count measured by the observer at five predetermined speeds in the 5-min treadmill walking. The paired sample t test shows that there were no significant differences in step counts between the Fitbit Charge HR, Mi Band 2, and the criterion measures. Table 2 shows the figures of the paired sample t tests of Fitbit Charge HR. There was no significant difference between step counts measured by Fitbit Charge HR at all five predetermined speeds. At the slowest speed, 0.90 m/s, there was no significant difference in the step measurement for Fitbit Charge HR and observer (criterion) counted; t (29) = −1.362, p = 0.184. While at the fastest speed, that is 1.78 m/s, there was also no significant difference in the step measurement for Fitbit Charge HR and observer counted; t (29) = 0.166, p = 0.076.

Table 1.

Mean (Standard Deviation) of Step Count Measurements Using the Fitbit Charge HR and Mi Band 2 Activity Monitor in 5-Minute Treadmill Walking

SPEED (M/S)	FITBIT CHARGE HR	MI BAND 2	OBSERVER COUNT
0.90	500 (34)	500 (34)	501 (34)
1.12	535 (32)	536 (31)	537 (31)
1.33	567 (33)	569 (34)	570 (35)
1.54	597 (34)	598 (34)	598 (34)
1.78	634 (37)	634 (37)	636 (37)

Table 2.

Pair Sample t Tests on Step Measurements of Fitbit Charge HR and the Criterion Measure in Different Speeds

SPEED (M/S)	MEAN BIAS	t	p
0.90	−1.433	−1.362	0.184
1.12	−1.633	−1.601	0.120
1.33	−0.567	−0.488	0.630
1.54	−1.500	−1.934	0.063
1.78	−1.500	0.166	0.076

Regarding Mi Band 2, Table 3 displays the figures of the paired sample t tests. There were no significant differences between step counts measured by Mi Band 2 at all five predetermined speeds. At the slowest speed, 0.90 m/s, there was no significant difference in the step measurement for Mi Band 2 and observer (criterion) counted; t (29) = −1.697, p = 0.100. While at the fastest speed, that is 1.78 m/s, there was no significant difference in the step measurement for Mi Band 2 and observer counted either; t (29) = 1.602, p = 0.120. Therefore, both step count function of Fitbit Charge HR and Mi Band 2 were valid.

Table 3.

Pair Sample t Tests on Step Measurements of Mi Band 2 and the Criterion Measure in Different Speeds

SPEED (M/S)	MEAN BIAS	t	p
0.90	−1.233	−1.697	0.100
1.12	−1.30	−1.875	0.071
1.33	−0.900	−1.865	0.072
1.54	−0.200	−0.270	0.789
1.78	1.467	1.602	0.120

For visual and statistical interpretation, Bland-Altman plots were used to illustrate the differences between step count measurement by (1) Fitbit Charge HR and (2) Mi Band 2 at five predetermined speeds (Fig. 1).

Fig. 1.

Bland and Altman plots representing comparisons between the criterion measure (observer counts) and the Fitbit Charge HR and Mi Band 2 step count outputs for the five different predetermined speeds: (a) 0.9 m/s, (b) 1.20 m/s, (c) 1.33 m/s, (d) 1.54 m/s, and (e) 1.78 m/s. Solid line indicates the mean difference between observer count and device(s), and the dashed lines indicate limits of agreements (1.96±SD). SD, standard deviation.

In addition, there was a very strong correlation between step count measured by the observer and corresponding step count measurement using the Fitbit Charge HR, and also between step count measured by the observer and corresponding step count measurement using the Mi Band 2. Table 4 reports the Pearson's correlation coefficient at different speeds. The Pearson's correlation coefficients of Fitbit Charge HR and Mi Band 2 ranged from 0.985 to 0.993 and 0.991 to 0.997 correspondingly. The limits of agreement (±1.96 SD) reflect where 95% of all differences between measurements are expected to lie. Regarding the Fitbit Charge HR, the mean bias at speed 0.9 and 1.78 m/s were −1.43 ± 5.763 and −1.50 ± 4.46 with the 95% limits of agreement from 9.87 to −12.73 and 7.25 to −10.25, respectively. Concerning the Mi Band 2, the mean bias at speed 0.9 and 1.78 m/s were −1.23 ± 3.98 and 1.46 ± 5.01, respectively, with the 95% limits of agreement from 6.57 to −9.03 and 11.30 to −8.36 correspondingly. In both cases, there are no apparent systematic bias based on step count number, and most data points fell within the 95% limits of agreement.

Table 4.

Comparison of Step Count Measurements Using the Fitbit Charge HR and Mi Band 2 Activity Monitor

SPEED (M/S)	FITBIT CHARGE HR		MI BAND 2
	MEAN BIAS (STEP ± SD)	CORRELATION COEFFICIENT, r	MEAN BIAS (STEP ± SD)	CORRELATION COEFFICIENT, r
	0.90	−1.43 ± 5.763	0.986^a	−1.23 ± 3.98	0.993^a
1.12	−1.63 ± 5.59	0.985^a	−1.30 ± 3.80	0.993^a
1.33	−2.33 ± 4.88	0.991^a	−0.90 ± 2.64	0.997^a
1.54	−1.50 ± 4.25	0.992^a	0.20 ± 4.06	0.993^a
1.78	−1.50 ± 4.46	0.993^a	1.46 ± 5.01	0.991^a

Indicates significance at p < 0.001.

SD, standard deviation.

In addition, the percent relative error of Fitbit Charge HR (Fig. 2A) and Mi Band 2 (Fig. 2B) were <0.5% for all treadmill speeds.

Fig. 2.

(A) Mean (SD) percent relative error of step counts during treadmill walking for Fitbit Charge HR at five different speeds, (B) Mean (SD) percent relative error of step counts during treadmill walking for Mi Band 2 at five different speeds.

There was a significant (p < 0.001) and very strong correlation between step count measured by the observer and corresponding step count measurement using the Fitbit Charge HR (r = 0.99) and also between step count measured by the observer and corresponding step count measurement using the Mi Band 2 (r = 0.99) at all five predetermined speeds. The correlation plots for the Fitbit Charge HR and Mi Band 2 for step count measurements taken at different speeds are shown in Figure 3.

Fig. 3.

Scatter plots with regression function comparing the step measurement between the criterion (observer counts) and the Fitbit Charge HR and Mi Band 2 step count outputs for the five different predetermined speeds: (a) 0.9m/s, (b) 1.20 m/s, (c) 1.33 m/s, (d) 1.54 m/s, and (e) 1.78 m/s.

Discussion

The use of pedometer is associated with significant increase in the number of steps and significant decreases in body mass index and blood pressure in a previous study.⁴ Therefore, it is crucial to have accurate step measurement of these electronic activity monitor devices for health benefits. In this study, the step count output of the Fitbit Charge HR and Mi Band 2 are valid and reliable at all speeds. This is consistent with other research that Fitbit tracker is a reliable and valid device for step counts.¹¹ The primary aim of this study is to assess the validity of step count output from the electronic activity monitors, Fitbit Charge HR and Mi Band 2, in the treadmill walking. Our study shows that both of them are valid and reliable devices for monitoring step counts, with relative percent errors of <0.5%. In the previous study, the percent relative error of the Fitbit One is <1.3%.¹² This study further echoes accuracy of Fitbit tracker in its low percent relative error. In the meanwhile, the relatively cheap electronic activity monitor, Mi Band 2, also has low percent relative error in treadmill walking test, which show big improvement with its preceding model, Mi Band, in the author's previous study.¹⁷

Although some studies found that old models of Fitbit have overestimated step counts,¹⁸ it is found that this new model, Fitbit Charge HR is accurate in measuring steps. It may be due to different wearing positions of these devices. As both Fitbit Charge HR and Mi Band 2 are designed to fit for wearing as wristband, it is suggested to use the electronic activity devices as instructed for accurate measurement. As Fitbit Charge HR also showed its accuracy in measuring heart rate,¹⁹ the health-related information obtained in Fitbit Charge HR would be further utilized in the aspect of e-health and telemedicine. Besides, with the relatively low cost of using Mi Band 2 (USD $30), it is expected that more and more people are willing to take the lead to use electronic activity devices to facilitate their health status monitoring and recording.

Further investigation is necessary for these two devices in measuring step count in daily use, not only in the laboratory setting. Besides, different functions, for example, heart rate function during exercise, sleeping quality could also be validated for further use in the e-health aspect.

In short, as step count is a simple way to quantify the amount of physical activity and have been used as a guide for the amount of physical activity recommended by governments, it is important for these electronic activity monitors to be accurate.

Footnotes

Acknowledgment

The authors would like to thank Mr. Quach Binh from the Department of Physical Education, Hong Kong Baptist University, for his technical assistance with our research.

Disclosure Statement

There is no commercial association with either Fitbit, Inc. or Xiaomi, Corp. that would create a conflict of interest relevant to this study. K.M.T. is a lecturer of the Hong Kong Institute of Vocational Education. No competing financial interests exist.

References

Pate

, Pratt

, Blair

. Physical activity and public health: A recommendation from the Centers for Disease Control and Prevention and the American College of Sports Medicine. JAMA, 1995; 273:402–407.

Tudor-Locke

, et al. How many steps/day are enough? For older adults and special populations. Int J Behav Nutr Phys Act, 2011; 8:80.

Tudor-Locke

, Bassett

. How many steps/day are enough? Preliminary pedometer indices for public health. Sports Med, 2004; 34:1–8.

Bravata

, et al. Using pedometers to increase physical activity and improve health: A systematic review. JAMA, 2007; 298:2296–2304.

Chaudhry

, Wang

, Maglione

, Mojica

, Roth

. Systematic review: Impact of health information technology on quality efficiency and costs of medical care. Ann Internl Med, 2006; 144:742–752.

Starfield

. Patient lists and patient-focused care overtime. New York: Oxford University Press, 1998.

Michie

, et al. Effective techniques in healthy eating and physical activity interventions: A meta-regression. Health Psychol, 2009; 28:690–701.

Lyons

, Lewis

, Mayrsohn

, Rowland

. Behavior change techniques implemented in electronic lifestyle activity monitors: A systematic content analysis. J Med Internet Res. 2014; 16:e192.

Evenson

, Goto

, Furberg

. Systematic review of the validity and reliability of consumer-wearable activity trackers. Int J Behav Nutr Phys Act, 2015; 12:159.

10.

Lewis

, et al. Using an electronic activity monitor system as an intervention modality: A systematic review. BMC Public Health, 2015; 15:585.

11.

Tully

, McBride

, Heron

, Hunte

. The validation of Fitbit Zip™ physical activity monitor as a measure of free-living physical activity. BMC Res Notes, 2014; 7:952.

12.

Takacs

, Pollock

, Guenther

, Bahar

, Napier

, Hunt

. Validation of the Fitbit One activity monitor device during treadmill walking. J Sci Med Sport, 2014; 17:496–500.

13.

Paul

, et al. Validity of the Fitbit activity tracker for measuring steps in community-dwelling older adults. BMJ Open Sport Exerc Med, 2015; 1:e000013.

14.

Alharbi

, et al. Validation of Fitbit-Flex as a measure of free-living physical activity in a community-based phase III cardiac rehabilitation population. Eur J Prev Cardiol, 2016; 23:1475–1485.

15.

Leininger

, Cook

, Adams

. Validation and accuracy of Fitbit Charge: A pilot study in a university worksite walking program. J Fitness Res, 2016; 5:3–9.

16.

Diaz

, Krupka

, Chang

, Shaffer

, et al. Validation of the Fitbit One for physical activity measurement at an upper torso attachment site. BMC Res Notes, 2016; 9:213.

17.

Tam

. Validation of Fitbit HR and Mi Band during treadmill walking. 5th HKASMSS Stduent Conference on Sports Medicine, Rehabilitation and Exercise Science, Hong Kong, November 26, 2016.

18.

Reid

, Insogna

, Carver

, Comptour

, Bewski

, Sciortino

, Andersen

. Validity and reliability of Fitbit Activity monitors compared to ActiGraph GT3X+ with female adults in a free-living environment. J Sci Med Sport, 2017; 20:578–582.

19.

Wallen

, Gomersall

, Keating

, Wisløff

, Coombes

. Accuracy of heart rate watches: Implications for weight management. PLoS One, 2016; 11:e0154420.