Abstract
We developed an interactive sonification system that lets users listen to their own hand/arm movement and bicep muscle signals as sonic feedback during biceps curls exercise. The system aims to improve the exercise quality and user motivation. Based on two studies, we found that the sonification was more effective in providing temporal cues to slow down the repetition but not as effective as hoped in extending the vertical movement range and increasing repetition amount. The studies provide some design guidelines for multivariate sonification and also reflect a wider potential for applications that include general fitness, physiotherapy, and sports training.
Keywords
Giving Auditory Feedback to Physical Exercise
Many research studies highlight a key problem arising from modern lifestyles: physical inactivity, which leads to an increase in mortality (Allender, Cowburn, & Foster, 2006; WHO, 2014), heart disease (Fox & Riddoch, 2004), and many other health-related issues. A survey in 2012 shows that 31.1% of adults in 122 countries are physically inactive, with the percentages being higher among developed countries (Hallal et al., 2012). This all points to the need to discover new ways to encourage and improve physical exercise. Computing technology is moving in the direction of small-size, sensor-focused devices. At the same time, health data are becoming an important focus for many technology companies as well as the general public. The combination of these two trends presents many opportunities to develop new smart technology to improve our health.
This project explored the use of interactive sonification to improve the quality and motivation of general physical exercise. Interactive sonification defines a closed-loop interactive system where the user’s input, such as a gesture or movement, is transformed into auditory information and at the same time the user’s decision on his or her next movement is influenced by this auditory information (Hermann, Hunt, & Neuhoff, 2011). Several characteristics of sound (Kramer, 1994) make audio a suitable medium for assisting physical exercise: (a) Using sound as feedback can allow visual focus to be maintained on the main tasks in sports or exercise (e.g., watching the ball or noting the position of the feet), (b) sound is good at delivering a sense of urgency when changes are required, and (c) sound can also serve as a background signal that does not dominate the user’s attentional priority.
Here we look into various examples of research and applications for real-time sonification in human body movement. Ghez, Rikakis, DuBois, and Cook (2000) used sonification to provide feedback on accurate limb movements to assist patients with proprioception deficits. Balance training using a force plate in physical therapy with real-time sonification was studied in Chiari et al. (2005), which reported an improvement in balance control among participants. Elite sport sonification, such as in rowing, also demonstrated the advantage of auditory feedback for rhythmic movement (Schaffert, Gehert, & Mattes, 2013). The high temporal resolution of sonic information can potentially link to an improvement of body coordination in various sporting activities. Musclelab is an electromyography (EMG) biofeedback system that is used in footballer training programs to evaluate kicking action and provide players with audiovisual feedback for training (Musclelab, 2017).
We highlighted a few general points based on the literature and project examples:
Sonification is capable of delivering useful cues for movement-based activity. This is particularly useful when visual attention is occupied elsewhere.
In sports, the research focus has been mainly on professional athlete training. Not many examples seek ways to benefit the general public.
Affordable sensors (e.g., EMG) and computing units (e.g., Raspberry Pi, smartphones) are becoming increasingly accessible. Compared to high-cost professional biofeedback systems for athletes or medical applications, nowadays designers can build affordable assistive systems that deliver high-quality results.
Design of a Sonification System for Physical Exercise
These points motivated us to pursue the development of an interactive sonification system with affordable sensors to provide real-time guidance on physical exercise for the general public. For our research, we looked into the use of sonification to improve the quality and motivation of biceps curl exercises. Through this example, we hope to infer the possibilities of a wide range of applications in physiotherapy, sports training, and even the gamification of general exercise.
Bicep curls are a type of isotonic exercise that involve changes in muscle volume and movement (concentric and eccentric movement; Bubbico & Kravitz, 2010; Remaud, 2013). They are among the most common free-weight training workouts, but not all do it with good efficiency. For example, a common point that many people miss out is the eccentric contraction, which is the lengthening of the muscle (lowering the dumbbell). People tend to lower the dumbbell too quickly and let gravity finish the task, and as a result, the exercise does not produce its optimal effect.
Thus, we developed SonicTrainer, which can provide auditory feedback based on predefined criteria for the purpose of ensuring exercise quality. This prototype device can capture the user’s arm movement via a Microsoft Kinect Camera v1 as well as the muscular activity with a custom-designed EMG belt (see Figure 1a). The EMG signal is connected to an Arduino Duemilanove microprocessor and a Bluetooth modem that provides wireless data transmission. The sensor signals from the EMG belt and the Kinect are sent to a software platform we developed in Max/Msp (Figure 1b), where these signals are used to create the real-time auditory feedback.
(a) Electromyography (EMG) belt. (b) SonicTrainer graphical user interface.
For the sonification, we created four types of mapping:
Linear Mode maps the movement information to a two-oscillator subtractive synthesizer, creating a rich spectral content.
Discrete Mode triggers a musical melody based on hand position.
Music Player Mode reduces the quality of the user’s music if exercise quality does not reach the required standard.
Sea Wave Mode plays an ocean sound sample with a low-pass filter that is controlled by the EMG signal to represent the sound of a contracting muscle.
In a preliminary study, we evaluated the four mapping modes based on user feedback and found that the Linear Mode provided the most comprehensive information and was the most preferred (Yang & Hunt, 2013). Therefore, for the experiments presented in the following section, the Linear Mode was the only one used for the sake of a consistent mapping across all participants.
In Linear Mode, the vertical coordinate of the hand is mapped to a linear frequency scale between 0 Hz (straightened arm) and 620 Hz (fully bent forearm), so the sound sweeps up and down in frequency, tracking the height of the dumbbell in real time. The EMG signal affects the brightness of the sound by controlling the cutoff frequency of a band-pass filter (because in the preliminary study, participants noted that a change of sound brightness perceptually corresponded to muscle tension). In addition, there is an extra occasional sound when the vertical velocity exceeds a threshold value. A noise generator is triggered to indicate to the user when he or she is lowering the dumbbell too fast (typically letting it drop under gravity alone).
Experiments in Sonifying Bicep Curls
Designing the Experiments
We hypothesized that the use of SonicTrainer for biceps curls can improve the exercising quality. Specifically, we wanted to investigate whether the auditory feedback can be useful to (a) slow down the repetition speed and (b) aim for a large movement range of the muscle contraction.
To examine the system, we conducted two experiments: a three-session comparative study (between-subject) and an eight-session within-subject crossover trial. Both were independent with no participant overlap. In both studies, three attributes were measured: repetition time, movement range, and effort. The repetition time refers to the average time for the participant to complete a full cycle of a biceps curl (concentric + eccentric phase). The movement range is the relative distance between the hand’s vertical position at the beginning of the concentric phase and the beginning of the eccentric phase. The value is normalized between straight-arm position (0) and head height (1.0). Lastly, the effort is defined as the product of total repetition and dumbbell weight. Although effort is also an important measure, we did not exclusively mention this to the participants. They had total freedom on the exercise quantity, thus allowing us to observe their own motivation.
We explained to all participants prior to the experiment that a good quality of exercise should consist of the following two criteria:
Participants should aim for a large movement range. This means trying to fully bend the forearm in the concentric phase and lower to the straight position in the eccentric phase while the upper part of the arm remains still.
The contractions should be executed at a steady and relatively slow speed, preferably at least 4 s per repetition, and the slower the better.
Studies found that slow-pace weight training can effectively increase muscle strength (Atha, 1981; Szabo, Small, & Leigh, 1999). Although there are other types of training method that suggest a different attitude to the speed of repetition, such as explosive training (Behm & Sale, 1993; Newton, Kraemer, Häkkinen, Humphries, & Murphy, 1996), the predefined criteria mainly aimed to introduce a degree of difficulty and restriction to the exercise. The system is not limited to one particular setting but is capable of readjusting the sonification mapping based on different exercising criteria.
For the comparative test (Experiment 1), 22 healthy adults (age 24 years ± 4 years) participated: 3 females and 19 males. Participants were randomly assigned to one of the two different groups—the sonification group and the control group. Only the sonification group participants received the auditory feedback. Prior to the experiment, participants were asked to select a relatively challenging dumbbell weight. A trial was given before the main experiment, allowing the participants to become familiar with the exercise. The average weights of dumbbell between the sonification (5.1 ± 1.2 kg) and control (4.7 ± 1.2 kg) group were compared using Bayes factor, with a result of B = 0.37, which supports the H0, and therefore we treated the two groups as similar in weight selection. The experiment consisted of three identical sessions with a gap less than a week between each session. At the end of the last session, a short interview was conducted with participants from the sonification group on their experience with the system.
Experiment 1 was limited in the number of sessions, and so it is difficult to show the development of a pattern of exercise results and quality measure over time. In addition, the individual differences of participants’ physical condition may also affect the outcomes. Subsequently, Experiment 2 investigated the exercise over a longer time scale. In this set-up, every participant experienced both with and without sonification. Fourteen participants (aged 24 years ± 3 years) took part in the eight-session experiment, 8 females and 6 males. The eight sessions had gaps of around 3 days between each session. Participants exercised with sonification for four sessions and without for the other four. Half of participants were in the son-con group (first sonification, then control), and the other half were in the con-son group, (control first, then sonification). Between the crossover point, a 1-week break was given, and participants agreed not to undertake any substantial biceps-related training during the entire experiment as a wear-off effect (Sibbald & Roberts, 1998). The participants were informed about the experiment and exercise criteria in the same way as the previous experiment. A questionnaire was given at the end of each session.
Experiment Results
The mean performance and t test results of the final session of both experiments are presented in Table 1. We found that in both experiments, participants with the auditory feedback exercised at a significantly slower pace on average than without the feedback. A strong effect especially is found in the between-subject study (d = 1.56). The result suggests that the temporal characteristic of sound led to the participants connecting the auditory cues to the temporal movement of the exercise. The feedback increased the awareness of the timing and resulted in a slower exercise speed, which was one of the aims of the system.
Collective Results of Both Experiments
Note. The effect size is presented as Cohen’s d. All results are the performance of the final session.
As for the movement quality, the result was not as quite expected. The decision to use Linear Mode (as the sonic mapping algorithm) was to provide a clear pitch reference for the relative position of the hand vertically. However, the result indicates that this pitch information was not suggestive enough for the users to modify their movement significantly.
The participants also showed no significant difference in effort according to the results. Since the participants were not exclusively asked to aim for more or less repetitions, it was possible that they tended to maintain a fixed amount of effort at each session. Since effort may be linked to the participant’s motivation, in the within-subject study, we also asked the participant how much he or she enjoyed each session on a scale between 1 (not at all) to 10 (very much). A paired t test showed that using sonification (6.9 ± 1.6) was significantly more enjoyable than without (5.7 ± 2.2), t(55) = 3.87, p < .001. The participants generally found the sonification enjoyable to exercise with. However, for the scope of the study, we do not have support to show what a long-term motivation of such usage would be. From the post-experiment interview in both experiments, several participants reported that the sonic feedback distracted them from feeling fatigue. However, because our survey was conducted in an exploratory way, we do not have quantifiable details to demonstrate this effect.
Interpretation of the Pros and Cons of Using Auditory Feedback
From the experiments, we see that portraying exercise information, such as muscular activity and kinematic data, is particularly effective at influencing the speed of the exercise movement yet not particularly useful in encouraging the users to achieve a larger contraction range. This points at a design intuition that using auditory feedback can be especially useful in events where rhythm and pacing are critical, such as cycling and running, or specific types of weight training. It also hints at a possibility of incorporating auditory cues into rhythmical music as feedback for achieving an even better temporal response. At the same time, this could be more pleasant for long-term usage as music is often used by people exercising to relieve boredom and maintain motivation. For example, one of the modes we designed but did not test in these studies is a musical mode that degrades quality of the music via pitch shifting and additional noise if the exercising quality does not reach certain specified thresholds.
Despite having a higher mean value in effort expended with the auditory feedback, analysis shows that the difference is not statistically significant.
Reviewing the sonification design, our intention was to create a multivariate auditory stream that displays pacing via temporal patterns, relative hand position via pitch, and muscular activity via spectral shifting (filtering). Reflecting on the experimental results, only the temporal pattern was shown to be effective, but the other two were not, especially the pitch-to-position mapping. Peres (2012) demonstrated a study to show different mappings in portraying auditory graphs and showed that pitch mapping performed well. During the experimental design, we felt that pitch mapping was the most straightforward, which was also in line with the preliminary study finding, and so we stuck to the mapping decision. Nonetheless, the results we found have some interesting implications for multivariate parameter mapping in a sonification design. We suspect in this case that the temporal pattern may be such a dominant cue that other raw information (e.g., pitch = position) that needs extra cognitive interpretation could have been neglected. Thus, as a lesson to future multivariate sonification design, it may be more effective to identify the information that is not dominant and use processed feedback for a more direct notification. For example, it could be possible to use speech reminders as an exercise repetition counter or a position threshold notification sound to remind user to use the full motion range.
As for future applications, screen-based design has many merits but could also introduce constraints such as occupying visual attention and being tied to a stationary system that limits the activity’s flexibility. By using sound, we explored alternatives to provide sufficient feedback to the user, freeing up visual attention that potentially allows applications such as outdoor usage. The use of a nonvisual data display can be particularly useful in wearable technology and small-size computing devices such as smartphone and smartwatches, making biofeedback-assisted physical exercises more practical and even safe as they no longer require visual attention.