Usability and Feasibility of Mobile Phone Diaries in an Experimental Physical Exercise Study

Abstract

Objectives: Wireless and mobile phone technology as a method of data collection will increase alongside conventional methods. The aim of the present study was to evaluate the use of a mobile phone application for recording symptoms and physical activity exertion during an experimental physical exercise study. Materials and Methods: An experimental study on the effects of physical exercise on the well-being of menopausal women included 158 subjects between 44 and 63 years of age. The women were randomized into intervention and control groups. All participants in both groups reported daily symptoms by responding to morning and evening questionnaires via mobile phones. The usability of the mobile phone as a data collector tool was evaluated with the System Usability Scale (SUS) questionnaire 2 months after the intervention. The feasibility evaluation was based on the frequency of responses and open questions. Results: The response rates were about 70% to both morning and evening questionnaires. The average frequency of responses (n =158) to morning questionnaires was 125 (±40)/170 (±14) and to evening questionnaires was 118 (±40)/171 (±14). The response rate did not differ between the intervention and the control groups. The SUS score was on average 75.4 (range, 0–100; n =107). Conclusion: A mobile phone diary is a feasible and usable tool for data collection in clinical trials.

Introduction

In clinical trials data collection includes recording of symptoms and measurements of the bodily functions or patient's daily condition. Paper-based data collection has been a traditional method of data collection for decades. New technology offers many other options in data collection including mobile phones or computer-based devices that utilize wireless and mobile phone technology. Handheld computers have been developed since the early 1990s, and free or commercial software for developing diaries is available. The use of electronic diaries as data collectors or for data monitoring has increased rapidly in clinical trials and research because they are faster and popular with users.^1,2 Traditionally entries have been made in paper diaries once a day, usually at the end of the day, and such data may be then based on selective memory. With electronic and mobile phone diaries, data such as symptoms could be measured beyond the place or time and recorded as they occur. It is known that patients (users) do not follow the instructions and may fill in the missing entries afterwards, possibly affecting the results of an intervention or treatment. The electronic diary marks all entries in the program, which ensures that the patient adheres to the protocol. Electronic diaries have been shown to generate valid data for studying symptoms and also in healthcare.^1,3

Electronic diaries have been used in clinical trials in various fields of medicine and health. The range of applications is wide, from electronic patient diaries to electronic case report forms. The sizes and the durations of the studies have varied.⁴ Lauritsen et al.⁵ compared paper diaries with electronic and telephone diaries for the recording of symptoms in a 4-week treatment of gastroesophageal reflux disease. The main conclusion was that paper diaries were not filled in at appropriate times and that data from electronic and telephone diaries was of better quality. Walker et al.⁶ compared paper and handheld computer diaries in a 6-month trial with patients treated for severe hemophilia; the compliance in the computer group was about 86.2% compared with only 48.3% among patients using paper diaries. Most of the studies concluded that electronic diaries have potential for data collection if validation and usability are taken into account.^2
–4,7,8 The aim of the present study was to evaluate the usability and feasibility of a mobile phone program in recording of symptoms during a physical exercise trial.

Materials and Methods

Subjects

One hundred fifty-eight women who had one or more responses in the mobile phone diary were included in the analysis. The women were on average 54.3 years of age, and about 72% were working full-time. The participants worked 7–60 h a week (mean working hours, 34.9±16.2). The subjects were mostly working in mentally demanding jobs (for example, as office workers), but also in physical jobs (such as cleaners) and mixed (physical and mental demanding) jobs (such as nurses). A third of the women had a university degree. Baseline characteristics of subjects are presented in Table 1.

Table 1.

Baseline Characteristics of the Subjects

	ALL (N=158)
Age (years)	54.2±3.5
Working hours (per week)	34.9±16.2
Work demands (%)
Physical	12.9
Mixed	24.5
Mental	62.6
Education (%)
Primary	33.5
Secondary	40.6
Tertiary	25.8
Type of employment (%)
Full-time	71.9
Part-time	9.8
Other^a	18.3

For example, unemployed, retirement, caring for close relatives.

Study Design

The study where we analyzed the use of mobile phones was a randomized exercise trial intended to study the effects of increased physical activity on menopausal symptoms and quality of life. The study was approved by the ethical committee of the Pirkanmaa Hospital District in Finland. The women were recruited through announcements in local newspapers. From a total of 350 women who contacted the researchers, 176 participated in the study; 174 were excluded because they did not meet one or more of the inclusion criteria. Inclusion criteria were as follows: symptomatic (severe, very severe, or moderate daily hot flushes), age 40–62 years, either no current use of postmenopausal hormone replacement therapy or withdrawal cessation of hormonal replacement therapy (washout period of 3 months), sedentary physical activity (physical exercise less than twice a week), and 6–36 months from last menstruation.

The duration of the intervention was 24 weeks. A more detailed description of the intervention has been documented elsewhere.⁹ The exercise program included aerobic training four times a week, 50 min per session, with a progressive increase in intensity. Adherence to the trial was supported by an option to participate in supervised aerobics or step aerobics sessions at the research institute twice a week. In the present study we analyzed also separately differences between the intervention and control group. The researchers met with the intervention group regularly every second week during the intervention, which could have had a motivating effect on their response activity. We also checked separately those who used their own phones and phones supplied by the research project.

Mobile Phone Diary

A mobile phone diary was used to collect data on symptoms and amount of physical exercise. The application program was commercial, Symbian-based and was downloaded primarily onto the subjects' own mobile phones. If the subject's mobile phone did not have the necessary features a phone was offered to her by the research project. In both groups 64.6% of the participants used their own phone. All subjects participated in the introduction and training session where the program was installed on their mobile phones. The subjects had a 2-week “running-in” period for the exercise program when they could also practice using the mobile diary program. All costs of the use of the mobile phone diary were covered by the research project.

All data were transferred via Internet (3 G or WAP) to a server of the service provider. The users' interface was protected by user name and password. After the application was installed on the user's mobile phone, she could use the application without logging in.

Questionnaires

Information on daytime and nighttime symptoms was collected by mobile phone questionnaires twice a day. The mobile phone questionnaire included prespecified questions on hot flushes, sleep, and other symptoms. Symptom questions were modified from the Symptom Check List (SCL-90).¹⁰ Participants were asked to fill in the morning questionnaires on waking up in the morning and the evening questionnaires on retiring to bed. Questionnaires could not be filled in afterwards.

The morning questionnaire included five specific questions on hot flushes (yes/no), a question on night sweats (yes/no), and two questions on sleep quality (yes/no and 1=good to 5=poor). The evening questionnaire included the same questions on hot flushes as the morning questionnaire, with additional (yes/no) questions regarding headache, mood swings, dysphoria, depression, dryness of the vaginal mucosa, and urinary and other symptoms (nine questions). If a participant had symptoms (she responded yes), specified questions were presented. The intervention group also reported their exercise after the training session via mobile phone questionnaire. All participants could also send messages to the researcher from their mobile phone programs.

Among the questionnaires there were three different types of response options: select yes/no, multiple-choice, and text response (caption-choice).

Usability Measures

The usability of the mobile phone as a data collection tool was evaluated with the System Usability Scale (SUS)¹¹ after the intervention. The SUS scale can be used to assess the usability of a variety of products or services. The questionnaire investigated users' experiences of questionnaires by mobile phone and the technical support required with the system. Responses were given on a scale from 1 (totally disagree) to 5 (totally agree). In the SUS questionnaire there are five positive statements and five negative statements, which alternate.¹¹ Experiences of the use of the mobile phone diary were elicited with two open-ended questions. The questions were: Which factors did you find positive when using the mobile phone diary, and which factors did you find negative when using the mobile phone diary?

Pilot Study

A 6-week pilot study developing the mobile phone program (n=11; mean age of respondents, 53.5 years) was conducted in spring 2008. Participants used mobile phones to fill in questionnaires on menopausal symptoms and physical exercise (intervention group). The system sent a reminder text message to the participant if she had not been active in the program during the last 48 h. Altogether nine of the 11 participants used mobile phone diaries as requested. None of them had used a similar system before. The average SUS score was 75.8 (range, 0–100), showing good usability.¹² The positive features of the mobile questionnaires were usability, simple response options, and availability of the mobile phone compared with the paper questionnaire. Fine-tuning after the piloting was as follows: A faster entry method, a guideline for all questions, easier usage, a text message reminder, and automatic feedback for subjects through the program or a text message. Web-based reports (e-mail) on inactive participants for researcher were also included.

Statistical Analysis

Statistical analysis was performed with SPSS version 15.0 for Windows. Characteristics of the groups were tabulated as means and standard deviations or percentages. Differences between the groups were assessed using t-tests for continuous variables when they were normally distributed and using the Mann–Whitney test when they were non-normally distributed. Normality was tested with histograms and the Kolmogorov–Smirnov test.

Results

The average frequency of responses (n=158) to morning questionnaires was 125 (±40)/170 (±14) and to evening questionnaires was 118 (±40)/171 (±14). The response activity did not differ between the intervention and the control groups (Table 2). There were no differences in the response rates between those who used their own phones or phones supplied by the research. Figure 1 describes the entries in the program every month.

Fig. 1.

Responses to the mobile phone questionnaires, morning and evening, by month of intervention in the intervention and control groups. SUS, System Usability Scale.

Table 2.

Response Rates to the Mobile Phone Questionnaires in the Study: Intervention and Control Groups

	INTERVENTION GROUP (N=79)	CONTROL GROUP (N=79)	P DIFFERENCES	ALL (N=158)
Morning questionnaires (mean±SD %)	74.4±22.3	71.6±21.7	0.282	73.0±22.0
Evening questionnaires (mean±SD %)	69.4±23.6	68.6±21.8	0.573	69.0±22.7
Distribution of response percentages (morning)
0–50%	16.5	17.7		17.1
51–70%	13.9	19.0		16.5
>71%	69.6	63.3		66.6
Distribution of response percentages (evening)
0–50%	16.5	19.0		17.7
51–70%	25.3	22.8		24.1
>71%	58.2	58.2		58.2

Usability

In all, 107 women completed the usability questionnaire after the intervention. The average SUS score was 75.4 (range, 0–100) (Fig. 2).

Fig. 2.

Distribution of mean scores of the System Usability Scale (n=107). CG, control group; IG, intervention group.

In the responses to the SUS questionnaire two questions out of 10 scored higher than 4 (product was easy to use, use quick to learn) (Table 3).

Table 3.

Responses to Individual System Usability Scale Statements

STATEMENT	MEAN (SD)
1. I would like to use this product frequently.	3.52 (±1.10)
2. The product was unnecessarily complex.	1.90 (±0.98)
3. The product was easy to use.	4.35 (±0.98)
4. I would need support from a technical person to be able to use this product.	1.76 (±1.04)
5. Various functions in the product were well integrated.	3.77 (±0.89)
6. There was too much inconsistency in this product.	2.28 (±0.98)
7. Most people would learn to use this product very quickly.	4.22 (±0.82)
8. The product was very awkward to use.	1.86 (±1.10)
9. I felt very confident using the product.	3.91 (±1.07)
10. I had to learn many things before I could get going with this product.	1.86 (±0.98)

Scoring was based on a scale from 1=Totally Disagree to 5=Totally Agree.

Users' Experiences

Of all respondents who answered the open-ended questions (n=111) 101 gave positive feedback. The mobile phone diary was considered a fast and easy way for participants to respond to questionnaires. Users commented that doing the diary "has become a routine." It was also noted that "since mobile phones are part of everyday life, it is easy to remember to fill in the questionnaires."

Eighty subjects responded on the question: “Which factors did you find negative when using the mobile phone diary?” The most common answer was "nothing." Negative comments included technical problems such as network connection problems. Participants also made negative comments on the relevance of the questionnaires: “Questionnaires were monotonous" and "there were no option to explain and add responses." Some of the participants who did not use their own phone felt that it was not easy to remember to respond.

Discussion

Our results permit the conclusion that the response rate and usability of the mobile phone diary were good throughout the intervention. This concurs with other studies on electronic diaries.^2,3,5,7,9 Burton et al.¹ in their review reported compliance rates from 76% to 100% of all possible entries. Compared with other studies the duration of our study was quite long and demanding for the participants. However, none of the dropouts discontinued because of the mobile phone diary, but for other reasons such as personal circumstances or chronic illness. In research it is important that subjects participate actively in the study and that there should be no missing data; this improves the quality and the reliability of the data.^5,13 However, according to Grady et al.,¹⁴ a long-term follow-up (7 days vs. 3 days) may increase the perceived stress instead of improving the quality of the data.

The usability of the mobile phone diary was rated good,¹² although none of the participants had used a similar application before. Earlier studies have shown that subjects prefer electronic diaries to traditional paper-and-pen methods. Participants who have used electronic versions of a diary have been more likely to keep it in a more usable and faster way than the traditional method (e.g., paper-and-pen).^2,5

The perceived problems during the intervention were mostly due to network problems with the specific mobile operator, which could not be anticipated. The problems were not continuous, but the subject had to try to log on several times, which was stressful to the user. It is important to ensure that the method does not strain the subject too much.¹⁵

The software used enabled an easy follow-up of the response activity, and there was also an opportunity to give the participants feedback. Online registration of the responses improved the quality of the collected material, and problems could be spotted quickly, as shown in other studies.^3,15

Limitations And Strengths Of The Study

As one weakness in our study we acknowledge that the mobile phone diary was not validated against another method (e.g., pen-and-paper). Although we tested the method in a pilot study, we did not compare the results obtained with the pen-and-paper method. Although the frequency of hot flushes was reliably reported, more than 60% of the daily mobile phone responses lacked information regarding their severity because this was not compulsory in the questionnaire. Most of the women were unsure about the severity of their hot flushes and left the area blank. This is a limitation that should be taken into account in future mobile phone questionnaires. In the feedback questionnaires after the intervention, the women reported difficulties in assessing the severity of their symptoms during the mobile phone survey. There were also problems in the research mobile phones offered to those subjects who could not use their own mobile phones. In phones other than the women's own, prepaid subscriptions valid for 6 months were used. Because in some cases the time was too short, a reload of the subscription was needed. Unexpected costs related to some subscriptions were also incurred. There were also some Internet connection problems with some operators in dead zones and outside Finland.

The strengths of our study include a long period of reporting symptoms and activities, good response rates, and good user experiences among the middle-aged women with no experience of the application before the study. We also performed a pilot study and evaluated the usability of the mobile phone application before starting the actual trial.

Conclusions

The mobile phone diary is a feasible and usable tool for self-reported data collection in clinical trials. In further studies more research on the usability of mobile solutions as a research tool in different subject areas is needed.

Footnotes

Acknowledgments

We thank Katriina Ojala from the UKK Institute of Health Promotion Research and Jaana Moilanen from the University of Tampere (School of Health Sciences) who participated in the project. Financial support for the trial was received from the Academy of Finland and the Juho Vainio Foundation.

Disclosure Statement

No competing financial interests exist.

References

Burton

, Weller

, Sharpe

. Are electronic diaries useful for symptoms research? A systematic review. J Psychosom Res, 2007; 62:553–561.

Lane

, Heddle

, Arnold

, Walker

. A review of randomized controlled trials comparing the effectiveness of hand held computers with paper methods for data collection. BMC Med Inform Decis Mak, 2006; 6:23.

Krishna

, Boren

, Balas

. Healthcare via cell phones: A systematic review. Telemed J E Health, 2009; 15:231–240.

Koop

, Mosges

. The use of handheld computers in clinical trials. Control Clin Trials, 2002; 23:469–480.

Lauritsen

, Degl'Innocenti

, Hendel

, Præst

, Lytje

, Clemmensen-Rotne

et al. Symptom recording in a randomised clinical trial: Paper diaries vs. electronic or telephone data capture. Control Clin Trials, 2004; 25:585–597.

Walker

, Sigouin

, Sek

, Almonte

, Carruthers

, Chan

et al. Comparing hand-held computers and paper diaries for haemophilia home therapy: A randomized trial. Haemophilia, 2004; 10:698–704.

Bolger

, Davis

, Rafaeli

. Diary methods: Capturing life as it is lived. Annu Rev Psychol, 2003; 54:579–616.

Kreindler

, Levitt

, Woolridge

, Lumsden

. Portable mood mapping: The validity and reliability of analog scale displays for mood assessment via hand-held computer. Psychiatry Res, 2003; 120:165–177.

Luoto

, Moilanen

, Heinonen

, Mikkola

, Raitanen

, Tomas

, Ojala

, Mansikkamäki

, Nygård

C-H

. Effect of aerobic training on hot flushes and quality of life. Ann Med, 2011 Jun 3[Epub ahead of print]10.3109/07853890.2011.583674.

10.

Derogatis

. Misuse of the symptom checklist 90. Arch Gen Psychiatry, 1983; 40:1152–1153.

11.

Brooke

. SUS: A “quick and dirty usability” scale. Jordan

, Thomas

, Weerdmeester

, McClelland

. Usability evaluation in industry. Abingdon, UK: Taylor and Francis, 1996; 189–194.

12.

Bangor

, Kortum

, Miller

. Determining what individual SUS scores mean: Adding an adjective rating scale. J Usability Stud, 2009; 4:114–123.

13.

Tomlinson

, Solomon

, Singh

, Doherty

, Chopra

, Ijumba

et al. The use of mobile phones as a data collection tool: A report from a household survey in South Africa. BMC Med Inform Decis Mak, 2009; 9:51.

14.

Grady

, Macer

, Kristof

, Shen

, Tagliaferri

, Creasman

. Is a shorter hot flash diary just as good as a 7-day diary? Menopause, 2009; 16:932–936.

15.

Palmblad

, Tiplady

. Electronic diaries and questionnaires: Designing user interfaces that are easy for all patients to use. Qual Life Res, 2004; 13:1199–1207.