A usability study to test the effectiveness,efficiency and simplicity of a newly developed Internet-based Exercise-focused Health App for Lung cancer survivors (iEXHALE): Protocol paper

Abstract

The Internet-based Exercise-focused Health App for Lung cancer survivors (iEXHALE) is a mobile web app being developed to provide lung cancer survivors with an algorithm-based, tailor-made, self-management programme to inform their exercise choices and improve symptom severity. The aim of this protocol paper is to detail the plan for conducting the usability study to test the effectiveness, efficiency and simplicity of an exercise-focused self-management mobile web app for lung cancer survivors. The mixed methods study will consist of three consecutive phases, each interspersed with elements of data analysis and app prototype redevelopment. The study will take place in Oxford, United Kingdom. Ethical approvals have been obtained. The study will contribute to lung cancer survivorship research and is important in the app developmental process. This study contributes to the international forum for the exchange of practice, innovation and research, increases transparency in mobile health developmental processes and contributes to the methodological evidence base.

Keywords

assistive technologies electronic health evidence-based practice information technology design and development methodologies mobile health

Background

Lung cancer affects over 33,000 people per year in the United Kingdom, and many of these have significant smoking-related co-morbidities, which can negatively affect their activities of daily living and subsequent quality of life (QoL). Individuals may experience a range of curative procedures, including surgery, chemotherapy and radiotherapy while undergoing lung cancer treatment. These treatments, while potentially life-giving, can exacerbate symptoms of breathlessness and fatigue, both of which can be long lasting and debilitating.^1–3 Lung cancer survivors also commonly experience feelings of depression as a result of the enforced lifestyle changes that cancer brings, as they struggle to come to terms with their new normality in survivorship.^1–3 This coupled with the stigma associated with lung cancer,⁴ and the feelings of isolation and abandonment many cancer survivors experience at the end of their active cancer treatment, can have detrimental consequences on the mental and physical wellbeing of lung cancer patients.⁵

People undergoing active cancer treatment utilise a variety of self-management practices (exercise, diet, psychological therapies, complementary and alternative medicine and spirituality/religion) at different stages throughout their treatment pathway, from pre-diagnosis, through treatment and into survivorship.⁶ Research has shown that the most common form of self-management practice used post-cancer treatment is exercise,⁶ with people using it for a several reasons, including to regain health and fitness, mitigate side-effects from treatment, relax the mind and body and to regain a sense of normality post-cancer.⁷

A central tenet of self-management is improving self-efficacy.⁸ Self-management programmes, including those with an exercise component, have found a marked effect on self-efficacy, empowerment, decreased levels of fatigue, increased QoL, decreased depressive symptoms, improved coping mechanisms and changes in healthy behaviours.^8–10 Furthermore, the literature suggests that exercise practices can positively affect the health outcomes and QoL of cancer survivors.^11,12 A recent systematic review designed to examine the impact of exercise interventions in improving breathlessness, fatigue and depression in lung cancer survivors found evidence to suggest that exercise does improve these symptoms in this population group, although the quality of the evidence was mixed, suggesting that further research in this area is required.¹³

Increasing financial and clinical burdens on the National Health Service (NHS) in England have mounted in recent years, partly due to an ageing population, coupled with an increase in people living with long-term conditions, chronic illness and multiple co-morbidities.^14–16 As such there is a need to drive many of the services that are based in secondary care in to community based, primary care settings, to ease the pressure on acute hospital services. One way of doing this is by providing lung cancer survivors with the tools to self-manage certain aspects of their care, without the need for regular interventions from health care professionals. As well as relieving pressure on NHS services, exercise-based self-management interventions have the potential to provide tailor-made, individualised care to lung cancer survivors by enabling them to access local services, facilities and resources that are designed to relieve some of the physical, psychological, emotional and social burdens that cancer has caused.

The Internet continues to be an important resource for cancer patients and oncology professionals,¹⁷ and using mobile health (mHealth) technologies can be a useful way of encouraging patients to self-manage aspects of their own health and lifestyle.^18–24 Computer-based tools are commonly used in healthcare practice to inform and guide treatment decisions and lifestyle behaviours.^25–27 Electronic health (eHealth) and mHealth interventions have been identified as feasible approaches to providing supportive care, promoting behaviour change and managing specific symptoms affecting cancer survivors.^28–30

mHealth technology use is increasing, with self-monitoring mobile apps commonplace due to their ability to reach a large number of people.³¹ In 2016, seven billion mobile subscriptions were recorded worldwide with 90 per cent of people in developing countries and 96 per cent globally using mobile devices.²⁷ Flexibility, ease of access, usability, clarity and relevance of content, interactivity and tailorability have been highlighted as features of mHealth apps that are acceptable to users and increase engagement.^28,30,32 In addition, adopting a person- or user-centred approach by involving stakeholders and end users at various stages of the application (app) design process has been identified as a key component of the development and evaluation of digital healthcare interventions.^33–36

The Internet-based Exercise-focused Health App for Lung cancer survivors (iEXHALE), is a mobile web app that is currently being developed and will be designed to provide lung cancer survivors with an algorithm-based, tailor-made, self-management programme that can be used to inform their exercise choices with the aim of positively improving their symptom severity, QoL and self-efficacy. The app will provide information about types of exercise interventions that may be beneficial in reducing breathlessness, fatigue and/or depression. Table 1 shows the sections contained within the app.

Table 1.

Components of mobile application for lung cancer survivors.

Mobile application sections	Description of content
Introduction	Provides information about lung cancer, its treatment and common symptoms of fatigue, breathlessness and depression. Benefits of exercise for improving these symptoms are presented.
Self-rating symptoms	Users will rate their severity of breathlessness, fatigue and/or depression, perceived fitness levels and current exercise regime (if any), their level of social support and preferences for group or individual exercise activities. This information will then be fed into an algorithm which will generate the top three recommended exercises, presented in textual and audio-visual formats. Users can regenerate the algorithm at any time.
Self-monitoring diaries	Participants will rate their levels of breathlessness, fatigue and/or depression on a weekly basis. Participants will also be asked to indicate how many times they have done their recommended exercise activities each week, for how long, and to provide a subjective rating. Text and/or email reminders will be sent to participants.
Embedded analytics software	Embedded Google Analytics software will collect anonymised data to help understand usage. These analytics will measure which features are being used, and how. They can also be used to detect any erroneous behaviours or mistakes.

To ensure that the design features and content of the app are theoretically grounded in the principles of behaviour change and targeted at the needs and preferences of lung cancer survivors, the development of the app has been informed by a number of methods. First, a systematic review was carried out to explore the effectiveness of exercise-based interventions in improving fatigue, dyspnoea and depression in lung cancer survivors.¹³ Second, a focus group study (NHS Research Ethics Committee (REC) Reference: 17/LO/1576; Oxford Brookes Faculty Research Ethics Reference: 2006/58) was undertaken with lung cancer survivors, carers and health care professionals to explore what kind of content they would like an exercise-focused mobile web app to provide and to identify any potential barriers and facilitators to exercise that might exist for lung cancer survivors.³⁷ The current research adopted an integrative theoretical framework as a basis for understanding the target behaviour and designing the intervention. The findings from the systematic review and focus groups were then mapped against the COM-B (‘capability’, ‘opportunity’, ‘motivation’ and ‘behaviour’) behaviour change wheel.³⁶ The app is intended for use by lung cancer patients from the oncology outpatient setting with the long-term aim for it to be rolled out into mainstream clinical practice to enable lung cancer survivors to use the app in their own homes, to help them manage their symptom control.

Once the initial app prototype has been developed, the effectiveness, efficiency and simplicity of the app will be tested in a usability study with lung cancer survivors. Usability in eHealth interventions is associated with both usage and effectiveness,^38,39 and usability studies have been used to develop and refine eHealth and mHealth interventions targeting a range of health behaviours, including the self-management of symptoms following treatment for cancer.^32,40 There is a range of methods available to test the usability of digital healthcare interventions including interviews with users, think-aloud methods and participant observation.^36,41

Peer review and publication of research protocols has been identified as an important early stage step in reducing publication bias and improving the quality of research.^42,43 It has been argued that this may be particularly beneficial for smaller eHealth projects.⁴⁴ The purpose of this protocol paper is to detail the plan for conducting the usability study to test the effectiveness, efficiency and simplicity of an exercise-focused self-management mobile web app for lung cancer survivors in a comprehensive, transparent manner, making a contribution to the methodological evidence base in this important area.

Aims and objectives

The aim of the usability study is to test the effectiveness, efficiency and simplicity of a newly developed exercise-focused self-management mobile web app for lung cancer survivors.

The study objectives have been determined in an attempt to ensure correct learning and use of the mobile app by end users, without errors. The objectives will set out to determine the following:

The ease and effectiveness of participants’ navigation of the mobile app, measured by correct pathway selection and recognising icons;

The task success rate of participants;

The efficiency and safety of participants’ data entry;

The effectiveness of participants’ interpretation of the data that is presented by the app;

The effectiveness of the touch screen, measured by number entry and the responsivity of participants;

The simplicity of the mobile app, measured by the number and severity of errors made by participants.

Methods

Study design

The mixed methods study, using quantitative and qualitative methods, will consist of three consecutive phases (Table 2). Each phase will be interspersed with elements of data analysis and app prototype redevelopment. The three phases of the usability study were preceded by the development of Prototype 1 of the app as described earlier (Phase 0).

Table 2.

Usability study phases.

Phases of usability study	Description of phase
Phase 0	User centred development of the app which focused on user flow through the app in line with their needs, priorities and preferences
Phase 1	Users complete questionnaires and conduct tasks on app Prototype 1 Data analysis, resulting in recommendations being produced Redevelopment of app
Phase 2	Users repeat Phase 1 tasks on Prototype 2 Data analysis: results from Phases 1 and 2 are compared. Possible app redevelopment
Phase 3	Users take the app away for a specified period of time Semi-structured interviews with users Data analysis: qualitative interview data and quantitative analytics information

Setting, access and recruitment

The study will take place at Oxford Brookes University, Oxford, United Kingdom. Ethical approvals to undertake the study have been obtained (Oxford Brookes University Faculty Research Ethics Committee (FREC 2017/31); NHS Health Research Authority (HRA) Research Ethics Committee (Integrated Research Application System (IRAS) number: 239116)).

Access to study participants will be sought by contacting lung cancer survivors who have been treated at Oxford University Hospitals NHS Foundation Trust and who took part in the earlier focus group study.³⁷ Of these participants, those who consented to be re-contacted to consider participation in the usability study will be sent a participant information leaflet (PIL) in the post or via email to remind them about the study purpose. A member of the research team will then contact the participants 1 week later by telephone to ask them if they have any questions about the study and to see if they are willing to take part. For those participants who are happy to take part, a suitable date and time will be arranged for them to attend and participate in the usability study. At this meeting, participants will be provided with another copy of the PIL and will be asked to sign an informed consent form to participate in all three phases of the usability study. Participants will be informed that they have the right to withdraw from the study at any point.

In order to be eligible to participate in the usability study, participants must fulfil the following eligibility criteria:

Inclusion criteria

Participants must be aged over 18 years of age;

Participants must have completed active treatment (chemotherapy/surgery/radiotherapy) for lung cancer within the previous 6 months when they consented to the focus group study;

Participants must have been free from or have had stable disease at the time of consent to the focus group study;

Participants must be English speaking;

Able to provide written informed consent;

Participants must own or have access to a smartphone or alternative electronic device.

Exclusion criteria

Participants under the age of 18;

Participants receiving terminal care for cancer at the time of the focus group study;

Participants without access to a smartphone or alternative electronic device.

Sampling

A formal sample size calculation is not deemed necessary for this usability study, due to the small number of participants taking part. Rather, in line with published guidance on sampling and recruitment for usability studies, the sample size for the current study will be between 5 and 15 participants to maximise the expected level of problem discovery.⁴⁵ Up to 15 participants will be invited to participate in the study, but the exact numbers recruited will depend on the number of lung cancer participants who took place in the focus group study and who are still willing to be involved in the usability study.

Phase 1: task completion, evaluation and data analysis (Prototype 1)

Phase 1 of the study will incorporate a formative user-interface validation session (Figure 1). The researcher will use a standardised script as a guide with which to conduct one-to-one, 90-min sessions with participants, to ensure increased fidelity of the process.⁴⁶ The session will include welcoming participants and outlining the purpose of the session, providing participants with study information and asking them to provide written informed consent, participant completion of a pre-test survey, provision of brief system training by the researcher, usability test completion by the participant, completion of post-test survey and a brief, informal debrief to ask participants their views about the app and their experiences of the session.

Figure 1.

Sequence of steps required in Phase 1 formative user-interface validation session.

During the usability test component of the session, the researcher will give each participant a series of tasks to perform. These tasks have been developed by the research team and are based on the sequence of steps that lung cancer survivors would need to carry out if they were using the app in a non-experimental, real-life setting. Examples of tasks include accessing the section of the app that gives information about exercise interventions that might alleviate symptoms, filling out a weekly side-effect diary and switching on the voice-over feature of the app. Each task will be evaluated using quantitative metrics to assess the effectiveness, efficiency and simplicity of the app, as outlined in Table 3. Descriptive statistics will be used to summarise the data using a data management system such as SPSS.

Table 3.

Quantitative metrics for evaluating task scenarios.

Measurement	Attribute
% of tasks solved	Effectiveness
% of users able to complete a given task	Effectiveness
Number of attempts required to complete a given task	Effectiveness
Number of clicks/touches to solve task	Efficiency
Time taken to solve tasks	Efficiency
Number of errors per task	Simplicity
Severity of error (e.g. % error for numeric input)	Simplicity
Type of error (e.g. precision error – missed target, response error – user clicks multiple times, affordance error – wrong icon or incorrect gesture, mode error etc.)	Simplicity

The tasks will also be evaluated qualitatively using two methods. First, the ‘think aloud’ technique will be used to record the user’s experience.^36,47 Throughout the tasks, participants will be asked to behave as they normally would do and to verbalise what they are thinking and doing. These ‘think aloud’ behaviours will be digitally recorded, and the interaction of participants’ hands on the mobile devices will be video recorded without capturing any other body parts, so as to preserve the confidentiality and anonymity of participants. Second, the researcher will provide an opinion about the participant’s performance of each task using an assessment scheme (Table 4), along with an optional comment.⁴¹

Table 4.

Qualitative evaluation assessment scheme.

Assessment	Definition
0	User completed task with zero difficulty (zero frustration)
1	User completed task with only minor problem(s) (little frustration)
2	User completed task, but it required more effort/time/dead-ends than the user expected (medium/high frustration)
3	User did not complete task (point of failure)

Prior to carrying out these tasks, participants will be asked to complete a pre-test survey, which will collect some basic demographic information, together with information about their current smart phone usage and data from the Mobile Device Proficiency Questionnaire (MDPQ-16),⁴⁸ which has been specifically designed to assess older adults’ levels of mobile app proficiency. Following completion of the tasks, participants will be asked to complete a post-test survey, the validated ‘system usability scale’ to determine their perception of the apps usability.⁴⁹ In this context, the word ‘system’ will be replaced with the work ‘app’ in line with current guidance and best practice.⁴⁹

Google analytics will also be used within the interface to log any interactions and to collect anonymised data on app usage.⁵⁰ These analytics will be used to measure which features are being used, and how. They can also be used to detect any erroneous behaviours or errors from participants that might occur during this process.

Following analysis of the usability data collected in Phase 1, the app will go through a process of improvement for a period of 1 month before being re-tested with users in Phase 2. This app redevelopment will be undertaken in collaboration with the researchers and our industry collaborators and will consist of a brainstorming workshop to identify which key areas for development have emerged from the dataset most prominently. This workshop will be informed by the data findings from Phase 1.

Phase 2: task completion, evaluation and data analysis (Prototype 2)

Once Prototype 1 has been upgraded and revised as a result of the Phase 1 data findings, Phase 2 will commence. This will follow the same processes and procedures as Phase 1, with the omission of the initial pre-test survey. The same tests used in Phase 1 will be conducted on Prototype 2 of the mobile web app, and the same data analysis processes will be applied to ascertain whether there are any differences in usability between the two prototypes. Evaluation metrics will be compared across Prototypes 1 and 2, examining effectiveness, efficiency and simplicity. System usability scale scores will be normalised to measure and compare the performance between the two prototypes.

Once this has been established, if necessary, the app will be revised a second time for an additional 1-month period. However, if the results of the second usability test are good, there might not be a need for further modification. This decision will be made following discussions with industry collaborators, who have relevant expertise in this area.

Phase 3: interviews, app testing and data analytics analysis

Phase 3 of the study will consist of interviews with participants, app testing and a data analytics element. Study participants will be asked to take the newest version of the app prototype away for 2 weeks to use it in their own environments, as is intended in the longer term. After 2 weeks, participants will be invited to take part in individual, face-to-face semi-structured interviews with a study researcher. The interviews will be informed by a topic guide and will explore with participants their experiences of using the app in a real-life context. It will explore how useful they perceive the app to be and any perceived barriers and facilitators to using the app. It will also ask them if there are any features of the app which they particularly like or dislike, how easy or difficult it is to use and what they think about the content and design of the app. Interviews will be held in a room at Oxford Brookes University or in participants’ own homes, depending on individual participant preferences and will last approximately 45 min. All interviews will be digitally recorded and transcribed for analysis by a local transcription company. Any participant identifying data will be anonymised at the point of transcription.

Data collected during the think aloud tasks (Phases 1 and 2) and semi-structured interviews (Phase 3) will be transcribed and analysed thematically using the framework method.⁵⁵ A primarily deductive approach will be used to understand participants’ experiences of the content, design and functionality of the app, focusing on the data for any tasks that prove to be error-prone or low in completion scores. The data will be managed using Microsoft Excel. All transcripts will be coded, and a selection double coded by study researchers. Codes will then be grouped into categories, and similarities and differences in findings within and between participants’ data will be compared and contrasted, using the constant comparative approach. Themes emerging from the dataset will be generated, and key findings will be used to further inform the researchers’ knowledge about the usefulness and usability of the app. These findings will be used and compared with the results from the data analytics to inform any potential further modifications that may be made to the app prior to it being tested in a future, planned feasibility and acceptability study.

Study organisation and management

The usability study has been designed and developed in collaboration with colleagues in the United Kingdom and internationally, who have professional expertise in cancer survivorship, self-management, app development and usability processes, together with industry collaborators who are assisting with the app’s technical development. The conduct and progress of the study will be discussed and reviewed in study management meetings on a monthly basis for the 6-month study duration. The study management group will consist of the day-to-day research team, the chief investigator and national and international co-applicants and collaborators. Following the study closure, further publications relating to this study will be produced and findings will be presented at national and international conferences.

To ensure security, data obtained during the course of the study will be encrypted and stored securely, with access limited solely to the research team and industry collaborators who have provided a non-disclosure and confidentiality agreement. Any resulting publications using the data will not identify the participants, and any quotes will be anonymous. All data will be kept securely for a period of 10 years following completion of the project. All audio or video recordings will avoid the use of names or other easily identifiable statements. Any recordings that violate this will be edited to omit the statement and the original destroyed.

Discussion

Despite a growing body of literature focusing on self-management and survivorship issues for cancer patients,^51–53 lung cancer remains an area that is under studied compared to other cancer types, such a breast cancer.⁵⁴ As such, the usability study will make an important contribution to this field by focusing on the health needs of this population group. The app has the potential to help empower lung cancer survivors to self-manage specific aspects of their care pathways, improving their long-term health outcomes, wellbeing and QoL. The usability study is an important component in the app developmental process and will help ensure that the app is targeted on and focused around the needs of the lung cancer population group at which it is aimed and that the critical tasks are efficient and effective to execute.

Through the development of a tailored, population specific, mobile web app, lung cancer survivors will have a platform on which to aid their decision-making around the levels and types of exercise that they feel confident and able to undertake. The app itself will be distinct from other apps that have been developed for cancer survivors as it will focus on tailoring the intervention to the healthcare needs of the individual instead of adopting a ‘one-size-fits-all’ approach. It will also provide lung cancer survivors with a systematic way to record their exercise preferences and uptake, their symptom severity and their level of contact with health professionals, allowing them to observe and record any changes in these over time, monitor their progress and increase their awareness of any challenges they may face in the process. While, initially, the app will be designed to be used as an adjunct to patients’ interactions with clinicians to improve their physical and psychological health, the aim is that, in the longer term it may reduce overall contact with health professionals and NHS services as patients feel more confident to self-manage various aspects of their care, thus decreasing pressure on both NHS services and health professionals.

The usability study will be valuable in demonstrating the potential value of mobile web apps in improving the health and wellbeing of lung cancer survivors who are trying to regain health and fitness in the wake of their cancer diagnoses and treatments. It is hoped that the usability study will enable the successful development of a mobile web–based intervention which can be used to enhance the lives of this population group. This protocol paper makes a valuable contribution to the international forum for the exchange of practice, innovation and research, for both researchers, clinicians and telehealth specialists and makes the development process for an mHealth intervention of this kind more transparent as well as contributing to a broader methodological evidence base.

Once the app has been modified and enhanced in this usability study, it can be tested on a larger cohort of lung cancer survivors in a feasibility and acceptability study in the real-life setting. This will pave the way for a larger scale randomised controlled trial which will test the effectiveness of the app in improving symptom control, QoL and self-efficacy in lung cancer survivors – as well as measuring the cost-effectiveness of the app – with the long-term aim for the app to be rolled out into mainstream clinical practice.

Footnotes

Acknowledgements

The authors would like to thank industry collaborators, Oxford Computing Consultants, for their technical expertise in app development and the contribution of Matthew Standage, User Experience Design Lead, for the early drafts of this protocol. CH is a National Institute for Health Research (NIHR) Senior Nurse and Midwife Research Leader and is also supported by the NIHR Oxford cognitive health Clinical Research Facility. The views expressed are those of the authors and not necessarily those of the NIHR, UK National Health Service, or the UK Department of Health and Social Care.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by an Oxford Brookes University Research Excellence Award 2017–18.

ORCID iD

Catherine Henshall

References

Findley

Sambamoorthi

Preventive health services and lifestyle practices in cancer survivors: a population health investigation. J Cancer Surviv 2009; 3(1): 43–58.

Boon

Olatunde

Zick

SM.

Trends in complementary/alternative medicine use by breast cancer survivors: comparing survey data from 1998 and 2005. BMC Women’s Health 2007; 7: 4.

Sirois

Gick

ML.

An investigation of the health beliefs and motivations of complementary medicine clients. Soc Sci Med 2002; 55(6): 1025–1037.

Kendrick

‘Normalizing’ female cancer patients: look good, feel better and other image programs. Disabil Soc 2008; 23: 259–269.

Smithson

Paterson

Britten

, et al. Cancer patients’ experiences of using complementary therapies: polarization and integration. J Health Serv Res Policy 2010; 15: 54–61.

Shneerson

Taskila

Holder

, et al. Patterns of self-management practices undertaken by cancer survivors: variations in demographic factors. Eur J Cancer Care (Engl) 2015; 24(5): 683–694.

Henshall

Greenfield

Gale

The role of self-management practices as mechanisms for re-establishing normality in cancer survivors. Qual Health Res 2017; 27: 520–533.

Loh

S-Y

Ong

L-L

, et al. Qualitative experiences of breast cancer survivors on a self-management intervention: 2 year post-intervention. Asian Pac J Cancer Prev 2011; 12(6): 1489–1495.

Korstjens

May

van Weert

, et al. Quality of life after self-management cancer rehabilitation: a randomised controlled trial comparing physical and cognitive-behavioral training versus physical training. Psychosom Med 2008; 70: 422–429.

10.

Damush

Perkins

Miller

The implementation of an oncologist referred, exercise self-management program for older breast cancer survivors. Psychooncology 2006; 15(10): 884–890.

11.

Brown

Huedo-Medina

Pescatello

, et al. Efficacy of exercise interventions in modulating cancer-related fatigue among adult cancer survivors: a meta-analysis. Cancer Epidemiol Biomarkers Prev 2011; 20(1): 123–133.

12.

Speck

Courneya

Masse

, et al. An update of controlled physical activity trials in cancer survivors: a systematic review and meta-analysis. J Cancer Surviv 2010; 4(2): 87–100.

13.

Henshall

Allin

Aveyard

A systematic review and narrative synthesis to explore the effectiveness of exercise-based interventions in improving fatigue, dyspnoea and depression in lung cancer survivors. Cancer Nurs 2019; 42: 295–306.

14.

Osborn

Demoncada

Feuerstein

Psychosocial interventions for depression, anxiety and quality of life in cancer survivors: meta-analyses. Int J Psychiatry Med 2006; 36(1): 13–34.

15.

Wanless

The Wanless report: securing our future health: taking a long term view. London: HM Treasury, 2002.

16.

Hopkinson

JB.

How people with advanced cancer manage changing eating habits. J Adv Nurs 2007; 59(5): 454–462.

17.

Hesse

Greenberg

Rutten

LJ.

The role of Internet resources in clinical oncology: promises and challenges. Nat Rev Clin Oncol 2016; 13(12): 767–776.

18.

Vodopivec-Jamsek

de Jongh

Gurol-Urganci

, et al. Mobile phone messaging for preventive health care. Cochrane Database Syst Rev 2012; 12: CD007457.

19.

Shi

Mobile phone messaging for facilitating self-management of long-term illnesses. Int J Evid Based Healthc 2013; 11: 344–345.

20.

Marcano Belisario

Huckvale

Greenfield

, et al. Smartphone and tablet self management apps for asthma. Cochrane Database Syst Rev 2013; 11: CD010013.

21.

Brown-Johnson

Berrean

Cataldo

JK.

Development and usability evaluation of the mHealth Tool for Lung Cancer (mHealth TLC): a virtual world health game for lung cancer patients. Patient Educ Couns 2015; 98(4): 506–511.

22.

Nundy

Dick

Chou

, et al. Mobile phone diabetes project led to improved glycemic control and net savings for Chicago plan participants. Health Aff (Millwood) 2014; 33(2): 265–272.

23.

Nes

AAG

Eide

Kristjansdottir

, et al. Web-based, self-management enhancing interventions with e-diaries and personalized feedback for persons with chronic illness: a tale of three studies. Patient Educ Couns 2013; 93(3): 451–458.

24.

Payne Lister

West

Bernhardt

. Behavioral functionality of mobile apps in health interventions: a systematic review of the literature. JMIR Mhealth Uhealth 2015; 3(1): e20.

25.

Kanis

Hans

Cooper

, et al. Interpretation and use of FRAX in clinical practice. Osteoporos Int 2011; 22: 2395–2411.

26.

Snyder

Jensen

Courtin

, et al. Patient viewpoint: a website for patient-reported outcomes assessment. Qual Life Res 2009; 18: 793–800.

27.

Price

Dakin

Cook

, et al. ACHE – The Arthroplasty Candidacy Help Engine: using PROMs data to identify thresholds for referral in hip and knee replacement surgery. In: Advances in patient reported outcomes research, Oxford, 8 June 2017.

28.

Corbett

Singh

Payne

, et al. Understanding acceptability of and engagement with Web-based interventions aiming to improve quality of life in cancer survivors: a synthesis of current research. Psychooncology 2018; 27: 22–33.

29.

Cox

Lucas

Marcu

, et al. Cancer survivors’ experience with telehealth: a systematic review and thematic synthesis. J Med Internet Res 2017; 19(1): e11.

30.

Seiler

Klaas

Tröster

, et al. eHealth and mHealth interventions in the treatment of fatigued cancer survivors: a systematic review and meta-analysis. Psychooncology 2017; 26(9): 1239–1253.

31.

Voth

Oelke

Jung

ME.

A theory-based exercise app to enhance exercise adherence: a pilot study. JMIR Mhealth Uhealth 2016; 4(2): e62.

32.

Harder

Holroyd

Burkinshaw

, et al. A user-centred approach to developing bWell, a mobile app for arm and shoulder exercises after breast cancer treatment. J Cancer Surviv 2017; 11(6): 732–742.

33.

Van Gemert-Pijnen

Nijland

Van Limburg

, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res 2011; 13(4): e111.

34.

Yardley

Morrison

Bradbury

, et al. The person-based approach to intervention development: application to digital health-related behavior change interventions. J Med Internet Res 2015; 17(1): e30.

35.

West

Michie

A guide to development and evaluation of digital behaviour change interventions in healthcare. Sutton: Silverback Publishing, 2016.

36.

Michie

Yardley

West

, et al. Developing and evaluating digital interventions to promote behavior change in health and health care: recommendations resulting from an international workshop. J Med Internet Res 2017; 19(6): e232.

37.

Henshall

Davey

Development of an app for lung cancer survivors (iEXHALE) to increase exercise activity and improve symptoms of fatigue, breathlessness and depression. Psycho-Oncology. (unpublished).

38.

Eysenbach

The law of attrition. J Med Internet Res 2005; 7: e11.

39.

Chiu

Eysenbach

Stages of use: consideration, initiation, utilization, and outcomes of an internet-mediated intervention. BMC Med Inform Decis Mak 2010; 10: 73.

40.

Foster

Calman

Grimmett

, et al. Managing fatigue after cancer treatment: development of RESTORE, a web-based resource to support self-management. Psychooncology 2015; 24(8): 940–949.

41.

Cassano-Piché

Trbovich

Griffin

, et al. Human factors for health technology safety: evaluating and improving the use of health technology in the real world. Toronto, ON, Canada: Global Centre for eHealth Innovation, 2015.

42.

The PLoS Medicine Editors. Best practice in systematic reviews: the importance of protocols and registration. PLoS Med 2011; 8(2): e1001009.

43.

McNamee

Protocol reviews at the Lancet. Lancet 1997; 350: 6.

44.

Eysenbach

Peer review and publication of research protocols and proposals: a role for open access journals. J Med Internet Res 2004; 6(3): e37.

45.

Macefield

How to specify the participant group size for usability studies: a practitioner’s guide. J Usability Stud 2009; 5: 34–45.

46.

Krug

Rocket surgery made easy: the do-it-yourself guide to finding and fixing usability problems. Berkeley, CA: New Riders, 2010.

47.

Blandford

Furniss

Makri

Qualitative HCI research: going behind the scenes. San Rafael, CA: Morgan and Claypool, 2016.

48.

Roque

Boot

WR.

A new tool for assessing mobile device proficiency in older adults: the Mobile Device Proficiency Questionnaire. J Appl Gerontol 2018; 37: 131–156.

49.

Sauro

A practical guide to the system usability scale: background, benchmarks & best practices. Denver, CO: Measuring Usability LLC, 2011.

50.

Google Analytics Solutions. Turn insights into action. Menlo Park, CA: Google Analytics Solutions, 2018.

51.

Department of Health, Macmillan Cancer Support and NHS Improvement. The National Cancer Survivorship Initiative: vision 2010, https://webarchive.nationalarchives.gov.uk/20100809113601/http:/www.improvement.nhs.uk/cancer/LinkClick.aspx?fileticket=4apVUSvGcow%3D&tabid=214

52.

Richards

Corner

Maher

The National Cancer Survivorship Initiative: new and emerging evidence on the ongoing needs of cancer survivors. Br J Cancer 2011; 105(Suppl. 1): S1–S4.

53.

Department of Health. Cancer Reform Strategy, 2007, https://webarchive.nationalarchives.gov.uk/20130104165259/http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_081006

54.

Courneya

Physical activity in cancer survivors: a field in motion. Psychooncology 2009; 18(4): 337–342.

55.

Gale

Heath

Cameron

, et al. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol 2013; 13: 117.