Applying user-generated quality criteria to develop an Internet intervention for patients with heart disease

Abstract

Summary

Internet interventions can help people to self-manage chronic disease. However, they are only likely to be used if they meet patients' perceived needs. We have developed an Internet intervention in two stages to meet the needs of patients with coronary heart disease (CHD). First, user-generated criteria were applied to an existing US-based intervention called ‘CHESS Living with Heart Disease’ which provides information, emotional and social support, self-assessment and monitoring tools, and behavioural change support. This identifed the development work required. Then we conducted a user evaluation with a panel of five patients with CHD. Overall, users generally made positive comments about the information content. However they were critical of presentation, ease of navigation through the content, understanding what was offered in the different services and finding the information they were after. Applying user-generated quality criteria proved useful in developing an intervention to meet the needs of UK patients with CHD.

Introduction

Internet interventions that combine high quality health information with interactive components, such as self-assessment tools, behaviour change support, peer support or decision support, can benefit people with long-term conditions.¹ However patients are only likely to use such interventions, and experience benefits, if they meet their needs. Our earlier study² identified user-generated quality criteria for Internet interventions. Based on this, we have developed an Internet intervention in two stages to meet the needs of patients with coronary heart disease (CHD). First, the user-generated criteria were applied to assess the intervention and identify development work. Then we conducted a user evaluation. The two-stage process aimed to develop an intervention that was relevant to user needs and hence more likely to be used.

Methods

Development work was carried out on an existing intervention designed by the CHESS (Comprehensive Health Enhancement and Social Support) Team at the University of Wisconsin in the USA.³ The intervention ‘CHESS Living with Heart Disease’ provides information, emotional and social support, self-assessment and monitoring tools and behaviour change support.⁴ Development work was conducted in two stages.

Applying user-generated criteria

The intervention was assessed against the detailed user-generated quality criteria previously identified.² This identified elements of the intervention which required further development work. A multidisciplinary working group was set up to plan and carry out this development work. The working group was made up of two general practitioners (GPs), a consultant cardiologist, researchers specializing in health psychology, health services and cardiac nursing and a user representative. The working group was supported by two cardiology specialist registrars and a senior programmer from the CHESS team.

Development work was conducted by:

group discussion to decide the main principles for development including the extent of re-structuring, the extent and type of information included, and review progress;

individual work on different areas of information, and on different information and support services allocated according to expertise and available time;

review of each others' individual work. Clinicians reviewed and edited researchers' work, and reviewed each others' work for accuracy. Researchers reviewed and edited clinicians' work for readability and use of plain English;

technical work. The CHESS database software was installed, researchers were trained how to add, edit, format, link and re-structure text, documents and topics. Researchers then made changes to local files before sending updated files to the CHESS team to replace original files on their server. Content changes requiring more complicated programming work were carried out by the CHESS team.

Once this development work was complete, the new UK version of ‘CHESS Living with Heart Disease’ was run on a test server where three members of the working group (cardiologist, GP and user representative) performed a check for gaps, errors and inconsistencies.

User evaluation

The test version was further evaluated by a panel of patients with CHD. UK patients with heart disease were recruited by advertisements in local press and patient group newsletters, and from cardiac rehabilitation exercise classes to provide personal stories of experiences of CHD. Patients who provided personal stories and who had home access to the Internet were also asked if they were interested in participating in an evaluation panel. The user panel was made up of four men and one woman, aged 41–84 years. They did not know each other previously. The panel members were reimbursed for their time according to accepted guidelines.⁵ The study was approved by the appropriate ethics committee.

Members of the panel were shown how to use the test version of the intervention in their own homes, where they accessed it over a three-week period. They were asked to:

look for information that interested them;

use intervention services regularly during the three-week period;

record comments and suggestions in individual diaries provided. These allowed users to make as many comments and/or suggestions as they wished;

provide further evaluation and feedback through two focus group discussions, held after the first and third week.

The users' diaries were collected before each focus group to inform the focus group topic guide for discussion. The group discussions clarified comments and suggestions made in the diaries, and sought consensus for further development work. The test version of the intervention was projected onto a screen during focus group discussions to aid clarity.

Results

The outcome of applying the user-generated quality criteria to assess the intervention is shown in Table 1. Overall, users generally made positive comments about the information content. However they were critical of presentation, ease of navigation through the content, understanding what was offered in the different services and finding the information they required (Table 2). Discussions mainly concentrated on specific problems and suggestions for improvements. Recommendations for further development work identified by the user evaluation are shown in Table 3.

Table 1

Assessment of elements of the ‘CHESS Living with Heart Disease’ intervention by applying user-generated quality criteria²

User-generated quality criteria		Problems
Information content	UK focused information	Originally designed for US patients: – used US terms; – linked and referred to US resources; – contained information not relevant to or not correct for UK patients; – lacked information important for UK patients; – featured experiences of patients operating in US culture and health-care system
	Comprehensive and wide-ranging information	Insufficient detailed scientific and other information on:– surgery and other interventions;– rehabilitation and recovery
Presentation	Easy navigation	Had to use the topic structure, keyword and other searches to find information. No use of hyperlinks embedded in sections of text:– users could not move from one piece of information to another without going back to a menu page;– dictionary was stand-alone and could not be referred to while reading information
Up-to-date and accurate information		Unclear when information was last updated as no dates were displayed. In addition to inaccuracies introduced by US focus, the Internet intervention was originally designed in 1997 and was last updated in 2001 so featured:– out-of-date content;– information that did not refer to recent research studies;– links to other sites that were dead or no longer connected to relevant information
Manage the quantity and depth of information		Due to the lack of hyperlinks:– mostly flat information presentation;– unable to move from simpler information to more in-depth informationAlso information lists were in alphabetical order rather than simple first, then more in-depth
Links to other sites and trustworthiness		Featured extensive links to other websites but:– several topics where the intervention provided no information apart from external links;– unclear when users were leaving the Internet intervention and viewing information on an external website;– insufficient information provided about the source and content of information or resources hosted on external websites;– links often to US sites, organizations and resources not relevant to UK users, often commercial and/or featured advertising from sponsors

Table 2

Example comments from user evaluation focus groups

Relevant criteria	User comment
Information content – comprehensive and wide-ranging information	‘… you can choose your level of depth of information, so that you can go from [a] two-liner definition, to a news item, to a scientific journal, … on the same topic, and I thought that was excellent, because you could just pitch at your level of the kind of information that you're particularly seeking’
Manage the quantity and depth of information
Presentation – easy navigation	‘… what you've got there is a good tool for somebody suffering with cardiac problems to find out what they want, it just needs to be re-organized and tidied up so that it's easier…’

Table 3

Recommendations from user evaluation

Make the presentation more appealing by:– re-designing the home-page with simplified layout and clear, heart-focused images;– using more pictures to break up text

Make services more obvious from the start by:– using more intuitive names and grouping them under fewer headings on the homepage;– expanding the information guiding users about how to get the most out the intervention;– making services visually easier to distinguish from each other

Make information easier to find by:– showing all the main topics at a glance;– listing only the most relevant documents under any one topic;– improving the sensitivity of the search function;– improving the book-marking service to more easily return to information;– displaying dictionary definitions under topic names before listing relevant information

Make better use of the update section to flag new/updated information.

Discussion

Although the existing Internet intervention had been carefully designed, applying the user-generated quality criteria identified areas where it might not have met users' needs. The detail of the criteria helped to identify development tasks to improve the intervention. However, this work required substantial time and resources and would not have been possible without clinical expertise and technical support.

The comments and suggestions made by members of the user panel in evaluating the test version of the intervention echoed the user-generated quality criteria for information content and presentation. User evaluation was positive for information content but more critical of presentation and ease of navigation. For these criteria, the user evaluation provided important guidance for further refinement.

Applying user-generated quality criteria proved useful in developing an intervention to meet the needs of UK patients with CHD. Additional user evaluation was important in identifying specific development work required to meet some criteria. This approach may guide others developing Internet interventions, both in terms of process and cost. Continuing development of such interventions is necessary so that they continue to meet users' needs by remaining up to date and accurate, and user evaluations may be worth repeating periodically, depending on resources.

Footnotes

Acknowledgements

We are grateful to the BUPA Foundation Medical Research Charity for funding the work. We thank David Gustafson, Fiona McTavish, Haile Berhe and Gretta Wood, and the user evaluation panel.

References

Murray

, Burns

, See Tai

, Lai

, Nazareth

. Interactive Health Communication Applications for people with chronic disease. Cochrane Database Syst Rev 2005;(4):CD004274

Kerr

, Murray

, Stevenson

, Gore

, Nazareth

. Internet Interventions for long-term conditions: patient and caregiver quality criteria. J Med Internet Res 2006;8:e13

Gustafson

, Hawkins

, Boberg

, CHESS: 10 years of research and development in consumer health informatics for broad populations, including the underserved. Int J Med Inform 2002;65:169–77

Center for Health Enhancement Systems Studies. Pilot test of CHESS heart disease project. See http://www.chess.chsra.wisc.edu/chess/completed_studies/chess_heart_disease_pilot.aspx (last checked 15 February 2008)

Involve 2006. A guide to reimbursing and paying members of the public who are actively involved in research. See http://www.invo.org.uk/pdfs/Payment_Guidefinal240806.pdf (last checked 21 January 2008)