Implementing system-wide risk stratification approaches: A review of critical success and failure factors

Abstract

Risk stratification has become a widely used tool for linking people identified at risk of health deterioration to the most appropriate evidence-based care. This article systematically reviews recent literature to determine key factors that have been identified as critical enablers and/or barriers to successful implementation of risk stratification tools at a system level. A systematic search found 23 articles and four promising protocols for inclusion in the review, covering the use to 20 different risk stratification tools. These articles reported on only a small fraction of the risk stratification tools used in health systems; suggesting that while the development and statistical validation of risk stratification algorithms is widely reported, there has been little published evaluation of how they are implemented in real-world settings. Controlled studies provided some evidence that the use of risk stratification tools in combination with a care management plan offer patient benefits and that the use of a risk stratification tool to determine components of a care management plan may contribute to reductions in hospital readmissions, patient satisfaction and improved patient outcomes. Studies with the strongest focus on implementation used qualitative and case study methods. Among these, the literature converged on four key areas of implementation that were found to be critical for overcoming barriers to success: the engagement of clinicians and safeguarding equity, both of which address barriers of acceptance; the health system context to address administrative, political and system design barriers; and data management and integration to address logistical barriers.

Keywords

health services research health systems implementation risk stratification

Introduction

The growing burden of chronic illness poses some of the central problems of contemporary health policy and management in developed countries. Health systems designed around the problems of acute care have found the new problems difficult to manage. Policy approaches have included improved systems of integrated care, reforms to primary care to build better connections across the system and the use of disease management models. This new burden of disease is more than a simple set of medical and public health problems. It raises some fundamental questions for health administration, the allocation of resources and the focus of medical practice. Health professionals are reluctant to change established modes of practice and in fiscally constrained environments, shifts of resources to new problems threaten existing structures of power.^1,2

Predictive risk stratification has become a widely used approach for identifying people at risk of health deterioration or avoidable (and unexpected) hospitalisation.^3–5 The widely used Kaiser Permanente Triangle and similar disease management models identify levels of risk amongst people living with chronic illness, requiring very different levels of intervention.⁶ The apex of the Kaiser triangle represents a small fraction of patients, but those requiring the most intensive level of care and care coordination. Risk stratification instruments have been developed to identify the different levels of risk in a population and apply appropriate clinical and health management.

There is now a considerable literature evaluating the clinical value of different risk stratification instruments. Most is focused on the development and statistical validation of risk stratification algorithms. This research has concentrated on improving the power of the instruments – implementation gaps have been seen as a technical matter to be solved by improving algorithms.

Remarkably little work has been completed on broader management issues of the use of risk stratification tools – their use and acceptance by clinicians and the integration of the data produced into health administration. The use of algorithms takes at least some of the control over patients away from clinicians. As with many other areas of evidence-based health care, there is evidence of resistance from practitioners.^7,8

This study explores the major issues that arise in the implementation of risk stratification tools in health service and system delivery. Specifically, it seeks to determine: (1) how the implementation of risk stratification has been studied in the literature and (2) key factors that have been identified as critical enablers and/or barriers to successful implementation at a system level. We define “success” as the sustained uptake of a tool in a real-world setting over time with deliberate decisions to maintain the use of the tool due to its usefulness.

We use the term risk stratification tool to mean all models, tools and systems that use algorithms to predict future risk of health-service utilisation. These algorithms include variables and equations designed to protect against the over-simplification and inaccuracy of simple threshold models.^9,10 This emphasis on implementation means we say little about the technical content of the instruments and provide only limited information on the precise data required for each risk stratification tool and predictive accuracy. Reports on the development and validation of virtually all of the tools reviewed here can be found in the peer-reviewed literature.

We are aware of the existence of considerably more risk stratification tools than were found reported in the literature reviewed here. This suggests that while risk stratification tools have been developed and used widely, there has been little reported evaluation of how they are implemented in real world settings. The literature on the development and validation of risk stratification tools (e.g. for predictive accuracy) is considerably more abundant but outside of the scope of this review.

Method

We used a standardised approach to determine the breadth of the literature on the implementation of risk stratification tools, classify that literature and draw conclusions from their combined findings. To define the scope of the review in terms of the implementation of risk stratification tools we examined the spectrum of literature on risk stratification to determine the specific field of interest for this review. Papers that studied or described the adaption of a standard risk stratification tool for a new context or the implementation of a tool were of primary interest. Papers that focused on testing the predictive accuracy of a tool or the management of care following the use of the tool were only of interest if they also addressed adaptation or implementation. Papers that described care management following population risk stratification were only included if the use of the tool was sufficiently described as part of the intervention/case description (See Figure 1).

Figure 1.

Spectrum of literature on risk stratification and scope of review.

Our search strategy followed a two-pronged approach. First, a systematic search was conducted in Medline (via OvidSP), Embase, Scopus, the Cochrane Library and CINAHL databases for articles published between 2000 and 2016. Standard search terms are Box 1, and were adapted where necessary to each database.

Box 1.

Standard search terms.

Risk stratificat* OR Risk profil OR Population profil OR Population segment* OR Predictive risk OR Predict* model OR Risk predict* OR Risk Population* OR Risk model* OR Stratificat* strateg*

AND

Health service* OR Managed Care OR Integrated Care OR Primary Care OR Primary Health Care OR Aged Care OR Hospital OR Health System OR Population health

AND

Models OR Tools OR Program OR System

Second, focused searches were conducted in Medline, Embase and Google scholar for risk prediction models known to the authors. The total combined search results totaled 2222 citations. After removal of duplicates the total number of citations was 2207.

A title and abstract search eliminated 2148 references and a full-text assessment eliminated a further 32 papers using the following exclusion criteria that ensured we only included papers in the scope as shown in Figure 1. We also limited the review to risk stratification models used in system-wide rather than hospital-only settings.

Needs assessment or general potential applicability of risk stratification tools

Development of a tool

Validation of a tool/predictive accuracy testing

Care management following the use of risk stratification tools, but not the use of the tool itself

Risk-predictive tools used exclusively within the hospital setting “on the wards”

A total of 23 papers was found suitable for inclusion in the review, we also included four protocols of mixed-methods evaluations that showed considerable promise for expanding the knowledge in this area (Figure 2).

Figure 2.

Review of Implementation of Risk Stratification Approaches – PRISMA Flowchart. Generated online with prisma.thetacollaborative.ca.

The breadth of methods and study designs found in the reviewed literature meant that a quantitative meta-analysis of results was not possible. Instead, we first classified each of the included papers by study type and drew information from articles within each category to determine relevance for the main objectives of the review. This information is reproduced in the results section by study type as classified in Table 1. We then undertook a meta-synthesis (interpretive integration of the results of a number of different studies)^11,12 to identify common core elements and draw practical conclusions for health systems and policy planners. The discussion is thus divided into two sections. The first draws conclusions from the results to answer objective 1 of the review – how the implementation of risk stratification has been studied in the literature. The second part of the discussion uses meta-synthesis to identify key factors that have been identified as critical enablers and/or barriers to successful implementation at a system level.

Table 1.

Classification of articles included in review.

Type	Papers	Protocols
Impact studies (using control, multiple baseline or interrupted time-series designs)	9	2
Qualitative evaluations	4	1
Descriptive studies
Comparative case studies	3	1
Descriptive case studies	7
Total	23	4

Results

Studies of the implementation of risk stratification tools found in the literature can be grouped into three broad study-type categories. First, we found a total of nine studies that aimed to measure the impact of implementing risk stratification on population health outcomes, such as decreases in symptoms, illness, risk factors and health service usage and mortality using some form of controlled design. Within this group, we found a total of seven studies using a matched or randomised control group, one interrupted time-series evaluation and one multiple baseline evaluation design. We label this group “impact studies.” Second, we found four qualitative evaluations that were primarily aimed at determining reach, acceptability and practical use of risk stratification tools amongst users. Third, we found a total of ten descriptive studies including three comparative case studies and seven descriptive case studies, the purpose of which was to analyse a range of measures of success in implementation, including uptake, reach, acceptability, usability and sustainability.

The implementation of a total of 20 different risk stratification tools we documented in this literature. The tools varied in terms of origin, input data and applied setting as documented in Table 2.

Table 2.

Risk stratification tool included in the reviewed literature.

Risk stratification tool	Studies	Developer/origin	Input data and implementation
Case Smart Suite Germany (CSSG)	Freund²⁴	Commercial developer. Verisk Health, Munich, Germany.	This tool may be purchased for use “off-the-shelf” by health-care providers and insurers. The algorithm used is similar to that of diagnostic cost groups. Inputs include ICD-10-German Modification (GM) diagnosis codes assigned in outpatient and inpatient settings, prior costs, hospital admissions and demographic data. Clinically similar ICD-10-GM codes are classified into diagnostic groups which are collapsed into diagnostic categories. Generic models for adaptation in other countries are also available for purchase.
Combined Predictive Risk Model (CPRM)	Lewis³⁹; Lewis⁸; Lewis²⁷	Publicly developed tool (Kings Fund UK and Health Dialog). Now de-commissioned.	An algorithm for predicting re-hospitalisation in the next 12 months intended for use by Primary Care Trusts and other NHS organisations in the UK where both primary and secondary data are available. Available for use by NHS organisations as a stand-alone string code; requiring the local build of a user interface. Allows segmentation of an entire NHS population (all patients registered with a GP) into relative risk segments.
Community Assessment Risk Screen (CARS)	Doñate-Martínez¹⁹	Jointly developed by Carle Clinic Association, Urbana, Illinois, U.S. and University of Wisconsin-Madison.	This tool uses three variables to predict future hospitalizations: (1) pre-existing chronic diseases; (2) the number of prescription medications and (3) hospitalizations or ED use in the preceding 6–12 months. The score (0–9) is accumulative depending on the number or risk factors present. Data are obtained by medical staff directly from patients and the algorithm applied. In Valencia, the tool was adapted for use with administrative data.
Diabetes Population Risk Tool (DPortT)	Rosella³⁶	Public tool developed by Canadian Institutes of Health Research and the Population Health Improvement Research Network	Calculates the future risk of diabetes, for diabetes-free individuals. Uses publicly available national population health surveys administered by Statistics Canada (Canadian Community Health Survey). Publicly available for download directly into SAS statistics software or as a formula. Can be used to predict cases or attribute the contribution of specific risk factors (included in the algorithm) to population risk.
Devon Combined Predictive Model	Lewis²⁷; Lewis⁸	Adaptation of CPRM tool developed by NHS Devon, Devon, UK.	Predicts unplanned admission to hospital or an emergency re-admission in the following 12 months. This adaptation of CPRM added seven local factors as variables including length of registration with GP.
FINDRISC (Finnish Diabetes Risk Score)	Badenbroek³⁷	Publicly available tool developed in the Diabetes Prevention Unit, Department of Chronic Disease Prevention, National Institute for Health and Welfare, Helsinki, Finland	Questionnaire style risk stratification tool available for use or adaptation online. Assesses an individual’s risk of developing type 2 diabetes stratified as low, slightly elevated, moderate, high and very high. Included variables are: age, BMI, waist circumference, physical activity levels, consumption of vegetables, fruits or berries, high blood pressure requiring treatment, previous high blood glucose and family history.
Framingham Risk Score (FRS)	Jones³³	Developed as part of the Framingham Heart Study, Boston University. Algorithm publicly available.	The updated version of this algorithm (from 2002) uses eight variables to assess risk of developing cardiovascular disease in the next 10 years. Variable thresholds are calculated differently for men and women. The tool is open source and may be integrated into clinical decision support tools, other multi-faceted risk prediction tools or completed online for real time results using one of several user interfaces available online. Has been shown to overestimate risk when applied to patients in European settings.
Hierarchical Condition Category (HCC)	Boult¹⁶	Developed by and for the Centres for Medicare and Medicaid Services, USA.	Measures the burden of 70 disease categories which are correlated to diagnosis codes. Introduced in 2004 in the Medicare and Medicaid systems as the basis for capitation and reimbursements. The HCC for each patient is captured every 12 months and forms the basis of payments for the following 12 months.
Johns Hopkins University Adjusted Clinical Groups	Cohen³²; Arce²³	Developed at Johns Hopkins University with commercial licence rights.	Software package available for US or international licence (currently available Version 9). Uses various inputs that can be adjusted according to setting, such as: age, gender, total disease burden, medical conditions, population markers, resource use and medications. Available as a stand-alone product or a part of a service delivery package and electronic medical record administration.
Joint Asia Diabetes Evaluation (JADE) Risk Engine	Jiao¹³	Privately developed tool: Asia Diabetes Foundation and the Chinese University of Hong Kong.	A risk stratification tool that forms part of a web-based portal of care protocols, clinical decision and self-management support. Patients consent to enrollment in to the program from which point medical data are carried within the portal. A yearly health assessment is carried out and data entered into the portal which is cross-matched with administrative data to measure risk of 5-year probability of major clinical events. The full program is accessed by GPs through a secure web portal and key patient data are available for viewing at care appointments.
LACE	Dhalla¹⁸, Lewis²⁷; Tuso²¹	Publicly developed in Ontario, Canada.	Data inputs are length of stay (“L”); acuity of the admission (“A”); comorbidity of the patient (measured with the Charlson Comorbidity Index score) (“C”); and emergency department use (measured as the number of visits in the six months before admission) (“E”). Intended to be administered within the hospital at the point of discharge.
Nairn Case Finder	Baker¹⁵	Public developer. NHS Highland Health Board	Tool originally developed for Lodgehill Clinic in Nairn and measures risk of an unplanned admission to hospital in the subsequent 12 months. Primary care data are taken from the country-wide GP medical records system “General Practice Administration System for Scotland” (since changed for use with current system “GP Vision”). Primary care variables include age, sex and chronic disease status. Secondary care data were taken from the NHS Highland Patient Administration System and include outpatient attendance and unplanned admission to hospital in the previous 2 years. The tool was run monthly and GPs were provided with lists of at risk patients.
PARR (Patients at Risk of Readmission)	Lewis²⁷; Lewis⁸	Publicly developed tool (Kings Fund UK and Health Dialog). Now de-commissioned.	Public risk stratification tool intended to be used by Primary Care Trusts in the UK. Produces a patient “risk score” showing a patient’s likelihood of re-hospitalisation within the next 12 months. Risk scores range from 0 to 100, with 100 being the highest risk. PARR1 uses data on prior hospitalisations for certain “reference conditions” to predict risk of re-hospitalisation while PARR2 uses data on any prior hospitalisation to predict risk of re-hospitalisation. Further iterations of PARR (including PARR-30) were developed. The tool originally did not come with an in-built user interface, although two have been developed (PARR + and PARR ++.)
PRISM	Hutchings³⁵; Kingston²⁵	Public tool commissioned by NHS Wales Informatics Service to King’s Fund and Health Dialog.	Uses 22 variables from GP systems, 8 from hospital inpatient record, 3 demographic variables, data of outpatient visits following ED visits and the Welch Index of Multiple Deprivation to identify likelihood of an emergency hospital admission over the next 12 months. Both absolute risk (four risk levels based on percentage risk score) and relative risk (four risk levels based on risk score relative to the practice population) are measured. Care providers register for access and use Prism through a password-protected website. End users can view population level trends, view patient risk data by entering an NHS number) or filter populations by risk level or other criteria.
Probability of Readmission (Pra)	Donate-Martinez¹⁹; Boult¹⁶	Developed at University of Minnesota. Johns Hopkins University holds exclusive rights from the University of Minnesota to sublicense to others.	Estimates probability of hospital readmission within 4 years. Inputs include age, gender, poor self-rated general health, availability of an informal caregiver, having ever had coronary artery disease, having had diabetes during the previous year, a hospital admission during the previous year, more than six doctor visits during the previous year. A more recent version of the tool (PraPlus) also includes questions about medical conditions, functional ability, living circumstances, nutrition and depression. Widely used in the USA. Use of the instruments must be licenced.
Systemic coronary risk evaluation (SCORE)	Zachariadou³¹	Developed by a consortium of researchers for European Society of Cardiology funded by European Union BIOMED program	Developed in response to studies finding over-estimation of risk for CVD when tools developed in the US were applied in European settings. Comprises paper-based risk charts for high-risk and low-risk European populations; national or regional risk charts based on published mortality data and a computer-based interface “Heartscore” for risk estimation data entry and calculation. The publicly available website includes a pro forma for calculating patients’ risk, management advice and allows clinicians to save patient data (once registered with the site). A downloadable version is available.
Scottish Patients at Risk of Readmission and Admission (SPARRA)	NHS Scotland²⁶	Public developer, commissioned by Scottish Government Information Services Division to Health Dialog, UK.	SPARRA scores risk of admission in the prediction year and can be accessed securely online by authorised health care professionals in NHS Scotland Boards, Community Health Partnerships and GP practices. Three iterations of this tool have been developed. Version 1 stratified population >65 years, version 2 extended this to whole-of-population and version 3 includes new prescription data input. The algorithm is based on hospital inpatient admissions; community dispensed prescriptions; Emergency Department (ED) attendances; new outpatient attendances; and psychiatric inpatient admissions. Colour-coded data visualisation is available.
A purposefully developed risk stratification tool for the Indiana Chronic Disease Management Program (here ICDMP tool)	Katz²⁰; Rosenman³⁰	Commissioned by Indiana Medicaid to vendor Regenstrief Institute	Used to assign participants to different program services (nurse care managers to highest risk 20%; telephone care coordinators to remainder 80%). Algorithm based on 2 years of retrospective claims data (1) total net Medicaid claims in past 12 months; (2) Medicaid aid category (e.g. “aged” or “disabled”); (3) total number of unique medications filled in past year.
A purposefully developed risk stratification tool based on the American Diabetes Association Clinical Practice (here ADACP tool)	Clark¹⁷	Commissioned by Las Vegas Managed Care Organisation to Roche Diagnostics Corporation.	Uses laboratory tests and data from completed patient questionnaires to generate risk profiles (high-, moderate- or low-risk) groups in seven categories: (1) glycemic control, (2) cardiovascular disease, (3) nephropathy, (4) retinopathy, (5) hyper/hypoglycemia, (6) amputation and (7) psychosocial disorders. Data are entered and retrieved from a web-based interface.

Group 1 – Impact studies

Of the total of nine studies that measured the impact of implementing risk stratification tools on population health outcomes, only two studies^13,14 used a control group that did not receive any risk stratification. A study of the Joint Asia Diabetes Evaluation (JADE) Risk Engine¹³ randomly selected and matched patients with diabetes under the care of GPs in Hong Kong to a usual care group before implementing risk stratification and care management strategies according to risk score profile (n = 1248). At 12-month follow-up, the intervention group had a significant net decrease in HbA1c, cardiovascular event incidence and stroke risk. In a study of the StarT Back Tool for treating lower back pain, physical therapists were randomly selected to be trained in the use of stratified care. Patients receiving StarT Back stratified treatment from the trained therapists (n = 67) had greater improvements using a numerical pain-rating scale compared to patients receiving standard care.¹⁴

In the remaining seven studies that measured the impact of implementing risk stratification tools on health outcomes, both the intervention and usual care groups were stratified using the adopted tool and only the managed care program after stratification comprised the intervention.^15–21 For example, in Nairn, Scotland, patients from two clinical practices were identified as having a high risk of unplanned hospital admission using the Nairn Case Finder.¹⁵ Intervention practice patients then received an Anticipatory Care Plan. Mortality rates in the two cohorts were similar, but hospital bed days used in the last three months of life were significantly lower in the intervention group. In Minnesota, USA, Medicare beneficiaries aged 70 years and older were stratified using a self-completed Probability of Readmission (Pra) instrument survey and intervention group patients received an interdisciplinary care package.¹⁹ Mortality, use of health care services and total Medicare payments did not differ significantly between the two groups. However, follow-up interviews found that patients in the intervention group were less likely to lose functional ability.

The LACE tool was adopted across 13 Kaiser Permanente Southern California medical centres, resulting in a reduction on re-admission rates.²¹ The same tool was administered at discharge in four hospitals in Toronto after which patients were randomly allocated to either a Virtual Ward or usual care.¹⁸ In this study, there was no statistically significant difference between the groups on 30-day, 60-day, 6-month or 1-year readmission. Similar results were found in a study of the use of the Hierarchical Condition Category (HCC) in Baltimore and Washington DC, USA, although that study did show significantly higher participant ratings of the quality of care.¹⁶ Improvement in patient satisfaction was also the most significant outcome of the evaluation of the adoption of a purposefully developed risk stratification tool based on the American Diabetes Association Clinical Practice recommendations (ADACP tool) in a Las Vegas–managed care organisation.¹⁷ The use of another purposefully built tool as part of the Indiana Chronic Disease Management Program (ICDMP tool) was found to contribute to a flattening of cost trends.²⁰

While we only included studies that provided some information on context and implementation of the risk stratification tool, this was not the main subject of investigation in the impact studies described here. This diminished relevance, i.e. “indirectness,”²² meant that these studies offer only a limited understanding of what contributes to successful implementation of risk stratification tools in real-world settings and critical enablers and barriers. The controlled and longitudinal studies described above offer no conclusive evidence of the benefits or limitations of implementing risk stratification tools in real-world situations. However, the use of risk stratification tools in combination with a care management plan may offer some patient outcome benefits, including increased patient experience satisfaction. The use of a risk stratification tool to determine components of a care management plan may contribute to reductions in hospital readmissions, health service use and improved patient outcomes.

Group 2 – Qualitative evaluations

The four qualitative evaluations of the implementation of risk stratification tools aimed to provide specific insights into factors influencing successful implementation of risk stratification tools by researching the experiences of users.^23–26 They primarily reported on different types of user surveys, including focus groups, questionnaires and interviews.

In a study in the Basque Country, Spain, three focus groups were conducted exploring clinicians’ opinions and experiences related to the use of an adapted risk stratification tool based on the Johns Hopkins University Adjusted Clinical Groups (ACG) model.²³ The study identified several enablers and challenges to implementation including with high importance the need to frame the implementation of a new risk stratification tool within a wider strategy. In another study in Munich, Germany, primary care physicians that had used the Case Smart Suite Germany (CSSG) risk stratification tool,²⁴ were asked to compare CSSG with their clinical judgement and rate their experience of its use. Overall, the physicians rated the approach as a useful tool to identify patients likely to benefit from case management. However, they were concerned about time lags between data analysis and patient recruitment. A third qualitative study reported on focus groups and interviews undertaken with staff in 13 GP practices taking part in the demonstrator testing of PRISM, a purpose-built risk stratification tool for Demonstrator Sites for the Wales NHS Chronic Disease Management Program.²⁵ The study found that first impressions of PRISM were mixed and often developed following further exposure to the tool. Finally, NHS Scotland’s Information Services Division that developed and has carriage of the Scottish Patients at Risk of Readmission and Admission (SPARRA) tool undertook a qualitative survey of tool users at Community Health Partnerships (CHPs), Health Board and GP level.²⁶ The study found that end users were interpreting SPARRA data correctly and making suitable adjustments when necessary, but end users would prefer more options to be able to manipulate output data, such filters or highlighters.

Group 3 – Descriptive studies

In the three comparative case studies identified, differences in the implementation of a risk stratification tool were compared across localities and considered for possible effects on differences in uptake, acceptance, sustainability and outcomes.

In one comparative study, three adaptations of the “Virtual Wards” program in Croydon, Devon and Wandsworth UK that used stratification tools to determine catchment areas for Virtual Wards and select patients for admission were analysed.⁸ The Combined Predictive Risk Model was used in Croydon. An adapted version with a new user interface was created for use in Devon (henceforth the Devon Combined Predictive Model) and the PARR model used Wandsworth. The three cases described above were also included in a comparative review of six managed care programs including North Somerset UK (using no risk stratification tool) Toronto Canada (using the LACE tool) and New York City USA (who commissioned the build of their own model using Medicaid data model for their Hospital2Home scheme).²⁷ The third comparative study outlined basic concepts of predictive modelling, described some of the models that have been developed internationally and compared cases studies from within the Spanish National Health Service.²⁸

The seven descriptive case studies found included a case study of the Virtual Ward models²⁹ and the ICDMP tool in Indiana that are outlined above.³⁰ Other descriptive case studies included:

the application of the systemic coronary risk evaluation (SCORE) tool to risk stratify 1011 patients with diagnosis type-two diabetes mellitus hypertension or hyperlipidaemia living in Cyprus,³¹

Clalit Health Services’ (Israel’s largest managed care organisation) adaption of the Johns Hopkins University Adjusted Clinical Groups (ACG) risk model in for implementation to select patients for a multimorbid care management program,³²

Use of Pra and Community Assessment Risk Screen (CARS) to detect patients at risk of hospital readmission in a sample of 500 elderly people (65+) in Valencia, Spain,¹⁹

a feasibility test of the use of the Framingham Risk Score (FRS) in Pennsylvania, USA,³³ to risk-stratify patients and involve them in shared decision making

the use of the Finnish Diabetes Risk Schor (FINDRISC) tool across multiple settings in Europe.³⁴

We also note that that evidence in this area is scattered, yet rapidly emerging. We found protocols of four high-potential trials of the implementation of risk stratification tools that are due for reporting within the next 12 months; all of which intend to take a comprehensive mixed-methods approach to examining a broad range of aspects related to the implementation of risk stratification tools closely aligned to the objectives of this review. The PRISMATIC trial is currently underway that will evaluate the implementation of the PRISM risk stratification tool throughout Wales, UK.³⁵ The Diabetes Population Risk Tool (DPortT) predicts 9-year risk for diabetes and is being implemented in Ontario and Manitoba in Canada.³⁶ The planned evaluation will assess the effectiveness and impact of a proposed Knowledge-to-Action framework for facilitating the implementation of the tool. The INTEGRATE study³⁷ will assess the use of the Finnish Diabetes Risk Score (FINDRISC) tool; and a European-wide project Activation of Stratification Strategies and Results of the interventions on frail patients of Healthcare Services (ASSEHS) has been established to assess the use of existing health risk stratification strategies and tools throughout Europe.³

Discussion

How the implementation of risk stratification has been studied in the literature

The nine evaluation studies that measured the impact of implementing a risk stratification tool against population health outcomes only partially focussed on the effect of the risk stratification tool itself. The outcomes ranged from reductions in the incidence of severe cardiovascular events, improved management of lower back pain through to falls in avoidable hospitalisations across a broader range of chronic illnesses.^13–15 These studies do provide evidence that the use of risk stratification tools in combination with a care management plan may offer some patient outcome benefits and that the use of a risk stratification tool to determine components of a care management plan may contribute to reductions in hospital readmissions, health service use and improved patient outcomes. However, there is equivocal evidence to suggest that the use of a risk stratification tool solely for determining eligibility for managed care has a positive effect on patient outcomes. Those (few) studies which attempted to measure system level outcomes have made no attempt to separate the effects of risk prediction and the actual intervention. Lessons from these studies therefore need to be interpreted carefully. Positive outcomes in the intervention group indicate benefits of implementing a managed care program that includes the use of a risk stratification tool, but cannot attribute results to either the care package or risk stratification tool alone.

Evidence from qualitative studies and descriptive case studies identify a range of factors that contribute to successful implementation. Despite a weaker study design they provide the most promising evidence for barriers and enablers to successful implementation of risk stratification tools in the real world. These studies converged on general themes when it comes to successful implementation of risk stratification tools. They are therefore heavily drawn upon in the discussion on critical enablers and barriers below. These case studies offered the richest insights into the range of factors that may enable or facilitate successful implementation of risk stratification tools in terms of sustained uptake of a tool in a real-world setting over time with deliberate decisions to maintain the use of the tool due to its usefulness.

Critical enablers and/or barriers to successful implementation

Studies in this review with the strongest focus on implementation used qualitative and case study methods. We identify four key areas that were found to be critical in implementation in the reviewed literature; the engagement of clinicians, the health system context, data management and integration and concerns over equity.

Engagement of clinicians

As noted above, the use of a risk stratification tool containing an algorithm does, by design, take some decision-making autonomy away from clinicians. Thus, acceptance by clinicians of the use of the tool is a major barrier that needs to be overcome. The Basque Country study,²³ which looked at a population level adoption of the Johns Hopkins University Adjusted Clinical Groups, used qualitative methods to describe the engagement of clinicians. Those who had some training or practical background in the management of health at a population level were the easiest to engage. The support of their colleagues who had not worked outside clinical practice was harder to win, and usually required considerable education in population health approaches.

Clinicians were also more likely to use the tool if they were given some independence to access and use data from the tool to improve their own practice. This was a persistent theme. In Clarke’s study of the ADACP tool,¹⁷ stratification data were prepared in a form that patients could read, and was used as a method of improving health literacy. In the JADE controlled trial in Hong Kong,¹³ GPs could access this patient information with a portal that linked risk profiles to decision support tools and care guidelines following the recommendations of the International Diabetes Federation. GPs in the PRISM study²⁵ were encouraged to continually compare their own understandings and expectations of patients’ risk scores.

The study of risk stratification in primary care using CSSG in Munich, Germany, argued that acceptance among patients and primary care providers was higher if case finding involved some judgement by the clinicians. Risk stratification helped counter a personal sympathy/aversion element that biased doctor’s judgements of which patients to admit to a new program. However, risk stratification on its own lacked an important capacity to judge patients’ “willingness and ability to participate” and “manageable care needs.”²⁴

This factor became a barrier to the take-up of PARR in Virtual Wards in Croydon Primary Care Trust in London.²⁹ GPs resisted the selection of patients purely on the predictive risk model, and even asked to have a right to select which were admitted to treatment. The largest challenge to use of PARR remained a perception of referrals “from a computer.”

Alignment with broader health system contexts

Key to the successful introduction of new instruments in the Basque Country was seen as its location within a clearly articulated broader strategy with two-way communication between planners and health care providers.^17,23 While the technical task of linking primary care, hospital and other data made the implementation of risk stratification feasible, it was noted in the comparative review of Spanish cases that risk stratification should be developed in parallel with other initiatives, such as care integration and the re-organisation of clinician work time with regard to time spent on tasks such as case management.²⁸

A comparison of three English case studies of “Virtual Wards,” a model of integrated primary and social care,⁸ saw the wider operating environment as the main condition that enabled successful implementation of risk stratification tools. These elements included the organisational culture, the existence of multidisciplinary teams and active patient participation.

The Croydon Virtual Wards model was launched in 2006 in a national health policy climate that encouraged this type of intervention and especially the use of predictive tools for case finding. It received strong support at managerial level, from the Primary Care Trust and local medical committee, including access to GP data managed by the Trust, which fed into the Combined Predictive Model. The weakness of the Croydon model lay in its detachment from general practitioners. The model of case management was one-on-one by a matron, with no role for case management by a multidisciplinary team including GPs. As a result, the care plans were based on informal collaboration between matrons and GPs, plans that were often not documented and did not draw directly on risk modelling. In these circumstances, as the program matured there was a steady regression from multidisciplinary case management, using the Combined Predictive Model back to traditional care.

In contrast, a model in Devon was more rooted in primary care, championed by a GP and only taken up by the Primary Care Trust after his advocacy. The Devon model, struck some real problems, but these were mainly related to the bureaucratic structures, including perverse financial incentives for hospitals to admit more patients, which undermined one of the main objectives of the program.

A third model, in Wandsworth, also had considerable initial support from general practice. Its choice of risk prediction model strengthened this GP support, but at the expense of an effective risk predictive system. Wandsworth used the PARR model which throws a smaller net, only looking for patients with a prior hospitalisation. With fewer at-risk patients identified, it relied on GPs for referrals. As a result, it remained more popular with local GPs, who could refer their difficult-to-manage patients.

Data management and integration

Studies of clinician take-up³¹ emphasised the need for reliable, up-to-date data. This key factor relates to logistical barriers to the smooth functioning of the risk stratification tool. The Basque study¹⁷ found that clinicians wanted to be able to access and use the data independently, with usable information, social as well as strictly medical, at the group as well as the individual level.

A regular theme was the need for risk stratification tools and data to be usable in other contexts. The Framingham Risk Score, for example, is based on historical cohort of population, whose characteristics and needs differed from contemporary primary care populations. Attempts to modify its formula were found to be “sub-optimal.”³³

The evaluation of the implementation of PRISM³⁸ found complexity and difficulties in signing up and unforeseen incompatibilities in computer systems were major barriers to early take-up. The PARR model, which is based on recent hospitalisations, was easier to use, but had limited usefulness for the general population.²⁹ The more sophisticated Combined Predictive Risk Model, which can deal with broader populations, needed to be adapted to local circumstances, which made it costlier and time-intensive to implement.

The comparative study of “Virtual Wards” found that the Devon version of the CPRM, which had started with solid foundations in primary care, faced its worst difficulties with issues of data management, especially information governance.⁸ Major problems arose in extraction of data from GP systems for predictive modelling and with the system for transferring information to GPs to give their patients’ predictive risk scores. Most obstacles came from data protection and other legislative and administrative safeguards.

Equity issues

Most issues of equity came from the design of the instrument and other data issues. Equity is an important factor as it helps shape acceptance by clinicians, patients as well as the broader population. Equity problems can make it harder for elected officials and health service administrators to use an instrument to justify shifting resources to a narrower, high-morbidity group. The failings of highly targeted “impactability” models were discussed in the literature in terms of these ethical perceptions. These models take the results of a more standard predictive model and try to predict the sub-groups of these at-risk patients who are most likely to respond to case management. There is evidence that they are superior for identifying patients with complex but manageable co-morbidities.³⁹ The Croydon “Virtual Wards” study²⁹ rejected this approach on equity grounds, as the measure of predicted success is likely to exclude patients with substance abuse, mental illness or other disadvantages.

Conclusion

While risk stratification tools have been developed and used widely, there have been few studies of how they are implemented in real-world settings. Lessons for future implementation need to draw on information from a range of study types, including incidental findings from impact studies and descriptive studies. Every study involving primary care, especially general practice, saw the engagement of clinicians as the key to success. These ranged from studies of risk stratification within primary care^24,29 to the more integrated Scottish and Basque health systems.^23,26 These distinctions between drivers of successful implementation crossed system boundaries and were the one generic predictor of successful adoption.

Importantly, studies also found direct benefits for patients in having access to the results of risk stratification. This has often taken the form of web-based, user-friendly portals, often linked to evidence-based trusted decision tools offering appropriate guidance for the particular risks faced by a patient. In these instances, risk stratification tools are a supplement to, not a replacement of, clinical judgement.⁴⁰

Some of the trials currently underway may provide better answers to the broader effects of risk stratification and help improve implementation. The PRISM trial³⁵ is looking at the costs of implementation and the cost-effectiveness of the instrument – questions that no other study in this review has broached. It will also measure changes in the profile of the services provided to patients and levels of patient satisfaction. An early study of the Welsh PRISM Chronic Care Demonstration project²⁵ reported that first responses to the tool were “mixed” but found that user involvement (again from GPs) in developing improved versions of the tool helped reverse initial failures. Resistance from GPs to the risk modelling associated with the Croyden “Virtual Ward” was considered to be at least partly due to the novelty of the predictive risk model as a concept.²⁹

Importantly, where clinicians had easy, user-friendly access to data concerning their own patients, there was greater acceptance of risk stratification. This was especially true where the stratified data were linked to clinical guidelines to suggest directions for treatment. Furthermore, health care practitioners, especially in primary care, were more likely to embrace new methods of case finding if they were consulted at every stage. If they could see a clear benefit to their own patients, they were much more prepared to make some of the changes in practice required and less likely to see risk stratification tools as an attack on clinical judgement.

Footnotes

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Andrew Wilson is a member of the Board of the NSW Agency for Clinical Innovation.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Research undertaken to inform this article was supported by the NSW Agency for Clinical Innovation and the Sax Institute, NSW, Australia.

References

Weisz

. Chronic disease in the twentieth century: a history, Baltimore: Johns Hopkins University Press, 2014.

Fox

. Power and illness: the failure and future of American health policy, Berkeley: University of California Press, 1993.

de Manuel Keenoy

David

Mora

. Activation of stratification strategies and results of the interventions on frail patients of healthcare services (ASSEHS) DG Sanco Project No. 2013 12 04. Eur Geriatr Med 2014; 5: 342–346.

Dixon

Lewis

Rosen

. Managing Chronic Disease: What can we learn from the US experience?, London: The King's Fund, 2004.

Panattoni

Vaithianathan

Ashton

. Predictive risk modelling in health: options for New Zealand and Australia. Australian Health Review 2011; 35: 45–51.

Debbie S and Chris H. Improving Care for People with Long Term Conditions: A review of UK and international frameworks. NHS Institute for Innovation and Improvement, 2006.

Dopson

Fitzgerald

. Knowledge to action: evidence-based health care in context, Oxford: Oxford University Press, 2005.

Lewis

Vaithianathan

Wright

. Integrating care for high-risk patients in England using the virtual ward model: lessons in the process of care integration from three case sites. Int J Integr Care 2013; 13: e046–e046.

Georghiou

Blunt

Stevenson

. Predictive risk and healthcare: an overview, London: Nuffield Trust, 2011.

10.

Lewis

Curry

Bardley

. Choosing a predictive risk model: a guide for commissioners in England, London: Nuffied Trust, 2011.

11.

Finfgeld

. Metasynthesis: the state of the art—so far. Qualitat Health Res 2003; 13: 893–904.

12.

Walsh

Downe

. Meta-synthesis method for qualitative research: a literature review. J Adv Nurs 2005; 50: 204–211.

13.

Jiao

Fung

Wong

. Effects of the multidisciplinary risk assessment and management program for patients with diabetes mellitus (RAMP-DM) on biomedical outcomes, observed cardiovascular events and cardiovascular risks in primary care: a longitudinal comparative study. Cardiovasc 2014; 13: 127–127.

14.

Beneciuk

George

. Pragmatic implementation of a stratified primary care model for low back pain management in outpatient physical therapy settings: two-phase, sequential preliminary study. Phys Ther 2015; 95: 1120–1134.

15.

Baker

Leak

Ritchie

. Anticipatory care planning and integration: a primary care pilot study aimed at reducing unplanned hospitalisation. Br J Gen Pract 2012; 62: e113–e120.

16.

Boult

Leff

Boyd

. A matched-pair cluster-randomized trial of guided care for high-risk older patients. J Gen Intern Med 2013; 28: 612–621.

17.

Clark

Jr Snyder

Meek

. A systematic approach to risk stratification and intervention within a managed care environment improves diabetes outcomes and patient satisfaction. Diabetes Care 2001; 24: 1079–1086.

18.

Dhalla

O'Brien

Morra

. Effect of a postdischarge virtual ward on readmission or death for high-risk patients: a randomized clinical trial. JAMA 2014; 312: 1305–1312.

19.

Doñate-Martínez

Garces Ferrer

Rodenas Rigla

. Application of screening tools to detect risk of hospital readmission in elderly patients in Valencian Healthcare System (VHS) (Spain). Arch Gerontol Geriatrics 2014; 59: 408–414.

20.

Katz

Holmes

Stump

. The Indiana Chronic Disease Management Program's impact on medicaid claims: a longitudinal, statewide evaluation. Med Care 2009; 47: 154–160.

21.

Tuso

Huynh

Garofalo

. The Readmission Reduction Program of Kaiser Permanente Southern California—knowledge transfer and performance improvement. Permanente J 2013; 17: 58–63.

22.

Guyatt

Oxman

Kunz

. GRADE guidelines: 8. Rating the quality of evidence–indirectness. J Clin Epidemiol 2011; 64: 1303–1310.

23.

Arce

De Ormijana

Orueta

. A qualitative study on clinicians' perceptions about the implementation of a population risk stratification tool in primary care practice of the Basque health service. BMC Fam Pract 2014; 15: 150–150.

24.

Freund

Wensing

Geissler

. Primary care physicians' experiences with case finding for practice-based care management. Am J Managed Care 2012; 18: e155–e161.

25.

Kingston

. Initial uses of the PRISM risk stratification tool in CCM Demonstrator sites: a qualitative study, Wales: Chronic Conditions Management Demonstrators, NHS Wales, 2010.

26.

National Health Service Scotland. Investigating the use of SPARRA. Phase 1 – How and to what extent is SPARRA being used? ISD Scotland, 2011.

27.

Lewis

Wright

Vaithianathan

. Multidisciplinary case management for patients at high risk of hospitalization: comparison of virtual ward models in the United Kingdom, United States, and Canada. Popul Health Manage 2012; 15: 315–321.

28.

Nuno-Solinis R. Development and implementation of risk stratification tools: practical tools to identify patients with complex needs. O+berri Working Paper. Basque Foundation for Health Research and Innovation, 2013.

29.

Lewis G. Predictive modelling in action: how ‘Virtual Wards' help high risk patients receive hospital care at home. Issues in International Health Policy, The Commonwealth Fund, 2010.

30.

Rosenman

Holmes

Ackermann

. The Indiana Chronic Disease Management Program. The Milbank Quarterly 2006; 84: 135–163.

31.

Zachariadou

Stoffers

Christophi

. Implementing the European guidelines for cardiovascular disease prevention in the primary care setting in Cyprus: lessons learned from a health care services study. BMC Health Serv Res 2008; 8: 148–148.

32.

Cohen

Flaks-Manov

Low

. High-risk case identification for use in comprehensive complex care management. Pop Health Manage 2015; 18: 15–22.

33.

Jones

Shah

Bruce

. Meaningful use in practice using patient-specific risk in an electronic health record for shared decision making. Am J Prev Med 2011; 40: S179–S186.

34.

Cos

Barengo

Costa

. Screening for people with abnormal glucose metabolism in the European DE-PLAN project. Diabetes Res Clin Pract 2015; 109: 149–156.

35.

Hutchings

Evans

Fitzsimmons

. Predictive risk stratification model: a progressive cluster-randomised trial in chronic conditions management (PRISMATIC) research protocol. Trials 2013; 14: 301–301.

36.

Rosella

Peirson

Bornbaum

. Supporting collaborative use of the diabetes population risk tool (DPoRT) in health-related practice: a multiple case study research protocol. Implement Sci 2014; 9: 35–35.

37.

Badenbroek

Stol

Nielen

. Design of the INTEGRATE study: effectiveness and cost-effectiveness of a cardiometabolic risk assessment and treatment program integrated in primary care. BMC Fam Pract 2014; 15: 90–90.

38.

Kingston

Evans

. Introduction of a predictive risk tool in primary care: context and expectations. Emer Med J 2015; 32: e2–3.

39.

Lewis

. “Impactibility models”: identifying the subgroup of high-risk patients most amenable to hospital-avoidance programs. Milbank Q 2010; 88: 240–255.

40.

National Health Service England. Using case finding and risk stratification: a key service component for personalised care and support planning, England: NHS, 2015.