Development,content validity and inter-rater reliability testing of the Chronic Illness Management Implementation – Building Research Initiative Group: Chronic Illness Management and Adherence in Transplantation: An instrument to assess the level of chronic illness management implemented in solid organ transplant programmes

Abstract

Introduction

In this article we report the item generation, validity and inter-rater reliability testing of the Chronic Illness Management Implementation – Building Research Initiative Group: Chronic Illness Management and Adherence in Transplantation instrument.

Methods

The instrument was developed based on the World Health Organization’s Innovative Care for Chronic Conditions framework by the research team in consultation with a chronic illness management expert and a focus group of transplant nurses. Seven chronic illness management experts were used to assist in calculating the content validity index for each item and the total instrument. A pilot study was conducted in two transplant programmes to examine inter-rater reliability. Percent agreement by professionals working within the same centre was calculated by averaging the percent agreement on individual items.

Results

The instrument content validity index was 87%. The percentage agreement in both transplant programmes was 84.6% and 74.8%, indicating good inter-rater reliability. The final instrument consists of 55 items.

Discussion

The results are promising, suggesting its value as a tool to measure implementation of chronic illness management in transplant programmes.

Keywords

Chronic disease transplantation questionnaire development practice patterns coordination

Introduction

Over the past decades, 1-year patient and graft survival after solid organ transplantation improved significantly.¹ Advancements in immunosuppressive medications and better surgical techniques are the main drivers for improved short-term outcomes. However, this progress is not reflected in long-term outcomes, where survival rates remain largely unchanged.^1,2 One issue in transplantation is that post-transplant patients are not cured; yet move from one chronic illness state (i.e. end-stage organ disease) to another (transplantation and related co-morbidities). Transplant recipients indeed remain dependent on lifelong follow-up and must engage in health behaviours (e.g. taking medication, following diet restriction, limiting or stopping alcohol consumption, and being sufficiently active) for the rest of their lives to ensure favourable outcomes. This is not only required for their transplant but also for the multiple other co-morbidities that may carry over from pre-transplant or develop post-transplant.

Evidence in other chronically ill populations shows that patients benefit if their care is organized based on the principles of chronic illness management (CIM).³ These principles consist of a combination of continuity and coordination of care; partnerships with patients, families and communities; support for patients in improving their self-management (i.e. the actions that patients perform in daily life to manage their illness and treatment and to avoid functional or health deterioration);^4–6 attention to preventive measures; decision making support and the availability of clinical information systems.^4,5,7,8 Wagner⁹ developed a framework integrating the principles of CIM into the chronic care model (CCM). The World Health Organization (WHO) subsequently adapted the CCM to provide a global perspective by emphasizing community and policy aspects in their Innovative Care for Chronic Conditions (ICCC) framework (Figure 1).¹⁰ This framework can help decision makers or those who are working in the healthcare system to progressively create or redesign healthcare organizations and systems to expand their capacity to manage long-term health problems.^4,11 It is based on a set of guiding principles acting at three levels.^4,8 The micro level emphasizes the partnership between patients and families and healthcare professionals. The meso level refers to the healthcare organization and the community. The macro level refers to policy and financial contexts.^4,8,11

Figure 1.

Innovative care for chronic conditions framework.⁴

Recent systematic reviews in chronically ill patient populations demonstrated the value of implementing components of the CCM: it has led to a reduction in hospital admissions;^12,13 a reduced risk of mortality;^12,13 increased quality of life;^12,14 improvements in health behaviours (e.g. physical activity, dietary habits, medication adherence),^13,15,16 reductions in HbA1c, Low-density lipoprotein (LDL) cholesterol and arterial blood pressure;^13,14 increased satisfaction of patients, their informal and professional caregivers;^13,15 and lower healthcare expenditures.¹⁷ No studies in transplantation have assessed if and to which degree care of this patient population is organized based on the principles of CIM. Importantly one intervention study using a quasi-experimental design tested the effectiveness of a CIM programme in kidney transplant recipients in Canada on process and clinical outcomes.¹⁸ All kidney transplant patients with chronic kidney disease received care at an advanced practice nurse-led inter-professional collaborative chronic care clinic. The patient interventions were based on the CCM. More specifically, following components of the CCM were integrated: reinforcement of disease self-management, shared decision making, adherence enhancing interventions and community resources. Significantly more patients in the intervention group reached the targeted clinical outcomes (e.g. calcium, carbon dioxide, low density lipoprotein and parathyroid hormone targets reached), had fewer emergency room visits and hospital admissions, and had a discussion about end-stage treatment options relative to the control group, suggesting the value of this model for transplantation.¹⁸

Improving the level of CIM implemented in a healthcare organization requires adequate assessment. The level of CIM implemented in healthcare organizations can be assessed either from the patient perspective¹⁹ or from the perspective of the healthcare provider.²⁰ The Patient Assessment of Chronic Illness Care (PACIC) questionnaire was developed to assess the implementation of the CCM from the patient perspective.¹⁹ To assess the level of CIM through healthcare provider information, Bonomi et al.²⁰ developed the Assessment of Chronic Illness Care (ACIC) questionnaire.

Although the psychometric properties of the PACIC are acceptable, those of the ACIC were substandard. There were poor correlations between ACIC scores and chronic disease experts’ ratings of team performance (r = 0.28–0.52) challenging its validity and thus also its use in research and clinical practice.²⁰ The ACIC was specifically developed for the US setting, limiting its generalizability for other parts of the worlds. Neither the ACIC nor the PACIC was specifically developed for transplant settings.

Based on these shortcomings we developed an instrument to assess the dimensions of CIM implemented in organ transplant programmes (meso level) from the healthcare providers’ perspective. In this article we report on the development, content validity and inter-rater reliability of the Chronic Illness Management Implementation – Building Research Initiative Group: Chronic Illness Management and Adherence in Transplantation (CIMI-BRIGHT) instrument.

Methods

The development of the CIMI-BRIGHT questionnaire is theoretically embedded in the ICCC model and was conducted in different phases. First, items were generated (Phase 1: Item generation). The initial instrument was then reviewed by a focus group interview consisting of seven transplant nurses (Phase 2: validity – evidence based on test content, focus group review). The resulting draft version of the instrument was then used to assess the content validity index (CVI) of the instrument (Phase 2: Validity – evidence based on test content, CVI review). The instrument was subsequently revised integrating the results of the CVI review. This revised version was used for the third phase on Inter-rater reliability testing. The process utilized in the development and testing of the CIMI-BRIGHT instrument is illustrated in Figure 2.

Figure 2.

Overview of subsequent phases of development of the CIMI-BRIGHT instrument.

Phase 1: Item generation

Methods

For the initial item generation, the ICCC model was used as a guiding framework developed for worldwide application. The meso/healthcare organization level of the ICCC consists of five building blocks:⁴ First, promotion of continuity and coordination, which refers to the need of chronically ill patients for services that are coordinated across levels of care (i.e. primary, secondary and tertiary care) and across providers. Second, encouragement of quality care through leadership and incentives; senior and other influential leaders need to lend clear support and sponsorship for improving the care of chronic conditions in their healthcare organization. Third, organization and equipment of healthcare teams by providing them with the skills and knowledge (e.g. effective communication abilities) to manage chronic conditions. Fourth, support of self-management and prevention; healthcare professionals need to have competencies concerning self-management strategies and know how to educate patients and families about self-management. Finally, the last building block, the use of information systems, means that healthcare organizations must provide timely information about individual patients and populations of patients. Information systems are needed to gather and organize data about epidemiology, treatment and healthcare outcomes. Furthermore, information systems are essential to assure continuity and coordination of care. The goal is to use information systems to improve planning and the general standard of care.⁴

Furthermore, the initial item generation was also based on a review of the CIM literature (the literature search was conducted using the following keywords: chronic illness management, chronic disease management, care continuity, care coordination, quality of care, delivery of health, self-management support, self-care support, centre effects, skill mix, healthcare system, practice patterns, prevention support and information systems) and the transplantation-related clinical expertise of the research team. The items were examined for their fit with the ICCC framework. Items were deleted, adapted or generated to ensure that each building block was represented in the initial draft of the instrument. After the initial list of items was generated, the instrument was reviewed by an internationally recognized expert in CIM. This expert, who was part of the WHO team that developed the ICCC framework, evaluated the extent to which the items reflected and adequately represented the building blocks of the ICCC model. Based on the expert’s feedback the instrument was adapted by adding or removing items.

Phase 2: Validity testing – evidence based on test content

The framework used to validate this instrument was the Standards for Educational and Psychological testing proposed by the American Educational Research Association (AERA), American Psychological Association and the National Council on Measurement in Education.²¹ In this study, we tested one aspect of validity empirically, i.e. the test content of the CIMI-BRIGHT instrument. In the AERA standards, test content is described as ‘the themes, wording, and format of items, tasks or questions on a test, as well as the guidelines for procedures regarding administration and scoring’.²¹ Test content was obtained in two consecutive stages: during the initial stage, the completeness of the instrument, the clarity of the items and the clarity of the response options were evaluated using a focus group interview with seven international transplant nurses, all native English speaking (Phase 2: Validity – evidence based on test content, focus group review). Feedback received during this interview was integrated into the instrument prior to testing content validity with an interdisciplinary group of international (Europe and US) experts in CIM (Phase 2: Validity – evidence based on test content, CVI review). We defined an expert in CIM as a person who was recognized in the chronic illness scholarly world as evidenced by at least two publications or presentations on CIM. Two authors (LB and SE) searched on the world wide web for experts; a third author (SDG) approved the choices. The group was composed of nurses, physicians and health policy experts.

Measurement

For each individual item, the experts were asked to rate the relevance of each item on a 4-point Likert scale (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant and 4 = highly relevant) in relation to the construct ‘CIM’.²² Furthermore, the experts were encouraged to write comments (i.e. an explanation for their choice, suggestions to improve) for each item.

Data collection

The identified experts were contacted by email and asked to participate in this study. The CIMI-BRIGHT instrument and the Content Validity Form were mailed to those agreeing to participate. During a scheduled phone call, the background, aims and methodology of the study were briefly described by the first author (LB) and information on the purpose and use of the Content Validity Form was provided. Participating experts were asked to return the completed Content Validity Form by either electronic mail or regular mail. If the Content Validity Form was not returned within 1 month, the participants received a reminder email. If they did not respond to the reminder, they were considered non-responders. Data were collected in Autumn 2009.

Data analysis

To evaluate the agreement between experts concerning the relevance of single items and the entire CIMI-BRIGHT instrument, CVIs were calculated. CVIs were calculated for each item (content validity for items, I-CVI) and the instrument as a whole (CVI for scales, S-CVI/Ave). The I-CVI was computed as the number of experts giving a rating of either 3 (quite relevant) or 4 (highly relevant) divided by the total number of experts.²² To evaluate the content validity on the scale level, the I-CVIs were summed and divided by the number of items.²² An instrument is considered to have excellent content validity if it is composed of items with a I-CVI of 0.78 or higher and the S-CVI/Ave is 0.90 or higher.²² The written comments of the reviewers were collected for each item and taken into consideration when adapting the questionnaire.

Phase 3: Inter-rater reliability testing

To determine inter-rater reliability of the CIMI-BRIGHT instrument, we conducted a pilot study in two transplant programmes. For the purpose of this study we tested reliability as equivalence, more specifically inter-rater (or inter-observer) reliability, which represents the agreement among raters.

Sample and setting

The pilot study of the CIMI-BRIGHT instrument was conducted in one abdominal and one renal transplant programme in the eastern part of the US; both programmes were recruited through convenience sampling. Healthcare professionals were eligible to participate if they were working in the transplant programme for longer than 6 months, were employed 50% or more in clinical practice and had knowledge about the content of care provided in the outpatient transplant programme. No minimum number of participating clinicians per centre was determined a priori. Student nurses, nursing assistants and float pool nurses were excluded from participation. All healthcare professionals who met the inclusion criteria within each transplant programme were invited to complete the CIMI-BRIGHT instrument.

Measurement

Following content validity testing and revision based on the results, the final version of the CIMI-BRIGHT instrument was utilized for inter-rater reliability testing.

Data collection

The abdominal and kidney transplant programmes were identified and contacted by a member of the research team. A contact person at each transplant programme was identified. Data collection for the pilot study was conducted in the transplant programmes between March and December 2010. Healthcare professionals were surveyed voluntarily and anonymously. The CIMI-BRIGHT instruments were distributed by the contact person at the respective transplant programme, who stressed the importance of completing the questionnaires independently of the healthcare professionals. To ensure individual responses the questionnaires were collected in a secured box located at each of the participating transplant programmes. The contact person of each transplant programme mailed the completed instruments to the principal investigator (LB).

Ethical considerations

Approval was obtained from each transplant programme’s ethical committee prior to data collection. Informed consent of healthcare professionals was implied by the completion and return of the CIMI-BRIGHT instruments.

Data analysis

The percentage agreement between the participating healthcare professionals in the two transplant programmes was calculated per item and for the total CIMI-BRIGHT instrument. Due to the small sample size and few answers in some of the response options, responses were collapsed into: (1) strongly disagree and disagree; (2) agree and strongly agree; (3) don’t know, not applicable or missing. After collapsing the data, for each item the proportion of participants who selected the same answer was calculated. To calculate the overall percentage agreement, all items were added and divided by the number of items. An overall percentage agreement between the healthcare professionals of >60% was interpreted as indicating substantial inter-rater reliability, with 80% or more indicating almost perfect agreement.²³

Results

Phase 1: Item generation

The first draft of the CIMI-BRIGHT instrument, developed by the research team, consisted of 47 items. The CIM expert suggested deleting 12 items based on item redundancy, having no relationship with the concept of CIM and/or its building blocks. In addition, it was recommended to add 16 items in order to more fully capture the ICCC building blocks. The final initial draft of the CIMI-BRIGHT instrument consisted of 51 items. This draft was reviewed during a focus group interview.

Phase 2: Validity testing – evidence based on test content

Focus group interview

The seven international transplant nurses recommended rewording several of the items but did not add or delete any. The revised version of the CIMI-BRIGHT instrument consisted of 51 items covering the five ICCC building blocks at the healthcare organization level: (1) promote continuity and coordination (14 items), (2) encourage quality through leadership and incentives (six items), (3) organize and equip healthcare teams (seven items), (4) support self-management and prevention (19 items) and (5) use of information systems (five items). This draft was forwarded to the second group of experts for content validity testing.

CVI. A group of 11 experts in CIM (six from USA and five from Europe) were identified and asked to participate in the content validity evaluation of the CIMI-BRIGHT. Seven of them agreed to participate (three from USA and four from Europe), which corresponds with a response rate of 64%. One of the seven experts only rated part of the instrument. Her responses were included for the items she rated.

Forty-two out of the 51 items were rated as having good content validity with CVIs ranging from 0.83 (22 items) to 1.00 (15 items). The S-CVI/Ave was 0.83. Nine items had low content validity. Six items were revised and three items were deleted, as there was too much overlap with other items. With the deletion on the three items, the S-CVI/Ave was 0.87. Based on the feedback of the experts testing the content validity, we added seven items to the instrument (either new items or items that were divided into separate items).

After integrating the results of the content validity testing, the third version of the CIMI-BRIGHT instrument consisted of 55 items, namely 12 for promoting continuity and coordination, seven for assessing encouragement of quality through leadership and incentives, eight items for assessing organization and equipping healthcare teams, 19 items for assessing supporting self-management and prevention, and nine items that inquired about the use of information systems. The items in the CIMI-BRIGHT instrument are presented in Table 1.

Table 1.

The CIMI-BRIGHT instrument.

	Please circle the number that best represents the status of your transplant team and programme		Strongly disagree	Disagree	Agree	Strongly agree	Don’t know	Item CVI
Provide continuity and coordination	1	Inpatient and outpatient services are coordinated. For example, the outpatient transplant programme can arrange examinations while the patient is in the inpatient clinic and vice versa	1	2	3	4	5	0.50
	2	Services are coordinated across different healthcare providers. Each provider has access to the patient’s complete medical records	1	2	3	4	5	0.67
	3	Services are coordinated over time	1	2	3	4	5	0.67
	4	There is a specific long-term care coordinator who monitors and directs patients’ care, making sure that all involved healthcare workers are working together across settings and over time	1	2	3	4	5	0.83
	5	Patients are strongly encouraged (but not forced) to see the same healthcare workers over time	1	2	3	4	5	0.83
	6	Patients are assessed (face to face or by telephone) when they have concerns/questions outside of their normal appointments	1	2	3	4	5	0.83
	7	Outpatient follow-up appointments are scheduled with patients at the end of each visit	1	2	3	4	5	0.83
	8	Patients who cancel their follow-up visits are contacted to reschedule the missed appointment	1	2	3	4	5	1.00
	9	Patients who cancel their follow-up visit are contacted to identify and solve any problems that might prevent them from making their visits	1	2	3	4	5	Split
	10	Patients have access to their medical records or personal health records	1	2	3	4	5	New
	11	Electronic medical records are available to all members of the transplant team (including outpatient and inpatient clinicians, nurses, pharmacists and physician specialists)	1	2	3	4	5	0.83
	12	Electronic medical records are used by all members of the transplant team	1	2	3	4	5	Split
Encourage quality through leadership and incentives	13	Healthcare workers are given incentives (e.g. recognition or financial rewards) for effective clinical processes and/or health outcomes	1	2	3	4	5	0.83
	14	Patients are given incentives (e.g. recognition or financial awards) for effective self-management^a and/or health outcomes	1	2	3	4	5	0.83
	15	Senior and other influential leaders clearly help improve the quality of care in our transplant programme	1	2	3	4	5	0.83
	16	The transplant programme has a system for routinely monitoring the quality of care	1	2	3	4	5	0.67
	17	The transplant programme has a highly organized quality improvement process, focusing on a small number of important problems or conditions over a long enough time to be useful	1	2	3	4	5	0.50
	18	All members of the transplant programme’s team routinely take part in ongoing quality improvement processes	1	2	3	4	5	0.83
	19	The achievements of quality improvement processes are reported to the public	1	2	3	4	5	0.83
Organize and equip healthcare teams	20	Written guidelines for care are easily available	1	2	3	4	5	0.83
	21	Written guidelines for care are supported by education/courses	1	2	3	4	5	Split
	22	Written guidelines for care are built into care through prompts/reminders or computerized systems	1	2	3	4	5	Split
	23	Healthcare workers have enough time to assess, educate and support patients in self-management^a in view of, for instance, medication taking, smoking, physical activity, sun protection or following a diet	1	2	3	4	5	1.00
	24	The transplant programme is based on a system of interdisciplinary team care (not run by physicians alone)	1	2	3	4	5	0.83
	25	Team members are given roles and responsibilities for tasks according to their professional strengths and abilities	1	2	3	4	5	0.83
	26	Clinicians’ time is freed by shifting routine jobs to clinical assistants (patient care technicians, unlicensed assistive personnel, etc.) and/or administrative staff when possible	1	2	3	4	5	0.83
	27	Each member of the healthcare team is respected for his/her unique areas of expertise and contributions to overall patient care	1	2	3	4	5	0.83
Support self-management and prevention	28	The transplant team routinely works with patients to identify clear, measurable and workable self-management^a goals	1	2	3	4	5	1.00
	29	The transplant team helps all patients set up clear action plans for medication adherence^b	1	2	3	4	5	1.00
	30	Before making or changing a treatment plan, the transplant team discusses the alternatives with the patient and asks them about their preferences	1	2	3	4	5	0.86
	31	The transplant team lets patients decide on the treatment options they prefer	1	2	3	4	5	0.86
	32	The transplant team lets patients decide on the self-management^a goal(s) they consider best for them	1	2	3	4	5	0.86
	33	We always ask patients if they agree with their treatment plans	1	2	3	4	5	0.86
	34	The transplant team routinely assesses obstacles patients believe prevent (or might prevent) them from following their treatment plans	1	2	3	4	5	1.00
	35	The transplant team routinely assesses each patients’ ability to follow the agreed treatment plan	1	2	3	4	5	1.00
	36	The transplant team checks during each follow-up visit whether the patient is dealing effectively with the treatment plan	1	2	3	4	5	1.00
	37	The transplant team gives each patient a copy of the agreed treatment plan, including information on self-management^a in view of, for instance, medication taking, smoking, physical activity, sun protection or following a diet	1	2	3	4	5	1.00
	38	The transplant team routinely asks patients to report on their self-management^a efforts in view of, for, instance medication taking, smoking, physical activity, sun protection or following a diet in diaries, logs or charts	1	2	3	4	5	1.00
	39	The transplant team routinely uses interpreters for foreign language patients if there is a language barrier	1	2	3	4	5	1.00
	40	During follow-up visits, the transplant team routinely reviews, together with the patient, data on self-management,^a such as written logs or charts	1	2	3	4	5	1.00
	41	The transplant team routinely conducts follow-up contacts within 2 weeks for all patients who have set a new self-management^a goal	1	2	3	4	5	1.00
	42	The transplant team routinely follows up with the patient within 2 weeks on all patients who have started a new medication regimen	1	2	3	4	5	1.00
	43	The transplant team routinely assesses patients for their individual areas of concern or goals during follow-up visits	1	2	3	4	5	1.00
	44	The transplant team routinely refers patients to community service agencies or community-based programmes (e.g. programmes to quit smoking) that can help them self-manage their care after leaving the hospital	1	2	3	4	5	0.86
	45	If the transplant team refers patients to community service agencies or community-based programmes, the team routinely works directly with these agencies or programmes to coordinate care for individual patients	1	2	3	4	5	0.86
	46	If the transplant team refers patients to community service agencies or community-based programmes, we routinely follow up with patients to assess their progress	1	2	3	4	5	0.83
Use of information systems	47	The same information about individual patients is accessible in both inpatient and outpatient sections of our transplant programme	1	2	3	4	5	0.83
	48	Care guidelines are built into the information system through computerized prompts/reminders or other support tools	1	2	3	4	5	1.00
	49	The transplant team’s information system automatically gives healthcare workers specific guidance for individual patient care, such as reminders to schedule a follow-up visit or to perform a blood test	1	2	3	4	5	1.00
	50	The transplant team’s information system automatically flags patients who are overdue for routine follow-up	1	2	3	4	5	1.00
	51	The transplant team’s information system is used to give feedback to individual healthcare workers	1	2	3	4	5	0.83
	52	The transplant team’s information system is used to give feedback to the transplant team about the quality of care they are delivering and to identify areas for improvement	1	2	3	4	5	0.83
	53	The transplant team’s information system is used to give information about trends in the transplant population (such as information about survival rates)	1	2	3	4	5	0.67
	54	The transplant team uses information systems to monitor patients at home	1	2	3	4	5	New
	55	The transplant team uses information systems for research purposes	1	2	3	4	5	New

CVI: Content Validity Index.

^aSelf-management refers to actions performed by patients for themselves in daily life to manage their illness and treatment and to avoid health deterioration.

^bAdherence: deals with how well a person's behaviour (for example taking medication) matches the recommendations of a healthcare provider. Adherence is a care concept of self-management.

Phase 3: Inter-rater reliability testing

Demographic information

In total, six healthcare professionals from the abdominal and five from the renal transplant programme completed the CIMI-BRIGHT instrument, which corresponds with all invited healthcare professionals (100% response rate). All the participants in both groups were women with a median age of 33 years for the abdominal transplant programme Inter Quartile Range (IQR) (IQR = 11) and a median age of 42 years for those in the kidney transplant group (IQR = 11). In the abdominal transplant unit, three of the participants (50%) were transplant coordinators, one was an advanced practice nurse, one was a physician’s assistant and one participant was an agency nurse, who had been working in the transplant programme for more than 6 months. In the renal transplant programme all participants were clinical transplant coordinators (not procuring). The percentage of work time spent in transplant care in the abdominal transplant programme ranged from 50% to 100%, while in the renal transplant programme all participants were working 100%. The median years working in this particular renal transplant programme were 3 years (IQR: 5.6); for the abdominal transplant programme it was 7 years (IQR: 7).

Percentage agreement between healthcare providers

The overall percentage agreement between the different healthcare professionals completing the CIMI-BRIGHT instrument in the abdominal transplant programme was 84.5%, which corresponds with almost perfect agreement. Percentage agreement at the item level ranged between 50% (five out of 55 items, 9.1%) and 100% (24 out of 55 items, 43.6%). For the renal transplant programme, the overall percentage agreement was 74.8%, meaning that agreement was substantial. Agreement on individual items ranged from 40% (seven out of 55 items, 12.7%) to 100% (13 out of 55 items, 23.6%). Most items had an agreement of at least 80% (21 out of 55 items, 38.2%).

Discussion

We developed the first instrument designed to assess the extent to which CIM is implemented in the care of solid organ transplant patients from the healthcare professionals’ perspective. The CIMI-BRIGHT instrument will allow investigators to assess the level of implementation of different dimensions of CIM among solid organ transplant programmes at the meso/healthcare organization level of the ICCC framework. Despite the fact that transplant patients are chronically ill, we are not aware of any studies that have assessed the extent to which principles of CIM are implemented in the care of this patient population. Given the potential value of implementing the principles of CIM in post-transplant care and the lack of an instrument to assess implementation, the aim of this study was to develop and conduct initial validation of an instrument assessing the extent to which CIM is implemented in follow-up care of transplant patients.

The process used to develop the CIMI-BRIGHT instrument is consistent with the recommendations of the University of Wisconsin Survey Centre.²⁴ The development of the instrument was based on an iterative process in item generation using different sources, i.e. literature, CIM experts, clinicians, CVIs, combining a qualitative and quantitative appreciation of the items and scale value. Indeed, the CIMI-BRIGHT instrument underwent a circular process of item development, evaluation and revision. The items were reviewed by several groups of experts and the instrument was adapted accordingly based on their feedback. The content validity and the inter-rater reliability testing results suggest that the CIMI-BRIGHT instrument has value as a tool to measure the extent to which CIM is implemented in transplant programmes from the healthcare professionals’ perspective.

The focus was on developing an instrument that would be usable without adaptations in a variety of healthcare systems. This addressed one of the major shortcomings of the existing ACIC instrument. The ACIC was developed in the US.²⁰ One study that attempted to use it in Germany concluded that it was not translatable given the cultural differences between the US and German healthcare systems.²⁵ In another study that used the instrument in the German speaking area of Switzerland, major adaptations were required prior to use.²⁶ Consequently, when we developed the CIMI-BRIGHT instrument, attention was paid to develop an instrument that is applicable across a variety of healthcare systems.

The CIMI-BRIGHT instrument has the potential to be useful in both research and clinical practice. In research studies it can be used to describe the level of CIM implemented in transplant programmes, to examine the relationship between the level of CIM and patient behaviours and transplant-related outcomes, to identify potential areas for interventions in order to improve patient care and test the effects of those interventions and to test the effect of interventional approaches to improve CIM. Utilization of the CIMI-BRIGHT instrument facilitates clinicians in identifying areas where their care for chronically ill patients can be improved. It could be a valuable instrument in not only assessing the current practice patterns in view of CIM, but in also evaluating the impact of interventions designed to improve the organization of care for chronically ill patients. Furthermore, the instrument can be used to benchmark centres with each other, not only nationally but also internationally. Kay²⁷ describes benchmarking as ‘a process of comparison between the performance characteristics of separate, often competing organizations intended to enable each participant to improve its own performance in the marketplace’. Benchmark data are useful in improving the quality by allowing healthcare organizations to identify their strengths and weaknesses and to identify areas for improvement by comparing their performances to that of other organizations.²⁷

The instrument is currently being used in the Building Research Initiative Group: Chronic Illness Management and Adherence in Transplantation study (Clinicaltrail.gov ID: NCT 01608477), a multi-continental study designed to examine the prevalence of adherence to different health behaviours and the association between healthcare system factors and adherence to immunosuppressive medication in heart transplant recipients. Within this study the CIMI-BRIGHT instrument is being used to assess the variability in the level of CIM in 40 heart transplant programmes in 11 countries (i.e. Australia, Belgium, Brazil, Canada, France, Germany, Italy, Spain, Switzerland, the United Kingdom and the United States) and four continents (i.e. Australia, Europe, North America and South America). For this purpose the English version of the CIMI-BRIGHT instrument has been back/forward translated into six languages (Dutch, French, German, Italian, Spanish and Brazilian Portuguese). Data collected in the BRIGHT study will allow us to conduct further psychometric testing of the instrument, including examining the extent to which the CIMI-BRIGHT items conform to the ICCC building blocks, i.e. factor analysis.

Limitations

While this study provided initial evidence related to the validity and reliability for the CIMI-BRIGHT instrument, there are a number of important limitations that need to be acknowledged. First, the content validity was not reassessed after integrating the feedback from the content experts. Second, the pilot testing assessing the inter-rater reliability was only conducted in two US transplant programmes which limit the generalizability of the findings. However, the pilot tests were conducted in two different organ groups and two different teams with similar findings. Third, for both calculating the CVI and the inter-rater reliability, 4-point Likert scale answer options were collapsed into two dichotomous categories. The reason for collapsing the inter-rater agreement options was the limited number of responses in some of the response options. Had we been able to examine per cent agreement for the full range of responses, it is possible that the results would have been different. Yet, our strategy is recommended and used by others as well.²² Finally, the CIMI-BRIGHT instrument focuses on the implementation of CIM principles of the meso level. Future studies should develop instruments to examine factors at the micro- and macro levels as well.

Conclusion

This early testing of the CIMI-BRIGHT instrument suggests that it has promise to measure the extent to which practice patterns in transplant programmes are consistent with the meso level of the ICCC model as assessed by healthcare workers. Additional testing is, however, needed to fully evaluate the validity of the instrument and to confirm that its inter-rater reliability is acceptable in other transplant programmes.

Practical implications for healthcare, policy and research

The findings of this study have implications for clinical practice, research and ultimately for healthcare policy. Clinically this instrument can help transplant programmes to evaluate their current level of CIM, to identify areas for improvement and to evaluate the impact of quality improvement efforts. In research studies the instrument can be used to evaluate which individual or combination of building blocks has the greatest impact on outcomes. Furthermore, it can be used to assess the impact of interventions designed to improve the level of CIM. Findings of these research studies can provide useful information to inform healthcare policy and finance decisions.

Footnotes

Acknowledgements

We wholeheartedly thank all the nurses who participated in the focus group interview and the experts who participated in reviewing the questionnaires and assessing its content validity: Dr Epping-Jordan, Dr Vrijhoef, Dr Burnard, Dr Kane, Dr Lemmens, Dr Lorig, Dr Redman and Dr Peytremann. Furthermore, we want to express our gratitude to clinicians who participated in the inter-rater reliability testing of the CIMI-BRIGHT instrument.

Funding

The study was supported by an International Transplant Nurses Society (ITNS) nursing research grant.

Conflict of interest

None declared.

References

Lodhi

Lamb

Meier-Kriesche

. Solid organ allograft survival improvement in the United States: The long-term does not mirror the dramatic short-term success. Am J Transplant 2011; 11: 1126–1135.

Stehlik

Edwards

Kucheryavaya

. The Registry of the International Society for Heart and Lung Transplantation: Twenty-eighth Adult Heart Transplant Report-2011. J Heart Lung Transplant 2011; 30: 1078–1094.

Nuno

Coleman

Bengoa

. Integrated care for chronic conditions: The contribution of the ICCC Framework. Health Policy 2012; 105: 55–64.

Yach

. Innovative care for chronic conditions – Building blocks for action, Geneva: World Health Organization, 2002.

Pruitt

Epping-Jordan

. Preparing the 21st century global healthcare workforce. Brit Med J 2005; 330: 637–639.

Bodenheimer

Lorig

Holman

. Patient self-management of chronic disease in primary care. JAMA 2002; 288: 2469–2475.

Bodenheimer

Wagner

Grumbach

. Improving primary care for patients with chronic illness. JAMA 2002; 288: 1775–1779.

Nolte

McKee

. Caring for people with chronic conditions – A health system perspective, Berkshire: Open University Press, 2008.

Wagner

. Chronic disease management: What will it take to improve care for chronic illness? Effective Clin Pract 1998; 1: 2–4.

10.

Epping-Jordan

Pruitt

Bengoa

. Improving the quality of health care for chronic conditions. Qual Saf Health Care 2004; 13: 299–305.

11.

Epping-Jordan JE. Integrated approaches to prevention and control of chronic conditions. Kidney Int Suppl 2005; 68: S86–88.

12.

Drewes

Steuten

Lemmens

. The effectiveness of chronic care management for heart failure: Meta-regression analyses to explain the heterogeneity in outcomes. Health Serv Res 2012; 47: 1926–1959.

13.

Busse R, Blümel M, Scheller-Kreinsen D, et al. Tackling chronic diseases in Europe. Strategies, interventions and challenges. The European Observatory on Health Systems and Policies, Copenhagen (Denmark), 2010.

14.

Coleman

Austin

Brach

. Evidence on the chronic care model in the new millennium. Health Affairs (Millwood) 2009; 28: 75–85.

15.

de Bruin

Versnel

Lemmens

. Comprehensive care programs for patients with multiple chronic conditions: A systematic literature review. Health Policy 2012; 107: 108–145.

16.

Moullec

Gour-Provencal

Bacon

. Efficacy of interventions to improve adherence to inhaled corticosteroids in adult asthmatics: Impact of using components of the chronic care model. Respir Med 2012; 106: 1211–1225.

17.

de Bruin

Heijink

Lemmens

. Impact of disease management programs on healthcare expenditures for patients with diabetes, depression, heart failure or chronic obstructive pulmonary disease: A systematic review of the literature. Health Policy 2011; 101: 105–121.

18.

Bissonnette

Woodend

Davies

. Evaluation of a collaborative chronic care approach to improve outcomes in kidney transplant recipients. Clin Transplant 2013; 27: 232–238.

19.

Schmittdiel

Mosen

Glasgow

. Patient Assessment of Chronic Illness Care (PACIC) and improved patient-centered outcomes for chronic conditions. J Gen Intern Med 2008; 23: 77–80.

20.

Bonomi

Wagner

Glasgow

. Assessment of chronic illness care (ACIC): A practical tool to measure quality improvement. Health Serv Res 2002; 37: 791–820.

21.

American Educational Research Association APA, & National Council on Measurement in Education. Standards for educational and psychological testing, Washington, DC: American Psychological Association, 1999.

22.

Polit

Beck

Owen

. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health 2007; 30: 459–467.

23.

Montgomery

Graham

Evans

. Inter-rater agreement in the scoring of abstracts submitted to a primary care research conference. BMC Health Serv Res 2002; 2: 8.

24.

Office of Quality Improvement – University of Wisconsin-Madison. Survey fundamentals – a guide to desgning and implementing surveys, http://oqi.wisc.edu/resourcelibrary/uploads/resources/Survey_Guide.pdf (2010, accessed 20 November 2013).

25.

Steinhaeuser

Goetz

Ose

. Applicability of the Assessment of Chronic Illness Care (ACIC) instrument in Germany resulting in a new questionnaire: Questionnaire of chronic illness care in primary care. BMC Health Serv Res 2011; 11: 164.

26.

Steurer-Stey

Frei

Schmid-Mohler

. Assessment of Chronic Illness Care with the German version of the ACIC in different primary care settings in Switzerland. Health Qual Life Outcomes 2010; 8: 122.

27.

Kay

. Health care benchmarking. Hong Kong Med Diary 2007; 12: 22–27.