Reproducibility of an extensive self-report questionnaire used in secondary coronary prevention

Introduction

As a result of the contemporary management of coronary heart disease (CHD), an increasing proportion of patients survive and require optimal secondary prevention [1]. A high prevalence of unhealthy lifestyle and poor risk factor control in CHD patients was demonstrated in a large European multicentre study [2]. The reasons for these findings are complex and somewhat poorly understood, and the identification of optimal patient management and healthcare factors of importance for an improved coronary risk profile remains a public health priority [3]. The aims of an ongoing cross-sectional study, the NORwegian CORonary (NOR-COR) Prevention Study [4], are to identify medical and psychosocial factors associated with unfavourable risk factor control after a coronary event. Most of the data to be explored has been collected through a comprehensive self-report questionnaire.

Self-report questionnaires are frequently used in health research because they are easy to utilize, feasible, and cheap to apply. In order to ensure reproducibility and reliability, a test–retest study is of great importance. The reliability of such a test is assessed by measuring the responses of the same study sample to an identical questionnaire at two or more points in time. A reproducibility test will assess random measurement errors as well as the stability of the construct measured, but cannot in itself distinguish between the two [5]. Thus, one must take into consideration that any real change in the phenomenon of interest that may have occurred during the intervening period between tests will result in seemingly low levels of reliability.

There are no standards for the ideal time span between the initial test and the retest in reproducibility studies. The interval should be long enough to prevent memory effects and short enough to ensure that no real clinical change has occurred among participants [6]. Intervals of one to two weeks [7] and one month [8] have been suggested.

Self-reported information from questionnaires is frequently used in clinical epidemiological studies, but few provide information on the reproducibility of instruments applied in secondary coronary prevention studies. Those available are limited by only addressing single questionnaires with a moderate range of items. So far, few studies have explored whether reproducibility remains satisfactory in a comprehensive questionnaire applied in clinical patient studies. The purpose of this study was to evaluate the test–retest reliability of an extensive self-report questionnaire assembled and created to be used in the NOR-COR study [4]. Given acceptable reproducibility results, such a questionnaire could be valuable in future studies on risk factor control and lifestyle measures in long-term secondary coronary prevention.

Materials and methods

A complete description of the design and methodology applied in the NOR-COR study is published elsewhere [4]. In the present study, the self-report questionnaire used in NOR-COR was completed twice by 99 stable patients with an interval of four weeks.

Design of the NOR-COR questionnaire

The NOR-COR questionnaire contains 249 questions derived from a number of medical and psychosocial instruments that have previously, to some extent, been demonstrated to be associated with coronary risk factors, adherence to medication, and prognosis in cardiac patients [9–15]. As there were no validated instruments for revealing the patient’s needs and preferences, a number of questions/items were created de novo following an extensive process [16], described in detail previously [4]. The NOR-COR questionnaire was pilot-tested in two CHD patients in order to incorporate the patients’ perspective, and subsequently tested in 20 randomly selected eligible CHD patients in order to establish relevance, acceptance, and feasibility.

The following descriptive variables have been obtained from the questionnaire:

Results

A total of 99 patients completed the retest within an interval of 33 (±6.4) days. One patient who broke his leg and a woman who lost her son within the interval between tests were excluded from the reproducibility study. The mean time interval between index hospitalization and first-time completion of the questionnaire was 34 weeks (range 8–83). The amount of missing data was 1.1% in the first test session and 3.0% in the retest, at the same level throughout the questionnaire. Participant feedback revealed that the time used to fill out the questionnaire was 30 to 45 minutes. Reproducibility figures obtained from the first part of the questionnaire did not differ from those of the last part.

There were no significant differences between patient characteristics among the NOR-COR population and the reproducibility study sample (Table I). The reproducibility values were very good for exercise and smoking (Table II), good for the use of alcohol, and moderate for diet. The reproducibility coefficients for drug adherence were acceptable, and very good for obstructive sleep apnoea.

OPEN IN VIEWER

Table I.

Demographic and medical characteristics of the NOR-COR sample and the reproducibility sample.

	NOR-CORn = 1127	Reproducibilityn = 99	p-value
Age, mean (SD)	61.6 (9.6)	63.2 (8.8)	ns
Female gender, %	21	17	ns
Living alone, %	26	24	ns
Low education, a %	62	55	ns
Coronary diagnosis			ns
Non-ST elevation MI, n %	561 (50)	44 (44)	ns
ST elevation MI, n %	335 (30)	38 (38)	ns
Stable/unstable CHD, n %	231 (21)	17 (17)	ns

NOR-COR: NORwegian CORonary Prevention Study; n: sample size; SD: standard deviation; ns: non-significant; MI: myocardial infarction; CHD: coronary heart disease; ST: ST-segment.

a

Low education was defined as completion of primary or secondary school only.

OPEN IN VIEWER

Table II.

Test–retest reliability of lifestyle risk factors and medical factors.

	Test 1 Mean (SD)	Test 2 Mean (SD)	ICC, 95% CI	κ, 95% CI
Exercise
Frequency, times per week	3.0 (1.5)	2.9 (1.4)	0.85 (0.78–0.90)
Exercise sum score a	9.2 (1.2)	9.0 (1.1)	0.90 (0.85–0.94)
Smoking
Current smoking, n (%)	15 (15)	15 (15)		1.0
Previous smoking, n (%)	66 (67)	69 (70)		0.94 (0.87–1.02)
Never smoked, n (%)	28 (28)	25 (25)		0.87 (0.76–0.98)
Diet
Fish >3 times/week, n (%)	51 (53)	53 (55)		0.49 (0.32–0.66)
Fruit/veg ⩾ 2 times/day, b n (%)	41 (43)	39 (40)		0.44 (0.26–0.62)
Alcohol last 4 weeks, n (%)	81 (84)	84 (86)		0.75 (0.56–0.94)
Drug adherence
Morisky scale sum score	7.4 (0.9)	7.3 (1.0)	0.74 (0.61–0.83)
Obstructive sleep apnoea
Berlin category 1 sum c	1.59 (1.2)	1.55 (1.2)	0.87 (0.80–0.91)
Berlin category 2 sum	0.45 (0.8)	0.41 (0.8)	0.89 (0.83–0.93)

ICC: intraclass correlation coefficient; κ: Kappa agreement; CI: confidence interval; SD: standard deviation.

a

Exercise sum score, sum of frequency, duration, and intensity.

b

Fruit and/or vegetables at least twice a day.

c

Berlin category 1 sum, snoring, and sleep apnoea; Berlin category 2 sum, tired or exhausted.

The test–retest reliability calculations of the psychosocial factors presented in Table III show good reproducibility for quality of life (PCS12) and very good for all other psychosocial instruments. The majority of the questions covering the patient’s perceptions, needs, preferences, and motivation were above the limits for acceptable reproducibility (Table IV). The participants were asked about their preferences for follow-up to meet their present needs of optimal prevention. The reproducibility level for these replies was fair to good.

OPEN IN VIEWER

Table III.

Test–retest reliability of psychosocial factors.

	Test session 1 Mean (SD)	Test session 2 Mean (SD)	ICC (95% CI)
Quality of life, SF12
Physical health sub-scale sum score	41.89 (5.5)	41.44 (5.4)	0.77 (0.65–0.85)
Mental health sub-scale sum score	50.83 (8.5)	50.83 (9.6)	0.89 (0.84–0.93)
Hospital Anxiety and Depression Scale (HADS)
HADS-A sum; anxiety	3.63 (3.7)	3.30 (3.9)	0.92 (0.88–0.95)
HADS-D sum; depression	2.96 (2.9)	3.01 (3.5)	0.94 (0.91–0.96)
HADS-T sum; total score	6.59 (6.1)	6.32 (6.8)	0.95 (0.93–0.97)
Ruminative Response Scale (RRS)
RRS sum score	29.47 (9.3)	29.29 (11.2)	0.88 (0.81–0.92)
Penn State Worry Questionnaire (PSWQ)
PSWQ sum score	35.0 (12.2)	34.6 (12.8)	0.91 (0.86–0.94)
Type D personality (DS-14)
Social inhibition sum score	7.55 (5.5)	7.32 (5.5)	0.90 (0.85–0.94)
Negative affectivity sum score	5.76 (5.5)	5.40 (5.7)	0.91 (0.86–0.94)
Brief Illness Perception Questionnaire (BIPQ)
BIPQ sum score	25.9 (12.3)	25.8 (13.1)	0.91 (0.86–0.94)
Perceived risk perception (PRP), Likert scale 0–10
PRP 1	2.1 (2.1)	2.7 (2.4)	0.59 (0.39–0.73)
PRP 2	6.9 (2.5)	6.9 (2.6)	0.67 (0.50–0.78)
PRP 3	2.3 (2.3)	2.6 (2.5)	0.74 (0.60–0.82)
Bergen insomnia scale
Insomnia sum score	11.2 (10.3)	10.7 (10.5)	0.92 (0.88–0.95)

SD: standard deviation; CI: confidence interval; ICC: intraclass correlation coefficient; CI: confidence interval.

OPEN IN VIEWER

Table IV.

Test–retest reliability of beliefs about disease causes, motivation, perceived needs, and treatment desires.

	Test session 1 Mean (SD)Likert scale 0–10	Test session 2 Mean (SD)Likert scale 0–10	ICC (95% CI)
Beliefs regarding what caused CHD			0.78–0.95
Smoking	3.84 (4.0)	3.80 (4.0)	0.94 (0.91–0.96)
Lack of exercise	4.47 (3.0)	4.91 (3.0)	0.88 (0.83–0.92)
Motivation for lifestyle changes and already applied changes
Motivation to quit smoking	7.83 (3.6)	8.48 (2.5)	0.87 (0.69–0.95)
Motivation to improve diet	5.26 (2.9)	5.39 (2.9)	0.85 (0.78–0.90)
Motivation to increase exercise	5.06 (3.0)	5.52 (2.8)	0.75 (0.62–0.83)
Have changed diet	5.62 (2.6)	5.43 (2.8)	0.84 (0.76–0.89)
Have increased exercise	4.78 (3.1)	4.93 (2.8)	0.89 (0.84–0.93)
Perceived needs of additional information
Sufficient information about disease	8.19 (2.2)	8.23 (2.0)	0.73 (0.60–0.82)
Sufficient information about risk factors	8.58 (1.8)	8.20 (1.9)	0.80 (0.70–0.86)
	Test session 1n (%)	Test session 2n (%)	κ (95% CI)
Healthcare follow-up
Participated in cardiac rehabilitation	92 (93)	92 (93)	0.69 (0.41–0.98)
Current follow-up general practitioner ⩾ 3 times/year	48 (49)	44 (44)	0.72 (0.58–0.86)
Perception of the information provided
Cured from CHD	68 (69)	67 (68)	0.55 (0.38–0.74)
No need to change lifestyle	28 (28)	27 (27)	0.63 (0.46–0.81)
Treatment desires
Email, SMS, telephone	50 (52)	45 (47)	0.56 (0.39–0.73)
Cardiac nurse	66 (68)	61 (63)	0.34 (0.15–0.53)
Multidisciplinary cardiac rehabilitation	32 (33)	26 (28)	0.54 (0.36–0.72)
Physiotherapist	31 (32)	31 (32)	0.71 (0.56–0.86)
Dietician	32 (33)	31 (32)	0.69 (0.54–0.84)
Psychiatrist/psychologist	14 (14)	7 (7)	0.57 (0.44–0.70)
Internet	27 (28)	23 (25)	0.47 (0.28–0.67)
Mobile app	17 (18)	15 (16)	0.70 (0.50–0.90)
No need for follow-up	32 (36)	31 (33)	0.63 (0.46–0.81)

ICC, intraclass correlation coefficient; κ, Kappa agreement; CI, confidence interval; CHD, coronary heart disease.

Fair internal consistency was found for sleep apnoea Berlin Category 1 sum (Cronbach’s alpha = 0.45 in test 1 and 0.35 in retest); however, the values improved to good (Cronbach’s alpha = 0.68 in test 1 and 0.66 in retest) if item 4 (“does your snoring bother others”) was deleted from computation. Moderate internal consistency was found for the 8-item Morisky Medication Adherence Scale (Cronbach’s alpha = 0.54 in both tests) and SF12 (Cronbach’s alpha = 0.65 in test 1, 0.61 in retest). All other scales showed good to very good internal consistency and Cronbach’s alpha ranged from 0.69 to 0.95, with slightly higher values in the second test.

Significant differences in the level of reproducibility across gender, age, or education level were found in a small proportion of the variables; however, there was no consistency regarding which subgroup showed the highest level of reproducibility.

Discussion

The present study analysed the reproducibility of the questionnaire applied in the NOR-COR study. Our findings demonstrated acceptable to excellent values for almost all of the variables explored. This level of reproducibility in data from the NOR-COR questionnaire will be valuable in performing further analyses, findings, and, indeed, conclusions of the project.

There were few missing data in both tests, and the reproducibility remained high throughout the rather extensive questionnaire. The test–retest sample had similar patient characteristics to those of the total NOR-COR population. Information obtained by self-report questionnaires may be distorted by systematic errors such as the patient giving socially desirable answers, using scales and response options in idiosyncratic ways, as well as recall bias. Systematic errors and biases are hard to assess and control, and would in fact tend to boost test–retest correlations. On the other hand, poor or oscillating understanding of the underlying meaning of a question, being distracted or confused, or responding based on current mood will introduce random error or noise in measurements, thereby reducing statistical associations of substantive interest, as well as test–retest correlations. Test–retest correlations allow for estimates of random measurement errors to be established, given that the underlying construct is stable [5]. Thus, acceptable intra-individual reproducibility is reassuring in the sense that one has apparently minimized the risk of committing type II errors because of random error or noise. Conversely, reliability estimates typically based on internal consistency within a set of items tend to be boosted by systematic errors such as response scale effects and thus may yield misleadingly favourable results [27].

The mean time interval between index hospitalization and first-time completion of the questionnaire was eight months. After this relatively long period the majority had completed the rehabilitation process in our hospital. Possible early problems with medication habituation, anxiety, and depression were considered sufficiently diminished, and the patients’ physical activity level was restored. It is, however, not possible to guarantee total stability (i.e. lack of “true” change) over four weeks, but the abovementioned should have reduced the risk of clinically important improvements or deteriorations that might have influenced reproducibility in the data presented. Test–retest correlations tend to be higher when the time interval between the two points of measurement is short, because few changes have occurred, but there is also a risk of memory effects; that is, respondents recalling their response on the first occasion and choosing the same option on the second occasion in order to appear “consistent”. In order to avoid influence of memory effects and to reduce the possibility of significant events and real changes between the two tests, four to eight weeks has been suggested as the ideal time between the two measurements [28,29].

In post-Myocardial infarction (MI) patients, the assessment of type D personality has been shown to be very stable over 18 months [30] and comparable to the good reproducibility of frequency of exercise per week in stable coronary patients when measured with one week interval between the two tests, whereas reproducibility for exercise diminishes with a longer interval between tests [31–33]. In the present context, the majority of our study participants clearly belonged to the category of stable CHD.

We had expected a tendency towards lower reproducibility in questions from the last part of the questionnaire due to tiredness or fatigue. This did not turn out to be the case, as was also observed in a diet study where the length of the questionnaire had only a minor impact on the response rate and data quality [34].

In the INTERHEART study [35] structured questionnaires were administered to obtain information about socio-demographic factors and cardiovascular risk factors. Repeat measures of risk factors were made in 279 controls at a median interval of 409 days. Except from a nearly identical, and very good agreement rate for smoking in INTERHEART and the present study (κ = 0.94 vs. 1.0, respectively), the respective reproducibility values in INTERHEART and our study differed for depression (κ = 0.44 vs. ICC = 0.94), regular physical activity (κ = 0.56 vs. ICC = 0.85 for frequency), and alcohol (κ = 0.52 vs. ICC = 0.75). Different questionnaires and time interval that had elapsed between test and retest may explain these divergences.

The reproducibility of drug adherence, sleep apnoea, and psychosocial factors based upon widely used questionnaires in the present study was high and in line with most other studies [18–25,36].

Our findings of quite acceptable reproducibility data with only few exceptions can be explained by the extensive process used to develop de novo questions and inclusion of questionnaires that have previously been validated and found to have acceptable reproducibility [17–25,36].

These robust reproducibility data will have practical implications for future analysis of the association between potentially modifiable patient factors and unfavourable risk factor control. Since most of the data to be used in this context is derived from the questionnaire, the present findings are reassuring for further NOR-COR projects and for its application in future clinical studies of secondary CHD prevention.

Conclusion

Reliability studies based on test–retests are essential elements when it comes to establishing the quality of self-report data. A good to very good reproducibility was found for almost all of the items and scales used in the comprehensive NOR-COR questionnaire. Thus, this instrument emerges as a valuable tool for evaluating risk factor control in CHD patients in general, laying the foundation not only for further analyses, findings, and conclusions in the NOR-COR project, but also for similar comprehensive questionnaires applied in future clinical patient studies.