Improved active participation among patients with severe acquired brain injury in early rehabilitation: Implementation of a professional competence development intervention

Abstract

BACKGROUND:

In 2019, an educational programme was implemented in a sub-acute in-hospital neurorehabilitation clinic for patients with severe acquired brain injury (sABI). The programme was initiated to enhance staff competencies related to identifying and improving active participation among patients with sABI.

OBJECTIVE:

The purpose was to evaluate the implementation effectiveness of the educational programme.

METHODS:

Mixed methods were chosen to assess implementation effectiveness as perceived by staff and patients.

RESULTS:

A survey of the professional’s experience showed an increase in perceived competence after each completed seminar and from before the first seminar to after the last completed seminar. These results were confirmed and elaborated through staff focus group interviews. The proportion of patients achieving active participation increased from 45% before to 75% after implementation (six of eight patients).

CONCLUSION:

Exploring the implementation effectiveness of the educational programme seemed clinically valuable and showed a promising and probable effect of an implementation process.

Keywords

Health services research brain injuries rehabilitation education public health professional professional competence implementation

1 Introduction

Optimal treatment of people with severe brain injury in the early stages of rehabilitation should be based on the best available evidence and delivered with a maximum of professional competence [1]. A definition of evidence-based practice from the patient’s perspective is described as: “ . . . integrating the best available research evidence with clinical expertise and the patient’s unique values and circumstances” [2], which should be combined with contextual factors: “It [evidence-based practice] also requires the health professional to take into account characteristics of the practice context in which they work” [3] and is a mixture of the following four aspects: i) relevant research conducted using rigorous methodology, ii) clinical expertise combining the clinician’s experience, education and clinical skills, iii) the patient’s personal preferences, concerns, expectations and values, and iv) the practice context including characteristics of the situation in which the interaction between patient and health professional is taking place (e.g. the resources available) [3].

Translation of knowledge from research into clinical practice may take an average of 17 years [4]. Additionally, a gap exists between research and practice as a considerable amount of the research results are never implemented or applied in practice [4]. One reason for that could be low competence among health professionals regarding implementation of new knowledge. “Professional competence is the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.” For health professionals within rehabilitation, ability to apply competence in clinical decision-making is a requirement for professional practice [5, 6].

A study exploring work task distributions of health professional developers of three professions (physiotherapists, occupational therapists and nurses) in rehabilitation found, that their work task distributions contributed mainly to maintenance of existing professional knowledge within the organisation rather than implementation of new knowledge. The authors recommended an educational competence boost to better fulfil the intended responsibility for development of evidence-based practice [7]. Clinical education has shown effective improvement on professional competence such as professional behavior, knowledge, and application in practice [8, 8].

Clearly measuring the effect of implementation is a complex and difficult endeavour. To match the highest international standard of evidence, Proctor et al. have clarified that treatment effectiveness and quality of care must be ensured through successful implementation of new interventions, practices and services [9]. This implies that the effectiveness and quality of treatment and care depend on the implementation of new knowledge in daily work routines through practice development in organisations. Furthermore, Proctor et al. along with other researchers advocate a concept of “implementation outcomes” which is distinct from service system outcomes and clinical treatment outcomes; hence, implementation outcomes are defined as the effects of deliberate and purposive actions to implement new treatments, practices and services [10 –14].

A research project focusing on early rehabilitation of patients with a severe acquired brain injury (sABI) was conducted at the Clinic for Early Neurorehabilitation (CEN), Regional Hospital Hammel Neurocenter (RHN), Denmark [13 –15]. The study showed that promoting the active participation of patients with impaired conscious state in the therapeutic interventions and everyday activities is difficult but possible [13 –16]. The research showed that employing a specially designed strategy is fundamental in clinical rehabilitation, and that systematic, insistent efforts may enhance partial participation among minimally conscious patients [13 –15]. Strategies to promote participation were designed in the model coined GRIP –an acronym for (i) Gain contact, (ii) Register responses, (iii) Intentional interaction, and (iv) Partial participation [13]. The highly experienced therapists and nursing caregivers working in early rehabilitation have acquired some of these skills. The less experienced healthcare workers expressed difficulties in applying this particular approach and found that the challenge brought stress to their daily work [13, 16].

Severely injured people in general and those with impaired consciousness and severe communication problems in particular are rarely included in international research. The present study therefore aimed to explore the implementation of an intervention to enhance these vulnerable patients’ active participation in rehabilitation. Management, researcher and staff members designed an educational programme involving all employees in the department during 2019. The programme was intended to boost the competencies of staff members at the Clinic for Early Neurorehabilitation (CEN). The course was considered an implementation project.

1.2 Objective

1.2.1 Purpose

The purpose of this study was to explore the implementation effectiveness of an educational programme designed to enhance staff competencies to improve active participation among people suffering from sABI with no or limited active participation.

We used both quantitative and qualitative methods to explore the effect, focusing on two perspectives; (i) the staff’s perceived learning and the significance hereof for their daily practice; and (ii) the potential derived effect on patients’ ability to function and participate.

The aim was to investigate whether:

the perceived level of experienced competence among health professionals increase

if health professionals provide important reflections on their potential benefits and shortcomings of the educational programme

if professionals with less perceived competence reap the greatest benefits

if improvements in health professionals’ competences lead to better treatment and thus improved rehabilitation outcomes for patients.

2 Design

2.1 Methods and concept

The study was a mixed methods implementation study exploring both the staff and patient perspective. The study was divided into three parts. A survey regarding the professional‘s experienced level of competence improvement, focus group interviews on the health professionals’ reflections on their potential benefits of the educational programme, and a brief quantitative account of the possible derived effect on patients’ functioning. This was a prospective comparative before-after study.

2.1.2 Implementation framework

Proctor et al.’s article on terminology in outcomes in implementation research framed the investigation [9]. Furthermore, the educational programme was inspired by situated learning theory, in which learning is regarded as an ongoing, complex and often puzzling process [17]. In that process, learning and change occur via participants’ involvement, negotiations, exploration and experimentation in their daily practices. Sociocultural learning has an interactive character, and views and opinions are mutually co-designed with others also involved in the process [17]. These theories also understand learning as an incorporated part of a person’s embodied and sociocultural practice [18].

2.2 Setting

The CEN offers severe ABI patients early, intensive, specialised interdisciplinary treatment and rehabilitation [19], in some cases commencing while the patient is still in a coma, vegetative state (VS) or in a minimally conscious state (MCS), while not needing mechanical ventilation therapy [20]. Patients with sABI admitted to the CEN arrive in a sub-acute condition following an incident. The patient has survived, but now faces overwhelming and complicated problems, often suffering from dysphagia, palsy, cognitive deficits and other direct or indirect sequelae of the sABI. Patients require close monitoring and extensive use of equipment in close proximity. At the CEN, the interdisciplinary rehabilitation team members encounter patients in therapeutic and daily life situations in round-the-clock treatment and rehabilitation [20]. The therapeutic treatment in daily life situations involves a plethora of complex interactions and interventions, depending on the patient’s situation. The interdisciplinary rehabilitation team at the CEN includes a broad range of health professionals (physicians, neuropsychologists, social workers, nurses, occupational therapists, physiotherapists as well as social and healthcare assistants). To ensure continuity in the targeted efforts during 24-hour rehabilitation, health professionals work in interdisciplinary teams as recommended by Wade [21]. Each team member has in-depth knowledge about the patient’s daily living, social relationships and health condition and illness. Information regarding the patient is handed over via medical records and oral reports between health professionals working in day-, evening- and nightshifts. Professional competence development within the team is substantiated through systematic knowledge sharing, peer training and mentoring with communicative feedback as part of daily professional reasoning and practice.

2.3 Intervention –the educational programme

The intervention –the educational programme –is a competence development-training course for staff at the CEN led and taught by an experienced external instructor well educated in therapeutic treatment methods targeting improvement of active participation of patients with severe acquired brain injury in early rehabilitation.

The educational programme was imparted during three on-site seminars in 2019. Between each of the three seminars, the staff could share knowledge and they could discuss and communicate about sensitive or complicated dilemmas related to the contents of the programme and hereby translate research and implement the newly gained skills into practice.

The overall theme of the educational programme was strategies to promote participation of patients with a low level of consciousness. The GRIP model [13] was used as an important framework for practice throughout the three seminars. See the GRIP model in Appendix 1

Seminar I: A five-day educational programme (week five of 2019) followed by six weeks of situated learning.

Seminar II: A three-day educational programme (week 11 of 2019) followed by eight months of situated learning.

Seminar III: A two-day educational programme (week 47) and evaluation (November-December 2019).

Training consisted of a combination of introductory presentations, bedside learning with supervision for individual health professionals and summary plenaries with video illustrations and discussion of key elements. Between seminars, the individual teams were expected to work on the themes and elements themselves and to provide peer supervision. This process was supported by the nurse, occupational therapist (OT) and physiotherapist (PT) responsible for clinical development. The topics of the training sessions were: i) consciously supporting the patient in the moment aiming for contact; ii) focusing on the interaction between (the patient’s) body and the environment yielding effects on patient’s participation; iii) timing and flow of activities improving activity; and iv) comprehensive bodily support, guidance and intensity of activity and participation in daily life skills [15].

2.4 Participants

The staff at the CEN who supported the patients daily (nurses, OTs, PTs and social and healthcare assistants (HAs)) joined the survey and the focus group interviews.

2.5 Methods and procedures

2.5.1 Survey

The survey was distributed by email from the clinic manager to all health professionals working at the clinic one to two days after the seminars. The response period varied from one to 32 days.

Three reminders were sent out to improve response rates. The e-mail distribution to the entire staff meant that not only those directly involved in the educational activities were invited to participate. The rationale of this choice was that the general approach and work culture could be affected by the implementation.

2.5.2 Survey questions

Validated tools for evaluating implementation outcomes are not yet available [22, 23]. A non-validated questionnaire was therefore developed jointly by all the authors and set up in RedCap. Study data were stored and managed using RedCap electronic data capture tools hosted at Aarhus University, Denmark [23, 24]. Proctor et al.’s article on terminology in implementation research framed the questionnaire [9]. An evaluation form was prepared based on eight concepts for implementation outcomes [9] (Table 1).

Table 1
Concepts for implementation outcomes

Implementation outcome Nominal definition by Proctor et al. Outcome measurement

Acceptability The perception among implementation stakeholders that a given treatment, service, practice or innovation is agreeable, palatable or satisfactory

Adoption The intention, initial decision or action to try or employ an innovation or evidence-based practice Survey question 21–23

Appropriateness The perceived fit, relevance or compatibility of the innovation or evidence-based practice for a given practice setting, provider or consumer; and/or perceived fit of the innovation to address a particular issue or problem Survey question 26, 27

Feasibility The extent to which a new treatment or an innovation can be successfully used or carried out within a given agency or setting

Fidelity The degree to which an intervention was implemented as it was prescribed in the original protocol or as it was intended by the programme developers Survey question 24, 25

Implementation cost The cost impact of an implementation effort

Penetration The integration of a practice within a service setting and its subsystems Survey questions 13–20

Sustainability The extent to which a newly implemented treatment is maintained or institutionalised within a service setting’s ongoing, stable operations

Implementation outcome	Nominal definition by Proctor et al.	Outcome measurement
Acceptability	The perception among implementation stakeholders that a given treatment, service, practice or innovation is agreeable, palatable or satisfactory
Adoption	The intention, initial decision or action to try or employ an innovation or evidence-based practice	Survey question 21–23
Appropriateness	The perceived fit, relevance or compatibility of the innovation or evidence-based practice for a given practice setting, provider or consumer; and/or perceived fit of the innovation to address a particular issue or problem	Survey question 26, 27
Feasibility	The extent to which a new treatment or an innovation can be successfully used or carried out within a given agency or setting
Fidelity	The degree to which an intervention was implemented as it was prescribed in the original protocol or as it was intended by the programme developers	Survey question 24, 25
Implementation cost	The cost impact of an implementation effort
Penetration	The integration of a practice within a service setting and its subsystems	Survey questions 13–20
Sustainability	The extent to which a newly implemented treatment is maintained or institutionalised within a service setting’s ongoing, stable operations

2.5.3 Focus group interviews

Focus group interviews were conducted across the interdisciplinary teams who were in daily contact with the patients (nurses, OTs, PTs, HAs) (Table 2). Focus group interviews were chosen to facilitate dialogue and promote elaboration of experiences and ideas among colleagues [25]. A semi-structured interview guide prepared jointly by HP and LA was used as a checklist in all interviews. The interview guide was also based on Proctor et al.’s terminology on implementation outcomes [9]; however, it focused specifically on staff’s reflections on their learning process and on how to implement their new knowledge and skills into practice. The interview guide had a similar structure for all three interviews. However, questions about the implementation process, practice changes and learning sustainability were highlighted in the second and even more so in the third focus group interview. A focus group interview was held after each of the first two courses. The third focus group interview was held ten days before the final course. The final interview was held ahead of the course to get an impression of the implementation process up to this point and be able to focus the last course specifically on course-related difficulties participants had experienced. The interviews lasted 30–40 min each and were audiotaped and transcribed in full.

Table 2
Participants in focus group interviews

1. Focus group interviews

Profession Nurse Nurse OT OT OT OT HA

Gender Female Female Female Female Female Female Female

Years of experience at the CEN 1 7 9 1½ 2½ 2½ 6

2. Focus group interviews

Profession S-PT S-OT S-nurse PT Nurse Nurse

Gender Female Female Female Male Female Female

Years of experience at the CEN 22 5 4 1 2½ 2

3. Focus group interviews

Profession HA OT Nurse PT Nurse

Gender Female Female Female Female Female

Years of experience at the CEN 12 8 2 3 5½

1. Focus group interviews
Profession	Nurse	Nurse	OT	OT	OT	OT	HA
Gender	Female	Female	Female	Female	Female	Female	Female
Years of experience at the CEN	1	7	9	1½	2½	2½	6
2. Focus group interviews
Profession	S-PT	S-OT	S-nurse	PT	Nurse	Nurse
Gender	Female	Female	Female	Male	Female	Female
Years of experience at the CEN	22	5	4	1	2½	2
3. Focus group interviews
Profession	HA	OT	Nurse	PT	Nurse
Gender	Female	Female	Female	Female	Female
Years of experience at the CEN	12	8	2	3	5½

Abbreviations: Social and healthcare assistant (HA), Occupational therapist (OT), Physiotherapist (PT); Specialised physiotherapist (S-PT), Specialised occupational therapist (S-OT), Specialised nurse (S-nurse).

2.5.4 Functional assessment of patients

Early functional ability (EFA) assessments of patients were registered [25 –27]. All patients for whom at least two EFA scores were available during their admission at CEN were included.

EFA sub-items and sum scores for all patients admitted were extracted from the electronic hospital records in the following three periods;

Before the intervention (baseline) from 02 November 2018 to 24 January 2019

During the intervention from 25 January 2019 to 24 November 2019

After the intervention (follow-up) from 25 November 2019 to 29 February 2020

2.6 Data collection

The study focused on implementation effectiveness measured by staff’s perceived satisfaction and learning, as well as by patient outcome. Data were gathered from questionnaires of all staff members, focus group interviews across interdisciplinary teams with daily “hands on” contact with patients, and patient outcomes were measured by EFA [25, 26].

3 Analysis

A descriptive analysis of the questionnaires was performed. Data from the three focus group interviews were analysed in four steps and coded into categories by two researchers (HP, LA) in line with the content analysis described by Elo and Kyngäs [28]. Patient-related outcomes of the rehabilitation efforts were measured as improvement in EFA scores from admission to discharge from the CEN. Active participation was defined as an achieved sub-item score≥3 on either of the EFA items 14, 16, 17, 18 or 19, or a score≥2 on EFA item 20. Eligible patients were defined as those without active participation at inclusion.

3.1 Ethical considerations

The study was completed in accordance with the Helsinki Declaration of 2013. Informed written consent was obtained from the professionals. According to Danish Regional Scientific Ethical Committee regulations, the project was not notifiable as no human biological material was included and the study was categorised as a quality development project. Therefore, the study was exempt from the notification and permission requirements by the Danish Data Protection Agency [29].

4 Results

4.1 Questionnaires

Nurses and HAs represented 56–63% of respondents, which paralleled a larger proportion of the total staff relative to therapists. Respondents at the first survey time had the highest median number of years of experience (7.5), and the lowest experience level was recorded at the second survey time (3.0).

Participants were asked about their perceived level of competence before and after the programme immediately after having participated in each educational programme module. They reported a high level of competence with a range from 64 to 91 out of 100. They reported a similar competence level over time (medians were 72, 74 and 76, respectively) which an expected increase post programme (with medians of 80, 79 and 82, respectively). Up to 78–88% reported that the new skills were applicable in clinical practice (Table 3).

Table 3
Characteristics of survey respondents

Educational programme seminar I n = 40 II n = 32 III n = 30

Profession n (%) Nurse 19 (48) 19 (59) 13 (43)

HA 6 (15) 1 (3) 4 (13)

OT 9 (23) 8 (25) 7 (23)

PT 6 (15) 4 (13) 6 (20)

Years of experience in neurorehabilitation median (Q1–Q3) 7.5 (2.0–9.3) 3.0 (2.0–8.5) 5.5 (3.0–9,8)

Perceived level of competence before educational programme (0–100) median (Q1–Q3) 72 (64–80) 74 (59–80) 76 (66–88)

Perceived level of competence after educational programme (0–100) median (Q1–Q3) 80 (68–86) 79 (65–84) 82 (71–91)

Perceived professional gain from the educational programme (some-large), n (%) 30 (75) 22 (69) 22 (73)

Perceived professional level of educational programme as appropriate, n (%) 28 (70) 21 (66) 22 (73)

Reported improved professional understanding of promotion of patient participation, n (%) 28 (70) 20 (63) 22(73)

Reported improvement of competence regarding promotion of active patient participation, n (%) 22 (55) 20 (63) 22(73)

Reported that new skills were applicable in clinical practice, n (%) 35 (88) 25 (78) 24 (80)

Reported that new principles were transferable into daily planning of activities, n (%) 28 (70) 22 (69) 22 (73)

Educational programme seminar		I n = 40	II n = 32	III n = 30
Profession n (%)	Nurse	19 (48)	19 (59)	13 (43)
	HA	6 (15)	1 (3)	4 (13)
	OT	9 (23)	8 (25)	7 (23)
	PT	6 (15)	4 (13)	6 (20)
Years of experience in neurorehabilitation median (Q1–Q3)		7.5 (2.0–9.3)	3.0 (2.0–8.5)	5.5 (3.0–9,8)
Perceived level of competence before educational programme (0–100) median (Q1–Q3)		72 (64–80)	74 (59–80)	76 (66–88)
Perceived level of competence after educational programme (0–100) median (Q1–Q3)		80 (68–86)	79 (65–84)	82 (71–91)
Perceived professional gain from the educational programme (some-large), n (%)		30 (75)	22 (69)	22 (73)
Perceived professional level of educational programme as appropriate, n (%)		28 (70)	21 (66)	22 (73)
Reported improved professional understanding of promotion of patient participation, n (%)		28 (70)	20 (63)	22(73)
Reported improvement of competence regarding promotion of active patient participation, n (%)		22 (55)	20 (63)	22(73)
Reported that new skills were applicable in clinical practice, n (%)		35 (88)	25 (78)	24 (80)
Reported that new principles were transferable into daily planning of activities, n (%)		28 (70)	22 (69)	22 (73)

HA = Social and healthcare assistant, OT = Occupational therapist, PT = Physiotherapist; Q1/Q3 Interquartile range (25/75%).

4.2 Focus group interviews

Eighteen health professions (seven nurses, six OTs, three PTs and two HAs) joined the three focus group interviews (Table 2). Years of experience at the CEN varied from one to 22 years across the three interviews. The most experienced, the specialised nurse, OT and PT, all participated in the second interview because they had to facilitate the process of implementation of courses two and three. The gender composition of survey participants was in line with that of the staff as a whole, and only one of the participants was a male. The participants only took part in the interview once, and the participation depended on the shift schedule and the extent to which they were present in the daytime as all focus group interviews were conducted during dayshifts. Four important themes emerged from the analysis of the three focus group interviews (Table 4).

Table 4
Main themes that emerged from the analysis of the focus group interviews

Main themes Sub themes

Individual and professional benefit A basis of mutual knowledge

To be experienced in making the right choices

From novice to expert

Improving practice Common points of attention Shared language, uniform practice and documentation

Preserving skills and cultural change Time for contemplation and training Supervision

Criticism of course

Main themes	Sub themes
Individual and professional benefit	A basis of mutual knowledge
	To be experienced in making the right choices
	From novice to expert
Improving practice	Common points of attention Shared language, uniform practice and documentation
Preserving skills and cultural change	Time for contemplation and training Supervision
Criticism of course

4.2.1 Individual and professional benefit

Almost all the informants expressed that they had benefited from the course, both the three intensive course elements and the intermediate period in which they themselves were to continue to integrate new knowledge and skills in their own practice.

4.2.1.1 A basis of mutual knowledge. In the first focus group interview, the informants were keen to describe how the educational programme had developed a shared knowledge base among all department staff. They expressed that the course had opened their eyes to the importance of contact for a possible interaction with the unparticipating patient. Specifically, they voiced the importance of establishing a relationship, of touch as interaction and the different manners in which they may communicate; directly through the activities, with the surroundings, with signs and signals. Finally, they expressed having gained specific tips and tools that they found would be useful in practice.

Some of the points that I noticed was the connection between relationships which I thought about a lot... hmm how important that was...establishing the relation....And the point about how you present yourself... and also, I took note of some of the tips. (OT, more than 1½ years of experience at the CEN).

...establishing contact, physically I mean, where to place your hand... I guess I thought quite a lot about that. (Nurse, less than 1½ year of experience at the CEN).

In the second interview (two months later and after the second course part), the informants appeared to have progressed a step further in their learning process. They expressed that as a group they had gained a shared understanding, theory and language. In particular, they recalled having achieved this by using the GRIP model (Appendix1). The model was experienced as an acceptable and trustworthy frame for their interaction with the patients. For the novices, it posed as a good overview model which they described as useful in daily practice because it helped them understand and communicate with others about the level or manner of contact they were able to establish with the patient. For the experienced health professionals, the model introduced theoretical concepts and increased professional awareness of the patient’s participation.

We have had a model presented or we work with a model that we have learned a lot more about... (Nurse, more than one and a half years of experience at the CEN).

...we had clearly worked with some of this stuff earlier, but I think that by focusing on it and maybe also by having been introduced to some theoretical concepts, and ehh systematizing what we are looking for exactly, I think that we are becoming more aware of what we are doing, and I also believe that we are adding more stuff and I think we are spotting more things and are more attentive to things..., we have become more aware of what exactly we are looking for and how we may also start using it in our strategies... Is there anything to which we must pay special attention in relation to latency periods or contact relationships.

It is a fantastic opportunity for new colleagues to learn how to build contact, because it takes a long time to build that and for new people it may be difficult. (PT, less than one and a half years of experience at the CEN).

In the final interview (two weeks before the last course), the informants were occupied describing how they had used the model since the second seminar. They had started to integrate the knowledge they gained during the two previous seminars into their practice, e.g. when receiving a new patient at the ward, during morning planning, when preparing a rehabilitation plan or applying symbols or markers meant to nudge staff to keep the model in mind.

For example, I have sometimes experienced that GRIP has come up during the preparation of rehabilitation plans. Eeh where we have tried eeh to use it in that context. (Nurse, more than five years of experience at the CEN).

...and, actually, when receiving new patients, when we are already aware about, i.e. the level of consciousness of the incoming patient or the person who has been allocated to treatment, it has seemed natural in the planning that we also discuss that in this case we will likely need to come to “GRIP”, so to speak, with a patient. (OT, more than five years of experience at the CEN).

It has also been like a focus area during morning planning sessions. Where we have quite simply talked about it. But it has also been,...those small magnets we place. Grip has its own magnet (PT more than one and a half years of experience at the CEN).

4.2.1.2 To be experienced in making the right choices. Embedded bodily competencies emerged as a theme during the analysis. The content of the course was translated into practice, and each participant used the competences as part of their daily practice. In the interviews, the informants reflected several times on dilemmas and difficulties in stimulating the patient to participate in a timely manner and under the right circumstances.

...it was unfair to have talks with the patient about what kind of music he would like to listen to... maybe one should have followed his wish to listen to some relative energetic music when I had actually thought that he needed rest... but then, too, I think that is also a bit odd in relation to being true to what is good for him, I mean if you have introduced the topic, otherwise I should have not asked him what he would like to listen to... (OT, more than five years of experiences at the CEN).

4.2.1.3 From novice to expert. The analysis revealed that the length of employment and the level of experience influenced to which extent staff members experienced a change in professional behaviour in themselves and others. An experienced employee expressed the following.

I find it hard to see anything new in this. Possibly, you see others now making an extra effort to establish contact with the patient, but eeh I think that I’m always lucky when it comes to establishing contact even with the minimally conscious patients, at some level. The initiatives have maybe also differed depending on whom I was dealing with, but I think I’ve always worked like that. (HA, more than 10 years of experience at the CEN).

I actually think that it’s good that this has been introduced right now, with all the new employees and they have not “grown up” in neurorehabilitation as have some of the very experienced people in the department...that we can say that we use a model and that here are a number of points... That gives them a relatively quick way into also speaking about that part of neurorehabilitation... On the other hand, I believe that our more experienced colleagues have been doing this for a long time already. (PT, more than one and a half years of experience at the CEN).

I think it is completely fine that knowledge is introduced and a model to follow because this has been something that eeh... that we have done without anyone really noticing until now, and it actually deserves a lot more attention that we are, in fact, doing it... (HA, more than 10 years of experience at the CEN).

The course seemed to have made tacit knowledge more explicit.

4.2.2 Improving practice

Changing the former practice was another key theme that emerged during the analysis. The addressed need for changes was labelled at the individual level of interaction with the patient as well as at a structural level, e.g., how the staff could incorporate their new knowledge and enactment when planning the rehabilitation activities. The themes, shared language and points of attention became visible during the interviews and analysis.

4.2.2.1. Points of attentions. The data show that the course established an awareness in the participants of how small changes, e.g., guiding or stimulation at a certain point in the activity –a reaction from the patient being followed up, etc. –promote interaction with and participation of the patient. This may mean that the future daily interaction with the patient was given increased attention and a more prominent position owing to the course.

I may need to pay more attention to everything, for example just to tooth brushing, stimulation, displacement, it may be a general response from the patient, or a reaction. (HA, more than five years of experience at the CEN).

Sometimes, I think that the approach needs to be slower, I mean that we should take time to eeh and may present something and then actually see, so that it is not my agenda that comes into play, but how the patient registers it. (OT, more than one and a half years of experience at the CEN).

I think and have become more aware of, and have also become able to see in my colleagues,...this thing about spending a bit more time..., and also to inform the patient... and I try to insist on some kind of response. (PT, more than one and a half years of experience at the CEN).

4.2.2.2 Shared language, uniform practice and documentation. All informants agreed that their new, shared language along with the fact that every staff member knows how the concepts should be understood and used was of vital importance to the department’s professional level. They recognised that the educational programme had improved and created a uniform practice at the department.

It has been easier for us to put into words and describe in words what we see and do and what we interpret and eeh the actions we initiate based on this. I mean, we have established a much clearer language about this all of us... so we are more alike now, I think. (OT, more than five years of experience at the CEN).

Although the informants acknowledge that they have now acquired an important shared language, they wish that their focus on contact and interaction could be systematically documented and recorded.

But just like we have an SFI (documentation of specific health professional content) for other things, we could also just have a neurorehabilitation SFI, I mean a place where you can put it in writing and keep everything in one place, a GRIP SFI. (Nurse with more than five years of experience at the CEN).

Something you can tick off and say that you have observed. (PT, more than one and a half years of experience at the CEN).

...in some way or other to document this, and then evaluate the contact with and the patient’s participation, so that it can be used further... (OT, more than five years of experience at the CEN)

4.2.3 Preserving skills

In the last interview (which took place two weeks before the final course part), the informants described concerns about how they may maintain and further develope their knowledge and the approach they had gained during the course. In particular, they wanted the final course to focus on how to document their new knowledge, but they also requested continuous support and supervision.

4.2.3.1 Lack of time for contemplation and training. Prioritising time for immersion in self-study seems to be a problem.

I mean, I know that there are training videos that are readily available, but I also know that I have not yet had time to come in, sit down and look into it; I have not had the time to get it under my skin and to use it in the context of a patient. (Nurse, more than one and a half years of experience at the CEN). I mean, I know that we have had periods during which we have focused on feedback in relation to GRIP, of course. That we have seen patients in pairs, therapist and nurse, or whatever, and then we have been able to provide feedback to each other afterwards. That we have done. But I also agree that I have not had the time to watch those videos. (Nurse, more than five years of experience at the CEN).

4.2.3.2 Supervision. Although time for self-study was reported not to be an option, peer-studies, feedback in smaller groups seemed to be a suitable solution.

...but having time to talk about it afterwards is also important. What did you see? What are your thoughts about that? We often don’t. We should do that more (Nurse, one year of experience in the CEN).

I mean, to reflect, if two of us participate in a treatment and do cross-disciplinary reflection afterwards, and then allow each other to provide constructive criticism and to be able to receive criticism also. I don’t think we’re generally that good at that, but it would be useful (OT more than one and a half years of experience at the CEN).

Subsequently, the informants discussed and expressed how instructive it had been to use video recordings of the sessions followed by reflection and supervision. They all wished that it could be a way of continuing to sharpen their interest and keep the focus on patient contact and interaction in the future. They were unified and aware that video recordings produce insight but also that their use might provoke anxiety. For this reason, videos should be used with a focus on creating a safe and comfortable environment.

4.2.4 Criticism of the course

Everyone agreed that it had been fruitful to have bedside learning in the course planning, so the key points could come to life. This also generated constructive discussion about the dilemmas that staff often face and must deal with. That only part of the staff was able to participate in the full three courses due to the duty schedule was described annoying by the informants, but they also expressed their understanding that this was unavoidable. Furthermore, the informants reflected on the organisation of upcoming courses to enhance the effectiveness of the implementation process.

So I don’t know if it might have been possible to do something totally different from the start, maybe more planning of who were involved; I mean, I know it has also been about who could actually attend, but maybe to a greater extent choose who we have as a group to get the whole package rather than aiming broadly with less content, because, naturally, the idea was to include as many participants as possible. Because many people actually didn’t receive the full package...in that way at least one group, a resource group, will be there to pass it on rather than... (nurse more than five years of experience at the CEN).

4.3 Patient-related outcome

The proportion of patients (Table 5) who achieved active participation increased from 45% before implementation to 75% after implementation (Table 6). However, the results are not statistically significant as the number of patients was too low during the test period.

Table 5
Characteristics of included patients

Baseline Intervention Follow-up p-value test

n 11 37 8

Sex male, n (%) 8 (73) 20 (54) 7 (88) 0.16 Fisher’s exact

Age, mean (sd) 59 (11) 55 (17) 57 (14) 0.77 ANOVA

Acute care duration (days), mean (sd) 46 (21) 51 (68) 30 (18) 0.63 ANOVA

ABI diagnosis, n (%)

Stroke 5 (45) 14 (38) 2 (25) 0.58 Fisher’s exact

Encephalopathy 0 (0) 5 (14) 1 (13)

Infection 0 (0) 3 (8) 0 (0)

SAH 4 (36) 6 (16) 1 (13)

Traumatic 1 (9) 7 (19) 3 (38)

Tumour 0 (0) 1 (3) 1 (13)

Other 1 (9) 1 (3) 0 (0)

Length of stay in rehab (days), mean (sd) 111 (44) 96 (42) 99 (23) 0.59 ANOVA

Time from EFA1-EFA2, median (Q1–Q3) 48 (19) 51 (35) 42 (16) 0.74 ANOVA

EFA1 sum, median (Q1–Q3) 41 (31–50) 37 (31–44) 58 (49–66) 0.60 Kruskal-Wallis

EFA2 sum, median (Q1–Q3) 45 (34–60) 53 (43–66) 18 (12–31) 0.37 Kruskal-Wallis

ΔEFA, median (Q1–Q3) 6 (–2–18) 16 (5–25) 18 (12–31) 0.12 Kruskal-Wallis

	Baseline	Intervention	Follow-up	p-value	test
Sex male, n (%)	8 (73)	20 (54)	7 (88)	0.16	Fisher’s exact
Age, mean (sd)	59 (11)	55 (17)	57 (14)	0.77	ANOVA
Acute care duration (days), mean (sd)	46 (21)	51 (68)	30 (18)	0.63	ANOVA
ABI diagnosis, n (%)
Stroke	5 (45)	14 (38)	2 (25)	0.58	Fisher’s exact
Encephalopathy	0 (0)	5 (14)	1 (13)
Infection	0 (0)	3 (8)	0 (0)
SAH	4 (36)	6 (16)	1 (13)
Traumatic	1 (9)	7 (19)	3 (38)
Tumour	0 (0)	1 (3)	1 (13)
Other	1 (9)	1 (3)	0 (0)
Length of stay in rehab (days), mean (sd)	111 (44)	96 (42)	99 (23)	0.59	ANOVA
Time from EFA1-EFA2, median (Q1–Q3)	48 (19)	51 (35)	42 (16)	0.74	ANOVA
EFA1 sum, median (Q1–Q3)	41 (31–50)	37 (31–44)	58 (49–66)	0.60	Kruskal-Wallis
EFA2 sum, median (Q1–Q3)	45 (34–60)	53 (43–66)	18 (12–31)	0.37	Kruskal-Wallis
ΔEFA, median (Q1–Q3)	6 (–2–18)	16 (5–25)	18 (12–31)	0.12	Kruskal-Wallis

Table 6

Active participation

Achieved functional ability level indicating active participation^*	Baseline	Intervention	Follow-up	Total
No n (%)	6 (55)	7 (19)	2 (25)	15 (27)
Yes n (%)	5 (45)	30 (81)	6 (75)	41 (73)
Total	11 (100)	37 (100)	8 (100)	56 (100)

p= 0.075 (Fisher’s exact test). ^*Active participation was defined as achieved sub-item score≥3 on either of the Early Functional Ability (EFA) items 14, 16, 17, 18 or 19, or a score≥2 on EFA item 20.

4.4 Discussion

The purpose of this study was to evaluate the implementation effectiveness of an educational programme initiated to enhance staff competencies in order to improve participation among patients with sABI and with no or only little active participation.

To clarify the discussion of patient-related outcomes of health professionals’ education, we distinguished between competencies, experienced applicability and impact on patients, measured by patient participation as defined by the EFA sub-items of items 14, 16, 17, 18 or 19, or a score≥2 on EFA item 20. These items measure patients’ change in behaviour or the observable impact of educational interventions on patient care. However, although patient outcomes are obviously important, they should not be the sole focus of attention in research into medical education. This perspective was applied both to highlight the limitations of outcomes research in medical education and to offer suggestions to facilitate a proper balance between learner- and patient-centred assessments [30]. Therefore, mixed methods were applied to assess two perspectives (for staff and patients) using three methods: firstly, a survey of professionals’ experience of competence development; secondly, focus group interviews on how health professionals experienced an educational programme; and thirdly, a brief quantitative account of the potentially derived effect on patient functionality.

The survey indicated increasing perceived competence after each completed educational seminar, as well as from before the first seminar to after the last completed seminar. Furthermore, the majority found that the achieved skills were applicable in a clinical context. Questionnaires were sent to the staff by the clinical manager. This could introduce a response bias, since staff could be inclined to answer more positively than they would have if questionnaires were sent by an unbiased individual. However, in the questionnaire it was made clear that no individual response would be revealed to the clinic manager. The above mentioned patterns were supported and elaborated in the statements extracted from the three staff focus group interviews. The proportion of patients achieving active participation increased from 45% before to 75% after the implementation (6 out of 8 patients). Though these results are not statistically significant, they are clinically valuable as they reveal a possible and probable trend.

Measures such as knowledge, skills, attitudes, time and satisfaction were second-tier “process measures”. The study investigated four assumptions: i) health professionals’ perceived level of competence will increase with each measurement period, ii) the health professionals will find the educational programme profitable, iii) the professionals with less perceived competence will reap the greatest benefits, and iv) a potential improvement in competencies for health professionals may improve treatment and thus improve patients’ rehabilitation outcomes. These four assumptions were not directly confirmed; they seemed credible and added relevant perspectives to the evaluation of the implementation effectiveness. Implementation of an educational intervention like the present one requires consideration of several methodological measuring issues (both qualitative and quantitative), including whether it has succeeded “well enough” as is clear from this example. These issues will be further discussed below.

When planning the educational programme, no fixed model for how to translate research into practice was used. However, our focus on early rehabilitation of ABI patients in states of minimal consciousness conducted in the setting of the CEN qualified the present study as a complex intervention [31] adopting a particular approach rather than as a simple intervention that could be equated with an efficacy study as defined by Glasgow et al. [32]. Furthermore, the content of the educational programme was perceived to be a humanistic topic: soft, sensitive, abstract issues translated into concrete actions through interaction with the patient emphasizing practical actions [33]. The study was didactically framed by reflections and discussions about how to learn and tackle various dilemmas and issues arising in practice. Therefore, situated learning [17] was chosen as the learning strategy. Almost all of the informants from the focus group interview expressed that they had benefitted from the course. Everyone agreed that it was fruitful to have bedside learning and feedback included in the course planning, so that the key points came to life in their practice. Thus, situated learning as a learning strategy seems well chosen, relevant and rewarding, and this was an important aspect of implementation effectiveness. However, several aspects relating to the evaluation of implementation effectiveness should be discussed.

4.4.1 Strengths and limitations

According to Glasgow [32 –34], a well-designed evaluation framework is characterised by five key characteristics: Reach, Effectiveness, Adoption, Implementation, and Maintenance (acronym RE-AIM). The implementation process was performed in the natural social and environmental context where staff normally perform their practice, and the process did reach (the first key characteristics of a well-designed evaluation framework) the entire staff at the ward, i.e. a broad, heterogeneous and representative and diverse sample with respect to clinical experience, professions, gender (more female than male, but representative for the staff). However, not all staff members were able to attend all three seminars due days off after weekend, evening or night shifts. Another important factor was that some of the participants did not receive individual guidance and feedback from the primary educators. Finally, a change of staff occurred during the year when the educational programme was implemented. These factors limited the specialised nurse, OT and PT in supporting the implementation process in the period between the three seminars. Effectiveness pertains to the impact of an intervention on specified outcome criteria and includes measures of potentially negative outcomes as well as intended results [35]. The effect of this educational programme in terms of the specified, selected outcome criteria (EFA sub-items and sum scores) appeared to be positive as the endpoint was increased patient participation. However, the results were not significant, and the study sample was very small. These results show a positive trend; adding the evaluation from the survey and interview findings –the implementation hence seems to be effective and clinically valuable. However, to ensure long-term implementation effectiveness the study should add a later end point (e.g. one year after last educational seminar).

Factors associated with adoption include political and cultural fit, cost, level of resources and expertise required, and how similar a proposed service is to the current practices of an organisation [32]. Framing the educational programme, the initial research [13 –15] was conducted in the same ward and with the same types of patients as that of the intervention itself. The education programme was also based on the department’s resources, culture, expertise and services. All these aspects have contributed to the adoption being relatively smooth. However, it seemed that less experienced and novice staff within this field of early specialised neurorehabilitation benefitted more from the education programme than the more experienced staff. However, the experienced staff members also benefitted, pointing out that they all now have a common foundation, have a stronger awareness and can put issues into words and thus contribute to changing daily practice towards increased active participation of patients with severe acquired brain injury.

Implementation refers to intervention integrity or the quality and consistency of delivery [32]. In this case, there was only positive criticism of the teacher and his way of imparting and massaging new competences into the department. However, some participants challenged the approach chosen and discussed whether the educational programme should have been organised as a top-down model to ensure that a larger group of professionals became highly educated and that the contents and competences were widely implemented at in the department.

Finally, maintenance operates at both the individual and the setting or organisational level. At the individual level, maintenance refers to how well behaviour changes can be maintained in the long term. At the setting level, maintenance refers to the extent to which a treatment or practice becomes institutionalised in an organisation [32]. This study made the final follow-up three months after the final seminar. Evaluating this factor, the study should have conducted another implementation evaluation one year after the final seminar, including interviews with staff as well as patient assessments.

4.4.2 Some considerations above of the conducted materials

Survey: The non-validated questionnaires were e-mailed to all staff members of the department despite absence due to disease, maternity leave, leave of absence or vacation. Hence, the specific number of possible respondents was unknown and so was therefore the response rate. Free answers were secured through anonymity achieved by storing and managing electronic data using RedCap capture tools. Focus group: The characteristics of participants in the survey and interviews may differ as participants in focus group interviews were chosen at random, i.e. who participated was determined by which staff members were at work at specific dates. Hence, informants differed between all three interviews, and the two heads of the department participated in the first interview. This might influence the power in the staff members’ interactions and statements towards more positive vocabularies as the education programme was a management priority. The outcome measure of patient participation was decided by the authors as a surrogate marker, and the results of the EFA assessment must be interpreted with caution as the subjective measurements conducted by the staff might be influenced by the education programme and the daily focus at the ward.

The primary goal of this effectiveness trial was to determine whether an intervention works by measuring its impact when tested within a population representative of the intended target audience [32].

5 Conclusion

Exploring the implementation effectiveness of the educational programme seemed clinically valuable and showed the promising and probable effect of an implementation process.

The current implementation study aimed to document the effect of implementing an educational training programme to promote staff’s ability to maximize minimally conscious patients’ active participation in their own rehabilitation. The implementation process has been successful to some extent. The concept of “implementation outcomes” was useful to conceptualise the study. The mixed methods design to explore both the staff and patient perspective proved to be valuable and recommendable for other implementation studies.

6 Author contributions

The four authors HP (Physiotherapist, Senior Researcher), HH (Occupational Therapist, Phd student), LA (Nurse, Senior Researcher) and SSK (Physiotherapist, Phd) conceptualised the study.

Analysis of survey, HH and SSK; interviewing and analysis of the interviews, HP and LA and analysis of the effect on patients’ functioning, SSK. Writing— original draft preparation, HP; writing— review and editing, all four authors. All four authors were employed as researchers at the Research Unit at the time of the study. The two authors HH and SSK had several years of clinical experience within highly specialized clinical rehabilitation and knew the staff and were therefore not involved in interviewing or analysing the interviews to insure minimising response bias.

All authors have read and agreed to the published version of the manuscript.

Footnotes

Acknowledgments

The authors would like to thank the participants for their time and willingness to share their experiences as part of the present study. They would also like to thank the management and the staff at CEN for their contributions to data collection.

Conflict of interest

The authors declare no conflict of interest.

Funding

This research received no external funding.

References

Sackett

, Rosenberg

, Gray

, Haynes

, Richardson

Evidence based medicine: What it is and what it isn’t. BMJ. 1996;312(7023):71–2. doi: 10.1136/bmj.312.7023.71.

Straus

, Glasziou

, Richardson

, Haynes

Evidence-based medicine: How to practice and teach it (4th ed.). Edinburgh: Churchill Livingstone Elsevier; 2011.

Hoffmann

, Bennett

, Mar

Evidence-based practice across the health professions (3rd ed.). Chatswood (Australia): Elsevier Australia; 2016. pp. 4.

Morris

, Wooding

, Grant

, The answer is 17 years, what is the question: understanding time lags in translational research, J R Soc Med 2011;104(12):510–520.

Epstein

, Hundert

, Defining and assessing professional competence, JAMA 2002;287:226–35.

Travis

, Cross-Disciplinary Competency Standards for Work-Related Assessments: Communicating the Requirements for Effective Professional Practice, ” Work (Reading, Mass.) 2002;19(3):269–80.

Honoré

, et al., “Mind the GaHow Job Task Distributions ofHealth Professional Developers Constitute Evidence-Based Practice, ”Work (Reading, Mass.) 2021;68(1):223–33.

Anderson

, Irwin

, Self-assessment of professionalism in physical therapy education, Work 2013;44(3):275–8110.3233/WOR-121504PMID: 23324679.

Proctor

, Silmere

, Raghavan

, Hovmand

, Aarons

, Bunger

, et al., Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda, Admin Policy Ment Health Ment Health Serv Res 2011;38(2):65–76.

10.

Fixsen

, Naoom

, Blase'

, Friedman

, Wallace

Implementation research: A synthesis of the literature (No. FMHI Publication #231). Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health. Institute, National Implementation Research Network; 2005.

11.

Glasgow

, eHealth evaluation and dissemination research, Am J Prev Med 2007;32(5S):S119–S126.

12.

Pallesen

, Buhl

, Roenn-Smidt

, Early rehabilitation and participation in focus –a Danish perspective on patients with severe acquired brain injury, Eur J Physiother 2016;18(4):233–6. doi: 10.1080/21679169.2016.1189594.

13.

Buhl

, Pallesen

, Early rehabilitation of patients with severe acquired brain injury: Strategies to promote participation. Scandinavian Journal of Occupational Therapy. 2015.

14.

Pallesen

, Buhl

, Interdisciplinary facilitation of the minimal participation of patients with severe brain injury in early rehabilitation, Eur J Physiother 2017;19(1):13–23. doi: 10.1080/21679169.2016.1229027.

15.

Pallesen

, Lund

, Jensen

, Roenn-Smidt

The body participating: a qualitative study of early rehabilitation participation for patientswith severe brain injury and low level of consciousness. European Journal of Physiotherapy. 2018;20(1):2–11. https://doi.org/10.1080/21679169.2017.1347706.

16.

Danish Health Authority. Hjerneskaderehabilitering –en medicinsk teknologivurdering. Copenhagen; Danish Health Authority; 2011. https://www.sst.dk/da/Udgivelser/2011/Hjerneskaderehabilitering–enmedicinsk-teknologivurdering.

17.

Lave

, Wenger

Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press; 1991

18.

Frank

For a sociology of the body: an analytical review. In: Featherstone M, Hepworth M, Turner B, eds. The body. Social process and cultural theory. London: Sage; 1991:36–102.

19.

Engberg

, A Danish national strategy for treatment and rehabilitation after acquired brain injury, J Head Trauma Rehabil 2007;22:221–8.

20.

Borg

, Roe

, Nordenbo

, Andelic

, de Boussard

, Af Geijerstam

, Trends and challenges in the early rehabilitation of patients with traumatic brain injury: a Scandinavian perspective, Am J Phys Med Rehabil 2011;90:65–73.

21.

Wade

, Rehabilitation –a new approach, Part three: theimplications of the theories. Clin Rehab 2016;30(1):3–10.

22.

Lewis

, Mettert

, Dorsey

, Martinez

, Weiner

, Nolen

, et al., An updated protocol for a systematic review of implementation-related measures, Syst Rev. 2018;7:66. https://doi.org/10.1186/s13643-018-0728-3.

23.

Harris

, Taylor

, Thielke

, et al., Research electronic data capture (REDCap)— A metadata-driven methodology and workflow process for providing translational research informatics support, J Biomed Informat. /04/01/ 2009;42(2):377–81.

24.

Harris

, Taylor

, Minor

, et al., The REDCap consortium: Building an international community of software platform partners, J Biomed Informat. /07/01/ 2019;95:103208.

25.

Khadjesari

, Vitoratou

, Sevdalis

, et al., Implementation outcome assessment instruments used in physical healthcare settings and their measurement properties: a systematic review protocol, BMJ Open 2017;7:e017972–81. doi: 10.1136/bmjopen-2017-017972.

26.

Kvale

, Brinkmann

Interviews: Learning the craft of qualitative research interviewing. 2nd edition. Thousand Oaks, CA: Sage; 2008.

27.

Heck

, Steiger-Bachler

, Schmidt

, Early functional abilities (EFA) –eine Skala zur Evaluation von Behandlungsverlaüfen in der neurologischen Frührehabilitation, Neurol Rehabil 2000;6:125–33.

28.

Alvsaeker

, Walther

, Kleffelgård

, et al., Inter-rater reliability of the early functional abilities scale, J Rehabil Med 2011;43:892–9.

29.

Stubbs

, Pallesen

, Pedersen

, Nielsen

Using EFA and FIM rating scales could provide a more complete assessment of patients with acquired brain injury. Disability and Rehabilitation. 2014;36(26):2278–81. https://doi.org/10.3109/09638288.2014.904935.

30.

Elo

, Kyngäs

, The qualitative content analysis process, J Adv Nurs 2008;62(1):107–15.

31.

Bekendtgørelse om ændring af bekendtgørelse omundtagelse fra pligten til anmeldelse af visse behandlinger, somforetages for en privat dataansvarlig [Executive Order amendingExecutive Order exempting from the obligation to notify certainprocesses conducted for a Private Data Controller], Danish ExecutiveOrder 410 of 9 May2012.

32.

Cook

, West

, Reconsidering the Focus on “Outcomes Research” in Medical Education: A Cautionary Note, Acad Med 2013;88(2):162–7.

33.

Craig

, Dieppe

, Macintyre

, Michie

, Nazareth

, Petticrew

, et al., Developing and evaluating complex interventions: the new Medical Research Council guidance, BMJ 2008;337:a1655.

34.

Glasgow

, Lichtenstein

, Marcus

, Why Don’t We See More Translation of Health Promotion Research to Practice? Rethinking the Efficacy-to-Effectiveness Transition, Am J Pub Health 2003;93:8.

35.

Mezirow

. Perspective Transformation, Adult Education 1978;28:100–110.

36.

Glasgow

, Vogt

, Boles

, Evaluating the public health impact of health promotion interventions: the RE-AIM framework, Am J Public Health 1999;89:1322–7.

37.

Glasgow

, McKay

, Piette

, Reynolds

, The RE-AIM frameworkfor evaluating interventions: what can ittell us about approaches to chronic illness management? Patient EducCouns 2001;44:119–27.

38.

Flay

, Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs, Prev Med 1986;15:451–74.