Abstract
Abstract
Background:
End-of-Life Nursing Education Consortium (ELNEC) train-the-trainer workshops prepare participants to teach ELNEC content to others. In 2010, researchers in Japan developed and validated the End-of-Life Nursing Education Questionnaire (ELNEQ) to measure the impact of ELNEC workshops on participants' readiness to teach the content.
Objective:
The study's objective was to evaluate the psychometric properties of the English version of the ELNEQ.
Design and Subjects:
The study was a pre-test/posttest cross-sectional survey design with 113 participants.
Measurements:
The English version of the ELNEQ is composed of 20 items measuring five domains related to teaching end-of-life (EOL) content: confidence in teaching; motivation for teaching; preparedness to teach; preparedness to lead initiatives; and expected influence on participants.
Results:
Reliability was established for all domains (Cronbach's α±range 0.83–0.99) and the scale as a whole (Cronbach's α±= 0.92), as was content validity at the item level. Confirmatory factor analysis showed excellent fit for both the first order and second order factor models. Sensitivity was confirmed as comparisons between pre- and posttest results on all scales and overall were significant.
Conclusions:
The English version of the ELNEQ is a psychometrically sound instrument for measuring the impact of ELNEC train-the-trainer workshops.
Introduction
ELNEC Core, the initial curriculum, consisted of nine modules: nursing care at end of life; 7 pain management; symptom management; ethical/legal issues; cultural/spiritual considerations; communication; grief/loss/bereavement; achieving quality care; and care at the time of death. Subsequently, the core curriculum was modified to address specific populations including pediatrics, 2 oncology, 3 geriatrics,4,5 critical care, and veterans. ELNEC workshops have been developed for both graduate 6 and undergraduate 7 nursing faculty in an effort to improve EOL education within academic nursing institutions. Three conferences were directed towards continuing education and staff development educators. 8 Realizing the need for EOL nursing education worldwide, the ELNEC international curriculum was developed. 9
Thus far, evaluation of ELNEC's impact has been achieved via participants' evaluation of the workshop attended, subsequent tracking of their ELNEC-related endeavors, and a knowledge assessment. Train-the-trainer programs have received outstanding evaluations from attendees and have resulted not only in dissemination of the content to other learners, but also in a multitude of projects directed at improving systems of care and the quality of life for those experiencing terminal illness and their loved ones.1,10 A 109-item knowledge assessment tool was developed and a shorter version (50 items) validated. 11 The validated version can be used pre- and post-ELNEC training to measure changes in participants' knowledge whether or not they intend to teach the content to others. While this tool is useful in assessing knowledge gained related to EOL nursing, it does not measure the impact of the train-the-trainer curriculum on the attendees' self-efficacy related to teaching EOL care to others.
Bandura's theory of self-efficacy has important implications with regard to confidence in teaching, motivation to teach, and preparation to lead initiatives in EOL care. A basic tenet of his theory is that people are likely to engage in activities to the extent that they perceive themselves to be competent at those activities.12,13 Mastering content does not guarantee one's ability or confidence to teach that same content. There exists a growing body of research on teacher self-efficacy.14,15 Skaalvik and Skaalvik 14 conceptualized teacher self-efficacy as “individual teachers' beliefs in their own ability to plan, organize, and carry out activities that are required to attain given educational goals.” Because ELNEC train-the-trainer workshops are directed towards enabling attendees to teach others, it is important to know whether or not such programs actually improve confidence and ability to teach others. Therefore, a tool to measure such impact is merited.
The ELNEQ questionnaire evaluates the magnitude of ELNEC train-the-trainer participants' changes in confidence, motivation, preparation to teach and lead initiatives in palliative care nursing, and expected influence of their teaching. Between January 2008 and February 2010, the ELNEQ was developed and validated in Japanese by Takenouchi, Miyashita, Tamira, Kizawa and Kosugi. 16 Forty items were initially developed based on the ELNEC Core course objectives, reviews of related literature, and input from focus groups of participants who attended ELNEC Japan as part of a faculty development program. The initial item pool was based upon focus group feedback related to the course objectives and participants' experience in the program. The questionnaire was then administered to 143 ELNEC Japan participants at three intervals: three weeks prior to attending a course, immediately prior to the first session of the course, and immediately after the last session of the course. Participants who completed 90% or more of the items were included in the study. Eight experts in palliative care reviewed the items for content and face validity. Items considered redundant and those with factor loadings less than 0.6 were eliminated. The remaining 20 items constituted the ELNEQ.
To date, there has been no published validation of the instrument in the United States. In November 2011, a group of Kentucky ELNEC trainers held an ELNEC train-the-trainer workshop in Louisville, Kentucky, attended by 156 professionals. As part of this effort, the ELNEQ was completed by attendees before and after the workshop both to evaluate the workshop impact and validate the ELNEQ in an English speaking population. The study was approved by the Human Subjects Protection Office of the University of Louisville prior to its initiation. This article presents the results of the psychometric testing of this instrument in a sample of ELNEC workshop attendees in the United States.
Methods
Workshop description
In spring of 2011, a group of 10 ELNEC trained nurses joined together for the purpose of organizing a train-the-trainer workshop to be offered to interested participants. The effort was begun with some seed monies from Norton Healthcare, Inc. and the University of Louisville Interdisciplinary Program for Palliative Care and Chronic Illness. The offering was advertised through hospitals and hospices in the state and via websites, newsletters, and email address lists of specialty nursing organizations. The offering was also posted on the ELNEC national website. Local sponsors were recruited in an effort to keep participant costs at a minimum. There were 156 participants registered for the two-day workshop, which was held at a University of Louisville conference site. All nine ELNEC Core modules were taught.
Data collection
All workshop participants received two copies of the ELNEQ along with other workshop materials as they registered at the workshop. A preamble consent form accompanied the questionnaires stating that completion of the questionnaires indicated consent to participate in the study. Participants submitted their questionnaires (pre- and post-) at the completion of the workshop. Data was manually entered into SPSS 20.0 (SPSS Inc., Chicago, IL).
Instrument
The Japanese researchers granted permission for the use of the ELNEQ in this study. The instrument was translated into English by professional translators and back-translated by one of the researchers to insure its fidelity. Besides translation, there were no modifications made to the Japanese instrument.
In the initial validation of the Japanese ELNEQ, five domains emerged: confidence in teaching; motivation for teaching; preparation to provide teaching; preparation to lead initiatives in EOL care; and expected influences on participants. Internal consistency was supported by Cronbach's alphas ranging from 0.84 to 0.97. Intraclass correlation coefficients for test-retest reliability were 0.63 to 0.76. All domains except motivation for teaching showed moderate to significant changes from pre- to posttest, supporting the sensitivity of the instrument. 16
Statistical analysis
The goal of the current study was to repeat the validation study of the ELNEQ in an ELNEC train-the-trainer workshop conducted in the United States, testing reliability of the domains with an internal consistency measure and using confirmatory factor analysis to test the validity of the five original domains in a different sample. As in the original study, only participants who responded to 90% or more of the 20 items were included as participants in the analysis. Also, sensitivity was enhanced by taking into account the ceiling effect. To account for this effect, respondents were excluded who responded to more than 90% of all items with either 1 or 5 on the five-point rating scale (1=not at all; 5=very much). The confirmatory factor analysis as well as the reliability testing was done on the pretest data.
The sensitivity of the instrument to detect changes between scores collected before and after the program was tested with repeated ANOVA analyses on the five domains, with the partial eta square used to determine effect size. For the partial eta square, a value of 0.01, 0.06, and 0.14 are considered small, medium, and large effects. All statistical analyses were conducted using IBM SPSS 20.0 and IBM SPSS AMOS 20.
Results
Respondents
Of the 156 ELNEC participants, a majority were nurses (91%, n=142); the others were chaplains (4%, n=6), social workers (2.5%, n=4), and others (2.5%, n=4). They were predominantly female (96%, n=150). From this group, a sample of 124 respondents completed the pretest for a response rate of 79%. Of these, 11 were excluded from the analysis due to either missing data or ceiling effects for a final sample of 113. Specific demographics of study participants cannot be reported, as surveys were not individually linked to a particular participant; but it is likely that the sample is representative of the total group.
Psychometric results
Reliability was investigated together with content validity on the item level. Reliability is concerned with how consistent the items are in measuring a specific domain, and was established by calculating the Cronbach's alpha coefficient. A coefficient above 0.80 was seen as acceptable in terms of reliability estimates. Content validity is concerned with whether the items actually measure the full range of meanings associated with a certain domain. 17 Content validity at the item level was established for all constructs by examining the inter-item correlation matrix and making sure they were >0.30, examining the corrected item-total correlations to assure they were >0.45, and determining the means of all corrected item-total correlations to assure that they were >0.50. 18 The results are shown in Table 1.
It is clear that the ELNEQ domains showed high internal consistency and appeared to consistently measure the meaning underlying each domain.
After the reliability and content validity on the item level was completed, the ELNEQ was subjected to a first order confirmatory factor analysis (CFA), using AMOS 20 (SPSS, Inc., Chicago, IL) to determine if the responses to the ELNEQ could be explained by the five domains. It was a priori assumed that each item would have a nonzero loading on the domain it was designed to measure, and zero loadings on all other factors. It was further hypothesized that all five domains would be correlated, and that the error terms associated with the item measurements would be uncorrelated.19,20 Model evaluation was done by first reviewing the parameter estimates in terms of their feasibility, appropriateness of their standard errors, and their statistical significance. The model as a whole was evaluated with different goodness of fit statistics as recommended by Byrne23 (the relative likelihood ratio (χ2/df) below 3; the comparative fit index (CFI) >0.92; the root mean square error of approximation (RMSEA) below 0.06). Model misspecification was detected by investigating the standardized residuals and modification indices. The standardized residuals represent estimates of the number of standard deviations the observed residuals are from the zero residuals that would exist if the model was a perfect fit. Values above 2.58 were considered large. 21 The modification indices provide information as to how much the χ2 would drop if the parameter was freely estimated. The focus here was on error covariances and regression weights. Additional parameters based on model misspecification were only included if they were substantively meaningful. Also, additional items were only removed if the standardized residuals showed clear evidence of model misspecification. 19
Table 2 shows the goodness of fit statistics for the first order CFA analysis of the ELNEQ. Model A shows the model fit based on the original assumptions. Model B shows the model fit after model misspecification was detected in the standardized residuals and modification indices. For the confidence domain, error covariance between items 3 (It is not difficult for me to teach EOL care) and 4 (I have no trouble teaching EOL care) were allowed due to the overlap in item content. For the domain preparation to lead initiatives in EOL care, error covariance between items 1 (I am prepared to contribute to improving the quality of EOL care in my institution or region) and 3 (I am prepared to make plans to improve the quality of EOL care in my institution or region) were allowed due to the overlap in item content. These corrections significantly improved model fit based on the Browne-Cudeck Criterion (BCC) that showed a significant decrease in value from Model A to Model B. The final model, with the standardized estimates, is shown in Figure 1.
First order confirmatory factor analysis model for ELNEQ.
Criteria for good fit – relative likelihood ratio (χ2/df) <3.
Comparative Fit Index (CFI) >0.92.
Root Mean Square Error of Approximation (RMSEA) <0.06.
Criteria for improved fit – Browne-Cudeck Criterion (BCC for Model B<BCC for Model A).
A second order CFA model was tested for ELNEQ, to test the assumption that responses to the ELNEQ could be explained by the five first order domains and one second order factor that could be seen as a thorough evaluation index of the ELNEQ program. The goodness of fit statistics for the second order CFA model C showed good fit, similar to the first order CFA model, with no changes needed to the model to improve fit or correct structural errors (χ2/df=1.25; CFI=0.98; RMSEA=0.05). The second order CFA model for the evaluation index of the ELNEC workshop is shown in Figure 2.
Second Order Confirmatory Factor Analysis Model for ELNEQ.
All of the domains contributed significantly to the second order evaluation index, with the preparation to provide teaching and preparation to lead initiatives in EOL care domains contributing the most and the motivation for teaching and expected influence on participants domains contributing the least. The results of the reliability and content validity for this second order 20 item evaluation index of the ELNEC workshop is shown in Table 3.
It is clear that the ELNEQ showed high internal consistency and seemed to relatively consistently measure the meaning underlying the second order evaluation index. Some items did not perform as well as others, especially two items in the confidence domain and all items in the motivation domain. However, as part of an overall evaluation index they were not seen as problematic, mainly due to a mean Item-Total correlation that was above the 0.50 threshold and a mean Inter-Item correlation that were above the 0.30 threshold.
Sensitivity responsiveness
As shown in Table 4, respondents reported improvements in all five domains as well as the overall scale, with large effect sizes reported for all changes. The biggest changes were seen in the total ELNEQ mean scores, the preparation to provide teaching and the preparation to lead initiatives in EOL care domains, with the smallest changes in the motivation for teaching and expected influence on participants domains.
p<0.01.
Discussion
This study found the English version of the ELNEQ to be valid, reliable, and sensitive to change in this sample of ELNEC Core train-the-trainer participants. Error covariance between two item pairs indicates that the two items may be repetitive to English-speaking participants. Perhaps the meaning of these items differed before translation into English. Future testing of the instrument might consider using only one of the items, or referring back to the Japanese version to determine if the difference between the items got lost in translation.
Three of the domains—preparation to provide teaching, preparation to lead initiatives, and confidence in teaching—demonstrated the most change (pre- versus post-) in this sample population. The content of the train-the-trainer workshop was directed towards preparing participants to subsequently develop and teach workshops on their own; the significant changes in pre- and posttest scores on these domains reflect accomplishment of this purpose.
Motivation to teach and expected influence domains showed less (although still significant) change in the pre- to posttest comparison. This is to be expected, as these participants were obviously motivated as they self-selected to attend the workshop, and their motivation derived from a belief that ELNEC Core workshops could influence participants.
This instrument measures attributes (confidence, motivation, preparation to lead and teach, expected influence) related to teaching ELNEC content rather than knowledge related to the ELNEC content; therefore, it can be used with all ELNEC train-the-trainer curricula. Because train-the-trainer workshops focus on preparing attendees to teach the content, this instrument can be very useful in evaluating the participants' progress toward independent teaching. Those conducting train-the–trainer workshops and wishing to measure the impact on teaching can use the English version of the ELNEQ with confidence. As ELNEC is now offered internationally, other countries using the curriculum would benefit from translation and validation of the instrument in other languages.
Future research should couple the results of pre-/post-ELNEQ results with evaluations of actual teaching outcomes to verify actual improvements in teaching. Also, future studies might include a comparison group of nonworkshop participants to ascertain initial differences on the instrument between those motivated and interested in attending a workshop to improve their teaching abilities and those not planning to attend such an offering.
Limitations of the study include potential validity issues related to the translation of the ELNEQ from Japanese to English, the small sample size, homogeneity of the sample in terms of gender and profession, self-selection of participants, and lack of descriptive variables (i.e., years of experience, current position, age) linked to the data for further analysis. Also, cultural differences between Japan and the United States could impact individuals' responses to and interpretation of the ELNEQ items and limit the generalizability of study results.
Conclusion
The English version of the ELNEQ is a psychometrically sound instrument for measuring the impact of ELNEC train-the-trainer workshops on participants' confidence, motivation, preparation to teach and lead initiatives, and expected influence when teaching end-of-life nursing care.
Footnotes
Acknowledgments
Thank you to Sayaka Takenouchi and the Japanese team of researchers who granted permission for the use of the English version of the ELNEQ for this study.