Examining implementation determinants of a culturally grounded,school-based prevention curriculum in rural Hawai‘i: A test development and validation study

Abstract

Background:

This study examined the implementation determinants of a culturally grounded, school-based drug prevention curriculum in rural Hawai‘i. Test development and validation procedures were used to examine the impact of implementation barriers and facilitators of the curriculum in public or charter middle/intermediate schools on Hawai‘i Island.

Method

A five-phase, mixed-methods approach toward test development and validation was used. These phases included item generation (Phase 1), item refinement and selection (Phase 2), item reduction (Phase 3), reliability testing (Phase 4), and validity testing (Phase 5). Educational administrators, teachers, and staff employed by the Hawai‘i State Department of Education (HIDOE) participated in the study.

Results

Phases 1 and 2 yielded 50 implementation barriers and 27 implementation facilitators that were evaluated by 204 HIDOE administrators, teachers, and staff. Factor analysis of the barrier items indicated a four-factor solution: (1) Innovation Barriers, (2) HIDOE State-Level Barriers, (3) Teacher-Level Barriers, and (4) Administrator-Level Barriers. Mean comparisons indicated that several barrier and facilitator items differentiated teachers from administrators in the sample.

Conclusions

This study contributes to the implementation measurement literature, specifically in the areas of mental health and substance use. It also highlights the importance of addressing multiple contextual levels in the implementation of culturally focused prevention interventions.

Plain Language Summary Title

Examining Implementation Barriers and Facilitators for School-Based Prevention in Hawai‘i.

Plain Language Summary

Compared with other major ethnic groups, Native Hawaiian and Pacific Islander (NHPI) youth have substantially higher rates of substance use and higher adverse mental and physical health effects related to their use. Despite these disparities, educational and community-based practitioners have long struggled with implementing and sustaining promising substance use interventions for NHPI youth. This study contributes to our understanding of measuring implementation barriers and facilitators for substance use prevention for NHPI youth in rural school settings. It contributes to the field of health disparities and health equity promotion, by addressing calls for research to understand factors affecting successful implementation of prevention programs. This is critical toward achieving health equity for underrepresented and vulnerable populations, such as NHPI and rural youth.

Keywords

implementation determinants Native Hawaiians Pacific Islanders substance use prevention

Introduction

For many years, Native Hawaiians and Pacific Islanders (NHPIs) have disproportionately suffered from health disparities compared to majority ethnic groups. Compounding this issue, these disparities have been overlooked for years, as they have been masked or underreported through the inclusion of NHPIs within the broader rubric of Asian Pacific Islanders (Kaholokula et al., 2019). Over the past decade, however, significant advances have been made toward preventing several adverse health behaviors and outcomes for NHPIs, including cardiovascular disease, hypertension, and substance use (Boyd et al., 2023; McLean et al., 2021). While these advances are promising, an emerging gap in the NHPI prevention literature is understanding how to implement these interventions in real-world settings to promote their adoption and sustainability over time. Understanding successful implementation, adoption, and sustainability of prevention interventions is essential toward concurrently moving the fields of prevention and implementation science forward (Spoth et al., 2013), and more importantly, is key to achieving health equity for NHPIs (Boyd et al., 2023).

The purposes of this study are (a) to develop a measure of implementation determinants (i.e., implementation barriers and facilitators) for culturally grounded, school-based drug prevention, and (b) to evaluate these determinants in the context of youth attending public and/or charter schools in rural Hawai‘i. Guided by the Consolidated Framework for Implementation Research (CFIR) (Damschroder et al., 2009, 2022b), we used test development and validation procedures to develop an initial inventory of implementation determinants for school-based prevention for rural youth in Hawai‘i. Our study contributes to understanding the measurement of implementation in school-based, rural settings with diverse populations, and points to primary systemic foci impacting successful implementation, adoption, and sustainability of youth-focused innovations over time.

NHPI Youth Substance Use Prevention

Compared with other major ethnic groups, research has shown that NHPI youth have disproportionately high rates of substance use, and associated adverse effects. Specifically, numerous studies have found that Native Hawaiian youth report the highest rate of alcohol, combustible cigarette, e-cigarette, and other drug use among ethnic groups in Hawai‘i (Glanz et al., 2005, 2007; Kim et al., 2007; Nigg et al., 2013; Nishimura et al., 2013; Subica et al., 2020; Wong et al., 2004). These youth also report substance-related adverse effects, such as school absences, suspensions, and infractions (Hishinuma et al., 2006), and suicidal behaviors (Else et al., 2007). To address NHPI youth substance use disparities in Hawai‘i, the National Institute on Drug Abuse supported research over the past two decades focused on the development, evaluation, and implementation of a culturally grounded, school-based substance use prevention curriculum called Ho’ouna Pono (HP). Developed in collaboration with Hawai‘i State Department of Education (HIDOE) public schools on Hawai‘i Island, the teacher-delivered classroom curriculum utilizes realistic, culturally relevant videos to facilitate drug resistance skills training for NHPI middle school-aged youth (Helm & Okamoto, 2013; Okamoto et al., 2014). HP was found to promote NHPI youths’ use of socially competent drug resistance strategies and to reduce their use of combustible cigarettes, e-cigarettes, and hard drugs over time (Okamoto et al., 2016, 2019).

Despite the promising effects of the HP curriculum, it exists within a systemic context that presents unique challenges for its implementation, adoption, and sustainability. The HIDOE is a highly centralized statewide school system that encompasses 255 public and 35 charter schools spread across six different islands. As a result of this structure, anecdotal and empirical evidence from educational and social service administrators throughout Hawai‘i Island have indicated haphazard, fragmented, or poorly planned implementation of school-based drug prevention programs (Waitzfelder et al., 1998). Research is needed to understand the implementation determinants for prevention curricula in rural, diverse contexts, in order to promote their long-term impact in ameliorating youth substance use.

Consolidated Framework for Implementation Research (CFIR)

The present study was conceptually informed by the CFIR. Regarded as one of the most widely used frameworks in implementation science (Damschroder et al., 2022a), the CFIR is based on a synthesis of existing behavior change theories and health care improvement research. Recently updated in 2022 (Damschroder et al., 2022b), the framework encompasses five major domains affecting implementation: (1) Innovation/Intervention, (2) Characteristics of Individuals, (3) Outer Setting, (4) Inner Setting, and (5) Process. It has recently been applied to community-based research on opioid use prevention (Bangham et al., 2023) and school-based nutritional (Meshkovska et al., 2022), educational (Wilhelm et al., 2021), and behavioral health (Hudson et al., 2020) interventions. Thus, the framework provided a clear conceptual lens to examine the implementation determinants of HP in the present study, including those within the school, island/region, and statewide inner setting levels of the HIDOE described in prior research (Okamoto et al., 2020).

Relevance of the Study

This study builds upon the measurement literature in implementation science, particularly in the areas of behavioral health and substance use, which has historically included measures of unknown psychometric quality (Mettert et al., 2020). It contributes to this literature by using test development and validation procedures to systematically examine implementation determinants for substance use prevention with youth in rural Hawai‘i. Furthermore, the present study examines the bidirectional impacts of a rural-, NHPI-focused intervention and its implementation determinants, broadening the contextual lens of implementation to a novel cultural context (Cabassa & Baumann, 2013). Finally, this study directly addresses our Hawai‘i Island educational partners’ expressed need to sustain HP in their public middle and intermediate schools.

Method

Due to the lack of research focused on the measurement of implementation determinants in rural Hawai‘i, this study utilized a five-phase, culturally grounded approach to test development and validation similar to prior research (Okamoto et al., 2010). Qualitative methods were used in Phases 1 and 2 of the research (item generation and item refinement and selection) to create survey items, while quantitative methods were used in Phases 3–5 (item reduction, reliability, and validity) to validate and evaluate these items within the public/charter school context on Hawai‘i Island. All research procedures were approved by the Institutional Review Boards at the University of Hawai‘i at Mānoa and the Hawai‘i Department of Education, and research participants provided written informed consent for their participation in this study. The STROBE reporting guidelines for cross-sectional research studies were used to prepare this manuscript (von Elm et al., 2007).

Authors’ Positionality

This study was conducted by an authorship team of predominantly (90%) community-based participatory researchers of color. Aligned with the demographics of the sample, 80% of the authorship team was of Asian American or Native Hawaiian ancestry. Four of the co-authors (SKO, KHO, SMM, and SKC) have had nearly two decades of experience working with and/or living among rural communities in Hawai‘i, and three of the co-authors (SKO, KHO, and SKC) have had nearly two decades of research and/or practice experience working with the HIDOE.

Phase 1: Item Generation

The goal of this phase was to generate a large, representative pool of potential survey items related to the implementation, adoption, and sustainability of the HP curriculum in HIDOE schools on Hawai‘i Island. Items were created from individual interviews of HIDOE administrators (e.g., principals, vice-principals; n = 5), teachers (n = 7), behavioral health (n = 2), and district-level staff (n = 2) on Hawai‘i Island. These interviews were conducted as part of an implementation, adoption, and sustainability pilot study for HP (Okamoto et al., 2020), in which participants were asked questions that reflected CFIR domains specific to the HIDOE educational setting (see Table 1 for sample questions). Phase 1 participants were selected based on their prior knowledge and experience with the HP curriculum and/or past prevention efforts in HIDOE schools. Ninety-four percent of them had 6 or more years of experience working for the HIDOE (Okamoto et al., 2020). Transcripts from the interviews were re-analyzed to extract barriers and facilitators to the implementation, adoption, and sustainability of HP. Particular attention was given to incorporating the language, phrasing, and constructs specific to the HIDOE faculty, staff, and administrators in constructing the items. This process yielded a total of 62 barrier items and 58 facilitator items from the interview transcripts.

Table 1

Selected Phase 1 Interview Questions and CFIR 2.0 Domains/Constructs

Question	CFIR 2.0 domain/construct
What can be done to further promote Ho’ouna Pono's alignment with teachers’ professional and educational needs?	Innovation/Innovation Adaptability
How do local, state, or national performance measures or policies influence the implementation of Ho’ouna Pono?	Outer Setting/Policies & Laws
Which teacher(s) should oversee the implementation of the curriculum in your school? Why?	Inner Setting/Work Infrastructure
How confident do you feel about implementing Ho’ouna Pono in your classes?	Individuals/Innovation Deliverers, Capability
Who do you see as the lead person implementing the curriculum in your school?	Process/Teaming

Phase 2: Item Refinement and Selection

The goals of this phase were to reduce the total number of barrier and facilitator items to be included in the survey. The total number of barrier items was reduced by eliminating redundant, non-representative, and vague items, and combining similar items described by HIDOE administrators, teachers, and staff. Barriers that were too specific or appeared random in nature were eliminated. For example, one eliminated barrier focused on how the Hawaiian terms used in the curriculum would be confusing for a “first-year transplant” student teacher from the Mainland United States. This item was flagged as a specific and infrequent event; therefore, the situation was deemed “non-representative” and was eliminated from the list of viable items. To control for bias, item refinement and selection were first conducted individually by research team members, followed by team validation process where members justified items for refinement, inclusion, or exclusion from the survey.

This process produced a final list of 50 implementation barriers to HP (see Table 2 for a full list of barrier items). Forty-four percent of the items focused on the HIDOE-level Inner Setting, 22% focused on Individuals, 20% focused on the Innovation (HP), 14% focused on the Outer Setting, and 10% focused on the School-level Inner Setting. HIDOE faculty, staff, and administrators were instructed to use two Likert scales to respond to each item—one that rated the magnitude of impact of the barrier, ranging from 1 (“no impact”) to 5 (“substantial impact”), and the other that rated the difficulty of overcoming the barrier, ranging from 1 (“very easy”) to 5 (“very difficult”). Emphasis was placed on evaluating implementation barriers in this study, based on the long-standing documented challenges in implementing and sustaining substance use prevention interventions in rural Hawai‘i (Waitzfelder et al., 1998), and the need to fully assess multi-level barriers that may be contributing to these issues.

Table 2

Ho’ouna Pono (HP) Implementation Barriers: Item Refinement and Selection

1. Natural disasters on Hawai‘i Island (e.g., lava, hurricanes) disrupt school-based prevention efforts, like HP.

2. HIDOE administrators are resistant to endorsing HP, due to fear of asking overburdened teachers to try something new in the classroom.

3. Some HIDOE teachers are resistant to trying new curricula like HP, because it feels like “one more thing you [administrators] want me to do in the classroom.”

4. Marijuana use is socially acceptable on Hawai‘i Island, diminishing the need for HP.

5. The turnover of state-level administrators will disrupt the ongoing implementation of HP.

6. Teachers may not use the HP curriculum, because they are afraid that it could prompt their students to ask them about their personal drug use histories.

7. There may be a lack of enthusiasm for the HP curriculum because health education is not a standardized or measured content area.

8. The HIDOE emphasis on vocational training diverts school resources away from drug prevention programs, like HP.

9. The frequent turnover of principals in my school will disrupt the ongoing implementation of HP.

10. Substance use is not perceived to be a major problem in my school.

11. HIDOE senior teachers at the top of the salary scale (Class 7) do not need PDE credits associated with implementing the HP curriculum for reclassification.

12. Lack of familiarity with HP makes me hesitant to use the curriculum.

13. The HP curriculum lacks a social media presence (e.g., Twitter, Instagram, Facebook).

14. The HIDOE does not specify the structure or content of substance use prevention in schools.

15. The HIDOE emphasis on college- and career-readiness diverts school resources away from drug prevention programs, like HP.

16. Restrictions on federal funding (e.g., Title 1) make it difficult to support curricula like HP.

17. The HP curriculum introduces language and concepts that may be academically advanced for the sixth-grade level.

18. Some HIDOE teachers feel that pre-packaged curricula like HP threaten their autonomy in the classroom.

19. The nine-lesson HP curriculum has too much drug-specific content to fit within a semester-long health course.

20. Some HIDOE teachers believe that drug prevention curricula like HP have limited impact in promoting internalization of youths’ drug-free attitudes.

21. The relatability of the HP curriculum is limited primarily to sixth and seventh graders. Older youth may find it to be “silly” or too “immature”.

22. Some HIDOE teachers believe that drug use is symptomatic of deeper issues (e.g., depression), which should instead be the focus of prevention.

23. Some HIDOE teachers believe their current drug and alcohol content is effective, so they don’t see any added benefit to the HP curriculum.

24. There may be a lack of enthusiasm for the HP curriculum because health education is not prioritized in the HIDOE.

25. There is a lack of HIDOE funding to support prevention curricula like HP.

26. Drug prevention curricula like HP encourage the discussion of sensitive topics, which could lead to overstepping student–teacher interpersonal boundaries.

27. Qualified adult instructors outside of the HIDOE are not available to teach HP lessons to youth.

28. Incorporating the HP curriculum into my curriculum map will take planning and effort.

29. After 10 years, the HP curriculum may need updating, by changing youths’ language and jargon depicted in the videos.

30. The HP curriculum does not extensively cover current or recent forms of substance use, such as vaping.

31. HIDOE job performance measures (e.g., CESSA) do not explicitly measure growth on student health indicators.

32. The Hawai‘i Board of Education may see HP as redundant to other drug prevention curricula in Hawai‘i Island schools, like DARE.

33. Some families live a “counter-culture” lifestyle, and might not want their children exposed to drug prevention.

34. Shifting National priorities (e.g., bullying) take the focus away from substance abuse prevention programs, like HP.

35. Senior HIDOE teachers are comfortable with their current drug and alcohol lessons, and are therefore not interested in the HP curriculum.

36. HIDOE teachers who prefer didactic teaching methods may not feel comfortable with the interactive approach of the HP curriculum.

37. Health education is not included in the middle school promotion policy, therefore there is no urgency to implement drug prevention curricula like HP.

38. The implementation and training related to the HP curriculum will be difficult to fit into teachers’ busy schedules.

39. The HP curriculum does not specifically address Common Core standards in writing and math.

40. The HIDOE emphasis on “moving” school-wide standards-based test scores detracts from health education and prevention efforts like HP.

41. The HP curriculum would require additional work for teachers in immersion schools who need to translate the lessons into Hawaiian.

42. The structure of the HP curriculum can get repetitive for some students over the course of nine lessons, affecting their engagement.

43. The HP curriculum is structured primarily on skill-building (e.g., decision-making) rather than on teaching standard health topics (e.g., facts about drugs and alcohol).

44. It is difficult for teachers to talk about drug and alcohol use in the classroom setting with their students.

45. Mainstream publications (Honolulu Magazine) do not use health indicators in establishing their annual rank-ordering of public schools.

46. The responsibility for teaching health frequently shifts from one teacher to another teacher in my school.

47. The HIDOE OCISS (State-Level) Health Resource Teacher has not explicitly endorsed the HP curriculum for Hawai‘i Island schools.

48. The HP curriculum introduces terms that might be difficult for ESL students to understand.

49. The delivery of the HP curriculum is assigned to a teacher who is resistant to using the curriculum.

50. The frequent turnover of teachers in my school will disrupt the ongoing implementation of HP.

Note. HIDOE = Hawai‘i Department of Education; PDE = Professional Development & Education; CESSA = Comprehensive Evaluation System for School Administrators; OCISS = Office of Curriculum, Instruction, and Student Support.

To reduce the facilitator items, we selected items that were identified by two or more HIDOE administrators, teachers, or staff. This produced a final list of 27 facilitator items (see Table 3). These items were presented side-by-side on a grid, and survey participants were asked to identify and rank order their top three implementation facilitators to HP, in terms of their ability to successfully promote the implementation, adoption, and/or sustainability of the curriculum in Hawai‘i Island schools. Shortened evaluation processes and measures were used to assess the facilitator items compared to the barrier items, in order to mitigate respondent fatigue. Furthermore, because the survey took less than 30 min to complete, we were able to ensure that it would not compete with participants’ HIDOE contractual job responsibilities.

Table 3

Ho’ouna Pono (HP) Implementation Facilitators: Item Refinement and Selection

1. HP is aligned with Superintendent Kishimoto's emphasis on Student Voice.

2. HP is aligned with the Hawai‘i Content and Performance Standards for Health (6–8).

3. The HP curriculum is developmentally appropriate for middle school-aged youth on Hawai‘i Island.

4. The HP curriculum is free, contingent upon receiving training to use the curriculum.

5. The HP curriculum can help new teachers from the Mainland United States understand and connect with local students.

6. In-person teacher training on the classroom implementation of HP is offered on Hawai‘i Island.

7. The HP curriculum is place-based, in that it is focused on the “local” culture on Hawai‘i Island.

8. The HP curriculum supports and complements socioemotional learning.

9. The videos in the HP curriculum are realistic and relatable to Hawai‘i Island youth.

10. The timing and length of the HP lessons can be flexibly implemented throughout the semester.

11. My district-level health resource teacher recommends the HP curriculum.

12. The structure and content of the HP curriculum are presented in a statewide HIDOE workshop.

13. The HP curriculum was developed through research in HIDOE schools on Hawai‘i Island.

14. My Complex Area Superintendent evaluates and promotes health education in my school complex.

15. The HP curriculum is approved by the HIDOE as part of the middle/intermediate school health curriculum.

16. The HP curriculum fills a need. There are a lack of school-based substance use prevention programs that work.

17. The HP curriculum includes a teacher implementation website, where teachers can help each other online.

18. The HP curriculum focuses on teaching skills to youth, like decision-making and problem-solving.

19. The HP curriculum focuses on culturally appropriate resistance strategies, like avoiding drug-related problem situations.

20. A motivated teacher supports the use of HP.

21. The HP curriculum helps to engage youth in age-appropriate discussions about their feelings and emotions.

22. The HP curriculum promotes students’ critical thinking in difficult social situations.

23. The HP curriculum aligns with HĀ^a, because it contributes toward the development of the “whole child”.

24. My principal supports the use of the HP curriculum.

25. The HP curriculum promotes active student participation (e.g., problem-solving through group discussions).

26. HP can be woven into different educational settings outside of health, such as advisory.

27. The HP curriculum is based on Native Hawaiian culture and values.

Note. HIDOE = Hawai‘i Department of Education.

HĀ is a HIDOE framework focused on developing skills, behaviors, and dispositions that are reminiscent of Hawai‘i's unique context, and that honor the qualities and values of the indigenous language and culture of Hawai‘i.

Phase 3: Item Reduction

Participants

Participants were HIDOE administrators, teachers, and staff who came from 34 geographically dispersed public or charter middle, intermediate, or high schools on Hawai‘i Island, as well as from complex- and state-level offices on Hawai‘i Island and O‘ahu. Two hundred and thirty-seven participants were recruited to participate in the study and opened the online survey. Thirty-three of these surveys were found to be incomplete and non-useable (i.e., no implementation barrier or facilitator items were completed). Thus, survey data from 204 participants were analyzed in Phase 3. The analyzed sample size exceeded the minimum recommendations for exploratory factor analyses conducted in this study (MacCallum et al., 1999). We surveyed a wide variety of the HIDOE public educational workforce, including complex area superintendents, principals, vice-principals, district-level resource teachers, school counselors, and school-based behavioral health staff (see Table 4 for sample demographics). Participants were selected based on their knowledge of and/or responsibility for health promotion and substance use prevention in middle or intermediate schools on Hawai‘i Island. Ninety-three percent of the sample did not have prior direct experience implementing the HP curriculum; however, prior experience with the curriculum was not a prerequisite for participation in the study. Participant recruitment began in 2019–2020 academic school year and was completed in the 2022–2023 academic school year.

Table 4

Demographics of the Sample (N = 204)

Participant characteristics	%	Participant characteristics	%
Gender		HIDOE role
Male	29	Principal/VP^a	11
Female	71	School counselor/behavioral health	14
Age (years)		Superintendent^a	1
18–21	1	Teacher	55
22–30	9	District/complex-level staff	4
31–40	20	Other	9
41–50	33	HIDOE length of employment (years)
51–60	28	0–2	9
61–70	8	3–5	12
71 or more	1	6–10	19
Ethnic identification		11–20	30
Chinese	1	21–30	26
Filipino	7	30 or more	4
Hawaiian/Part Hawaiian	31
Hispanic/Latino/Spanish	5
Japanese	14
Korean	1
Portuguese	4
White	35
Other Asian	1
Other Pacific Islander	2

Note. HIDOE = Hawai‘i Department of Education.

These categories were combined into an Administrator variable that was used in Phase 5 analyses.

Procedures

The survey was administered in an online format. At the beginning of data collection, we administered the survey in person, where we were able to explain the informed consent procedures and purpose of the survey and provide a broad description of the HP curriculum. Participants were also shown a brief introductory video prior to responding to survey questions, which provided a visual overview of the videos and lessons in the curriculum. After the COVID-19 Pandemic forced all HIDOE public and charter schools to close for in-person instruction, we pivoted our survey administration procedures to a virtual (Zoom) format. On average, participants took approximately 24 min to complete the online survey and received a $5.00 gift card as acknowledgment of their time and effort.

Exploratory Factor Analysis

Separate correlation matrices of questionnaire items for the impact and difficulty scales were completed to examine relationships among items. Principal axis factoring with promax rotation was conducted with the full sample on the impact and difficulty scales of the barrier items, in order to identify and validate latent constructs. Due to the lack of variability in rank-ordered items on the Facilitator Scale, we did not factor analyze these items. They were used primarily for construct validation purposes (see Results, Phase 5: Validity). The use of the common factor model rather than principal components analysis was selected to control for the influences of specific variance and error variance on the factor structure. An oblique, versus orthogonal, rotation was selected based on the predicted correlation between potential factors. We anticipated that the factors emerging from the data would conceptually map onto one or more CFIR domains.

The number of factors was determined using a scree plot. Scree plots consist of the magnitude of eigenvalues as a function of their ordinal position (Nunnally & Bernstein, 1994), and using them to determine the number of retained factors in exploratory factor analysis is preferred over other methods (Costello & Osborne, 2005). Because the primary criticism of the use of the scree plot is its reliance on subjectivity (Gorsuch, 1983), several plausible factor solutions were proposed and analyzed based on the scree test in order to determine the most interpretable solution. We applied the .40-.30.-.20 rule to determine which items to retain within factors (Howard, 2016). Items were retained if they (a) loaded onto their primary factor above 0.40, (b) loaded onto alternative factors below 0.30, and (c) demonstrated a difference of 0.20 between their primary and alternative factor loadings. In order to control for sample bias, we employed pairwise deletion of missing survey data.

Phase 4: Reliability

Internal consistency was assessed with Cronbach's alphas for each of the subscales derived from the factor analysis.

Phase 5: Validity

Due to the lack of established measures focused on implementation determinants for school-based prevention, construct validity was established through a hypothesis testing approach (Smith, 2005). A similar approach has been used for construct validation of novel phenomena, such as social workers’ perceptions of high-risk youth (Okamoto, 2001) and culturally specific drug-related problem situations (Okamoto et al., 2010). A body of research has indicated that school administrators and teachers perceive educational constructs differently, including instructional leadership (Konan & Agiroglu, 2022) and school climate (Veletic et al., 2023). These differences emerge from distinct differences in their job roles and responsibilities. As a result, they most likely perceive implementation determinants from their own unique professional experiences and perspectives. In order to test the construct validity of the measure, we examined whether there would be significant differences in the assessment of implementation determinants based on respondents’ job role in the HIDOE. Specifically, compared with teachers, we predicted that HIDOE administrators would perceive administrative-level barriers and facilitators for HP as having a stronger impact and/or to be more difficult to overcome. Furthermore, compared with administrators, we predicted that HIDOE teachers would perceive teacher-level barriers and facilitators for HP as having a stronger impact and/or to be more difficult to overcome. These hypotheses were used to examine the discriminative validity of implementation determinants based on respondents’ job role in the HIDOE, which is an approach that has been used to establish construct validity in clinical psychometric research (Haynes et al., 2019).

Results

Phase 1: Item Generation and Phase 2: Item Refinement and Selection

Faculty, administrators, and educational support staff (e.g., school counselors; N = 16) generated 62 implementation barrier items and 58 implementation facilitator items. These items were systematically reduced in number to 50 barrier and 27 facilitator items (see Tables 2 and 3). The reduced number of barrier and facilitator items were used to evaluate determinants in the context of public/charter schools on Hawai‘i Island in Phases 3–5.

Phase 3: Item Reduction

Descriptive Statistics

For the impact scale, mean scores for the 50 items ranged from 2.23 (item 13) to 3.32 (item 30). All item impact scores were symmetrically distributed and had normal univariate distributions. Twenty-eight percent of the impact items had a median score of “2” (“minimal impact”) and 72% had a median score of “3” (“some impact”). The percentage of respondents who answered “3” or above for each item (indicating at least some impact of the barrier) ranged from 35.5 (Item 2) to 86.0 (Item 3). Six of the items had over 10% of the respondents answer “5” (“substantial impact”; Items 3, 4, 25, 30, 40, and 49).

For the difficulty scale, mean scores for the 50 items ranged from 2.22 (Item 13) to 3.53 (Item 3). All item difficulty scores were symmetrically distributed and had normal univariate distributions. The median difficulty score was “3” (“a little difficult”) for all items, except for item 53 (Mdn = 4 or “moderately difficult”) and item 63 (Mdn = 2 or “easy”). The percentage of respondents who answered “3” or above for each item (indicating at least a little difficulty in overcoming the barrier) ranged from 40.5 (Item 13) to 88.3 (Item 3). Six of the items had over 10% of the respondents answer “5” (“very difficult”; Items 1, 3, 4, 38, 41, and 49).

Exploratory Factor Analysis

The impact and difficulty scales of the 50 barrier items were entered into separate inter-item correlation matrices and were separately factor analyzed. The factor analysis for the impact scale is presented in this study. The factor analysis for the difficulty items is not presented, as it did not yield any interpretable factor structure. This result is consistent with past research that found a lack of factor interpretability based on similar difficulty assessments (Okamoto et al., 2010). The impact items in our study also generated a more complete data set than the difficulty items. The inter-item correlation matrix for the impact scale indicated low to moderate correlations among all items. Most of the items within the matrix had correlations ranging from .10 to .40. A scree plot was generated for the impact scale, which indicated a three-, four-, or five-factor solution (see Figure 1). Among the various solutions, the four-factor solution generated factor loadings that were the most interpretable. Factors 1 through 4 had eigenvalues of 27.54, 5.02, 4.03, and 3.71, respectively. The four-factor solution accounted for 33% of the common variance. Factors 1 through 4 accounted for 23%, 4%, 3%, and 3% of the variance, respectively. Variance explained by each factor was calculated using principal axis factoring with varimax rotation, as an orthogonal rotation method was necessary to calculate variance specific to each factor.

Figure 1

Scree Plot for Factor Analysis of Implementation Barrier Items

A factor solution was created by examining the difference between loadings for each item on each factor, adhering to the .40-.30.-.20 rule (Howard, 2016). Table 5 lists the 23 retained items with their respective factor loadings and communality estimates. Factor 1 was comprised of seven items that focused on innovation barriers. These items include two outer-setting items (34 and 45) that may impact the perceived value of the curriculum. Factor 2 was comprised of five items that focused on HIDOE state-level barriers. Factor 3 was comprised of seven items that focused on teacher-level barriers. Factor 4 was comprised of four items that focused on administrator-level barriers.

Table 5

Factor Loadings and Extracted Communality Estimates (h²) for Retained Items Using Principal Axis Factoring and Promax Rotation

Item		Factor 1	Factor 2	Factor 3	Factor 4	h ²
42.	The structure of the HP curriculum can get repetitive for some students over the course of nine lessons, affecting their engagement.	. 716	−.062	.107	−.034	.539
48.	The HP curriculum introduces terms that might be difficult for ESL students to understand.	.688	.037	.004	−.176	.394
28.	Incorporating the HP curriculum into my curriculum map will take planning and effort.	.659	.006	−.275	.057	.324
34.	Shifting National priorities (e.g., bullying) take the focus away from substance abuse prevention programs, like HP.	.654	.290	−.065	−.227	.494
41.	The HP curriculum would require additional work for teachers in immersion schools who need to translate the lessons into Hawaiian.	.572	.045	−.060	.033	.338
45.	Mainstream publications (Honolulu Magazine) do not use health indicators in establishing their annual rank-ordering of public schools.	.513	.133	−.033	−.076	.289
43.	The HP curriculum is structured primarily on skill-building (e.g., decision-making) rather than on teaching standard health topics (e.g., facts about drugs and alcohol).	.414	−.013	.121	.185	.377
14.	The HIDOE does not specify the structure or content of substance use prevention in schools.	−.011	.748	−.141	.251	.642
15.	The HIDOE emphasis on college- and career-readiness diverts school resources away from drug prevention programs, like HP.	−.045	.646	.139	−.037	.473
37.	Health education is not included in the middle school promotion policy, therefore there is no urgency to implement drug prevention curricula like HP.	.068	.522	.089	.071	.425
40.	The HIDOE emphasis on “moving” school-wide standards-based test scores detracts from health education and prevention efforts like HP.	.288	.546	−.063	.110	.507
25.	There is a lack of HIDOE funding to support prevention curricula like HP.	.112	.492	.027	.059	.368
23.	Some HIDOE teachers believe their current drug and alcohol content is effective, so they don’t see any added benefit to the HP curriculum.	−.239	−.030	. 794	.026	.452
35.	Senior HIDOE teachers are comfortable with their current drug and alcohol lessons, and are therefore not interested in the HP curriculum.	.154	.042	.528	.001	.433
26.	Drug prevention curricula like HP encourage the discussion of sensitive topics, which could lead to overstepping student–teacher interpersonal boundaries.	.117	.039	.480	.116	.419
36.	HIDOE teachers who prefer didactic teaching methods may not feel comfortable with the interactive approach of the HP curriculum.	.190	.062	.457	.008	.401
17.	The HP curriculum introduces language and concepts that may be academically advanced for the sixth-grade level.	.053	.057	.440	.065	.294
32.	The Hawai‘i Board of Education may see HP as redundant to other drug prevention curricula in Hawai‘i Island schools, like DARE.	.115	.159	.434	.009	.385
20.	Some HIDOE teachers believe that drug prevention curricula like HP have limited impact in promoting internalization of youths’ drug-free attitudes.	.148	.020	.419	.090	.343
2.	HIDOE administrators are resistant to endorsing HP, due to fear of asking overburdened teachers to try something new in the classroom.	−.058	−.013	−.093	.660	.353
3.	Some HIDOE teachers are resistant to trying new curricula like HP, because it feels like “one more thing you [administrators] want me to do in the classroom.”	.202	.035	−.172	.584	.433
5.	The turnover of state-level administrators will disrupt the ongoing implementation of HP.	−.202	.224	.068	.482	.284
9.	The frequent turnover of principals in my school will disrupt the ongoing implementation of HP.	−.115	.068	.204	.445	.281

Note. Factor 1 = Innovation Barriers; Factor 2 = HIDOE State-Level Barriers; Factor 3 = Teacher-Level Barriers; Factor 4 = Administrator-Level Barriers. Rotated factor loadings have values ranging from –1 to +1.

Subscale scores for the impact items were calculated using the mean for items within each factor. In order, the means for Subscales 1 through 4 were 2.74, 3.00, 2.58, and 3.04 (SDs = 0.62, 0.75, 0.58, and 0.66, respectively). The inter-subscale correlations derived from the four-factor solution are reported in Table 6. Correlations among subscales ranged from 0.29 (Subscales 3 and 4) to 0.59 (Subscales 1 and 3).

Table 6

Inter-Subscale Correlations Derived From the Four-Factor Solution

	Subscale 1	Subscale 2	Subscale 3	Subscale 4
Subscale 1	1.00
Subscale 2	0.55	1.00
Subscale 3	0.59	0.44	1.00
Subscale 4	0.32	0.35	0.29	1.00

Note. Subscale 1 = Innovation Barriers; Subscale 2 = HIDOE State-Level Barriers; Subscale 3 = Teacher-Level Barriers; Subscale 4 = Administrator-Level Barriers.

Phase 4: Reliability

As a measure of internal consistency, Cronbach's alpha was calculated for each of the subscales derived from the factor analysis of impact scores. Reliability coefficients were .77, .77, .76, and .68 for Subscales 1 through 4, respectively. Coefficients for Subscales 1–3 were in the acceptable range, while the coefficient for Subscale 4 was on the borderline for acceptability (George & Mallery, 2003).

Phase 5: Validity

Independent-sample t-tests were performed on the four subscales, comparing administrators (i.e., superintendents, principals, and vice-principals) with teachers. None of these tests reached significance (ps > .05). Independent-sample t-tests were also performed on the impact and difficulty scales for each item on the survey. Significant differences on the impact scale were found for two items—8 (Vocational training diverts resources from HP) and 15 (college/career-readiness diverts resources from HP)—while significant differences on the difficulty scale were found for seven items—13 (HP lacks a social media presence), 15 (college/career-readiness diverts resources from HP), 16 (Federal funding restrictions), 23 (Teachers prefer their drug/alcohol content), 31 (Job performance measures do not measure health), 34 (Shifting National priorities shift focus from HP), and 48 (HP might be difficult for English as a Second Language [ESL] students, see Table 7).

Table 7

Impact and Difficulty Scale Comparisons Between HIDOE Administrators and Teachers

	Administrators (n = 22)		Teachers (n = 109)
Item	M	SD	M	SD	t	p	Cohen's d
Impact
Item 8 (Vocational training diverts resources from HP)^a	1.81	0.75	2.38	0.971	−2.53	0.01	0.94
Item 15 (College/career-readiness diverts resources from HP)	2.14	0.941	2.78	1.084	−2.57	0.01	1.06
Difficulty
Item 13 (HP lacks a social media presence)	1.6	0.883	2.24	0.99	−2.68	0.01	0.97
Item 15 (College/career-readiness diverts resources from HP)	2.1	0.852	2.96	1.02	−3.53	<.001	1.00
Item 16 (Federal funding restrictions)	2.14	0.94	3.16	0.99	−3.16	0.00	0.98
Item 23 (Teachers prefer their drug/alcohol content)	2.35	0.93	2.83	0.94	−2.11	0.04	0.94
Item 31 (Job performance measures do not measure health)	2.33	1.07	2.86	0.91	−2.36	0.02	0.93
Item 34 (Shifting National priorities shift focus from HP)	2.33	0.97	2.83	1.00	−2.11	0.04	0.99
Item 48 (HP might be difficult for ESL students)	2.38	0.50	2.83	0.91	−3.15	0.02	0.86

Items have been paraphrased.

To examine implementation facilitators, we conducted a frequency count of all 27 facilitators, regardless of their rank selected by the survey participants (i.e., first, second, or third). The five highest-ranked facilitators selected by administrators were 18 (HP teaches decision-making/problem-solving; n = 6), 25 (HP promotes active student participation; n = 6), 7 (HP is place-based; n = 5), 4 (HP is free; n = 4), and 22 (HP promotes critical thinking; n = 4, see Table 3). Three of these facilitators (18, 25, and 22) focused on student behaviors reflected in state-level health standards, such as decision-making, problem-solving, and critical thinking. The five highest-ranked facilitators selected by teachers were 7 (HP is place-based; n = 43), 9 (HP videos are realistic/relatable; n = 26), 3 (HP is developmentally appropriate; n = 23), 4 (HP is free; n = 22), and 10 (HP lessons are flexible; n = 18). Two of these facilitators (3 and 9) focused on social and developmental relatability of the curriculum to Hawai‘i Island youth, while the third (10) focused on the adaptability of the curriculum to teachers’ lesson plans.

Discussion

The primary foci of this study were to develop a measure of implementation determinants for a culturally grounded, school-based substance use prevention curriculum in rural Hawai‘i, and to examine these determinants in the public middle/intermediate school context on Hawai‘i Island. We used test development procedures to systematically generate a preliminary inventory of implementation barriers and facilitators. In Phases 1 and 2, items were developed through a series of interviews with HIDOE administrators, teachers, and staff, and were edited to eliminate redundant or non-representative items. Multiple phases were then used to evaluate content, factor, and construct validity and reliability of the questionnaire. Phase 3 addressed the factor validity of the questionnaire. Four factors emerged from the analysis of the impact scale: (1) Innovation Barriers, (2) HIDOE State-Level Barriers, (3) Teacher-Level Barriers, and (4) Administrator-Level Barriers. Although the communality estimates of items on the survey were modest (mostly within the .3–.5 range), the factor analysis appeared to generate a simple structure with an overdetermination of the four factors (between 4 and 7 items per factor), with relatively high factor loadings. This pointed toward an acceptable factor structure that reflected the population factor pattern (MacCallum et al., 1999). The lack of factor interpretability of the Difficulty Scale may have been the result of the broad range of participants’ perspectives on what makes a barrier “difficult” to overcome. These perspectives are shaped by diverse contextual and individual participant characteristics, including HIDOE job role, school size, amount of school resources and personnel, and community context, which may have impacted assessment of the barriers’ difficulty and factor interpretability. Phase 4 addressed the reliability of the subscales of the survey. Cronbach's alpha scores ranged from .68 to .77, indicating acceptable internal consistency among items within factors (Nunnally & Bernstein, 1994). Finally, Phase 5 examined the construct validity of the survey. While there were no significant differences in the subscale impact scores between administrators and teachers, we did identify multiple items that indicated significant differences between teachers and administrators. Furthermore, highly endorsed facilitator items for administrators and teachers appeared to reflect their reflective role priorities (i.e., administrators’ focus on state-level standards vs. teachers’ focus on student-level needs).

Factor 1 (Innovation Barriers) consisted of seven items focused on barriers to implementing the HP curriculum. These items appeared to draw from CFIR domains beyond the Innovation domain, such as the Outer Setting (Items 34 and 45) or Process (Item 28). Nonetheless, the items collectively appear to reflect different contextual influences that function as barriers to successful implementation of the curriculum. For example, Items 34 and 45 reflect national and state-level influences that diminish the perceived value of the curriculum, while Item 28 is a school-level barrier that impacts the feasibility of using the curriculum.

Factors 2 and 4 appear to reflect different levels of the CFIR Inner Setting within a highly centralized educational system. Factor 2 focused on the state-level setting of the HIDOE, reflecting broad educational policies and emphases across all public schools. These items were at the highest level of the CFIR Inner Setting, impacting the implementation of drug prevention in public schools across all islands/regions. This factor included implementation barriers that reflected competing statewide educational priorities toward prevention and HP, such as preparing students for college- and career-readiness and “moving” standardized test scores, as well as the role of health and substance use prevention in public schools. Factor 4 focused on administrator-level implementation barriers, including role conflicts between administrators and teachers and the turnover of administrators at the state and school levels.

Lastly, Factor 3 focused on the CFIR Individuals Domain. Specifically, this factor focused on teacher-level implementation barriers, such as teachers’ attitudes and beliefs about drug prevention and the content and approach of the HP curriculum, as well as their perceived need for the curriculum. This factor primarily focused on the impact of the innovation deliverers (i.e., teachers) on the implementation of school-based prevention. It emphasizes that teachers’ perceptions about drug prevention and drug use can impact effective implementation of curricula like HP.

The hypotheses tested in this study demonstrated mixed findings. They examined differences in the assessment of implementation determinants based on job role. We predicted that administrator-level respondents would assess Subscale 4 as more impactful and difficult than teachers, while teacher-level respondents would assess Subscale 3 as more impactful and difficult than administrators. These hypotheses were not confirmed. However, comparisons between administrators and teachers on the impact and difficulty scales of several individual barrier items demonstrated a pattern of significant findings. Based on item-level differences (see Table 7), teachers tended to find implementation barriers more impactful and difficult to overcome than administrators, particularly those that emerge from the federal and HIDOE state levels focused on competing educational priorities and initiatives. Competing educational priorities and initiatives are perceived by teachers, but not administrators, as implementation barriers to HP, because teachers struggle to balance the implementation of drug prevention with other competing educational priorities within their curricular planning. Furthermore, an analysis of facilitator items suggested distinct, yet congruent, perspectives from administrators and teachers. Facilitators that focused on the ability for the HP curriculum to improve statewide health standards and metrics were prioritized by administrators, while teachers prioritized facilitators that focused on the relatability and developmental appropriateness of the curriculum to their students. These types of facilitators focused on direct benefits of the curriculum for students, which aligns with teachers’ roles and responsibilities in the classroom. Thus, while subscale analyses were not supportive of construct validity in the present study, the patterning of individual item scores was consistent with HIDOE job roles, providing support for the construct validity of the measure.

Implications for Implementation Science and Practice

This study has implications for the implementation of prevention programs in rural and diverse schools and communities. Understanding context-specific implementation determinants in educational settings will contribute to the long-term adoption and sustainability of evidence-based youth prevention interventions. In the case of HP, addressing implementation barriers at multiple ecosystemic levels (e.g., national, state, and school levels) may help to sustain the use of the intervention within Hawai‘i Island schools. Furthermore, the four-factor structure suggests that addressing barriers and facilitators specific to the CFIR Innovation, Inner Setting, and Individuals domains may help to promote widespread use of the curriculum over time. HP, for example, directly addresses state-level health standards prioritized by HIDOE administrators, as well as engages Hawai‘i Island youth through culturally relevant videos and curriculum content, which are valued by HIDOE teachers (Helm & Okamoto, 2013; Okamoto et al., 2016, 2019). These characteristics target the CFIR Inner Setting and Individuals Domains through addressing barriers in Factors 4 (Administrator-Level Barriers) and 3 (Teacher-Level Barriers), respectively. Barriers in the CFIR Innovation domain (Factor 1) can be addressed by regularly updating the content of the HP lessons and videos while simultaneously promoting their flexibility in implementation around emerging HIDOE teacher and administrator demands. Past research on implementation determinants of school-based drug prevention has emphasized the importance of intervention-context fit, which can be promoted through regular updating of curricular lessons and videos (Eisman, Palinkas, et al., 2022). Past research has also suggested the importance of implementation flexibility in teachers’ perceived acceptability of prevention programs ((Eisman, Kiperman, et al., 2022; Eisman, Palinkas, et al., 2022).

The measure in this study could be used in future evaluation and implementation research with HP. Future research could focus on examining implementation determinants by geographic area, resulting in tailored approaches to overcoming specific barriers and/or promoting specific facilitators by region. With some surface structure adaptation to the items, the measure could also be used to examine the implementation of new or emerging school-based health and/or prevention curricula in the HIDOE and in rural schools in Hawai‘i. Adapted items could be compared to existing pragmatic measures of implementation antecedents like acceptability, feasibility, and appropriateness (Weiner et al., 2017), in order to inform uptake and sustainment of school-based prevention in rural Hawai‘i.

More broadly, the present study informs recent research on multi-level implementation barriers and facilitators for school-based interventions, which have similarly described innovation/intervention (Brown et al., 2023), inner setting (Arnold et al., 2021; Shoesmith et al., 2021), and individual/teacher-level (Arnold et al., 2021; Lawson et al., 2022) determinants and their impact of intervention implementation, adoption, and sustainment. Using a survey methodology and test development and validation procedures, our study validates recent qualitative accounts of school-based implementation barriers, including teachers’ attitudinal and workload issues that adversely impact intervention sustainment (Lawson et al., 2022) and a lack of funding to sustain school-based interventions (Arnold et al., 2021). While prior studies identified and described implementation determinants, the present study moves the field toward the measurement of those determinants with a broad sector of the educational workforce. Regarding future research, the mixed-methods approach used in this study to develop and validate our measure could also be used to examine implementation determinants in other remote and/or under-researched systems and communities.

Limitations of the Study

This study had several limitations. First, this study began prior to the COVID-19 Pandemic, and because HIDOE schools were closed for in-person instruction for approximately 2 years, data collection slowly commenced over a 3-year time frame. This may have not only introduced history effects into the study but also may have changed the validity of several of the survey items over time. Second, a larger sample size may have helped to clarify additional latent CFIR constructs that did not emerge in the present study. Finally, although an oral description and a video from HP were used to orient the participants to the curriculum, the majority (93%) did not have direct experience implementing it in the classroom setting. Replicating this study with participants with direct experience using the curriculum might have yielded different results, particularly within Factor 1 (Innovation Barriers). The present study may have reflected an assessment of anticipated innovation barriers, rather than actual innovation barriers, experienced by HIDOE educators and administrators.

Conclusions

Although there has been progress in the development and testing of preventive interventions for NHPI populations (McLean et al., 2021), there is a lack of understanding of the implementation determinants that promote long-term adoption and sustainability of those interventions. This study examined the implementation barriers and facilitators to a culturally grounded drug prevention curriculum in rural Hawai‘i. The findings of this study suggest that overcoming implementation barriers related to the intervention, the inner setting (including the state and administrative levels), and the individual/teacher level will be important for the successful long-term adoption of HP in rural Hawai‘i. Furthermore, addressing role-specific priority areas for HIDOE administrators and teachers is an important consideration in successful implementation of the curriculum. These findings will help to inform the implementation of similar school-based curricula in rural, diverse settings.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by funding from the National Institute on Drug Abuse (R34 DA046735, L60 DA059132) and the National Cancer Institute (R25 CA244073). Data analysis for this study was supported in part by the National Institute on Minority Health and Health Disparities (P20 MD002316).

ORCID iD

Scott K. Okamoto

References

Arnold

K. T.

Pollack Porter

K. M.

Frattaroli

Durham

R. E.

Clary

L. K.

Mendelson

(2021). Multilevel barriers and facilitators to sustainability of a universal trauma-informed school-based mental health intervention following an efficacy trial: A qualitative study. School Mental Health, 13(1), 174–185. https://doi.org/10.1007/s12310-020-09402-w

Bangham

Cunnington

Fielman

Hurley

Gergerian

Greece

J. A.

(2023). Using formative evaluation of a community-based opioid overdose prevention program to inform strategic communication for adoption, implementation, and sustainability. BMC Public Health, 23(341), 1–18. https://doi.org/10.1186/s12889-023-15229-2

Boyd

R. C.

Castro

F. G.

Finigan-Carr

Okamoto

S. K.

Barlow

Kim

B.-K. E.

Lambert

Lloyd

Zhang

Barksdale

C. L.

Crowley

D. M.

Maldonado-Molina

Obasi

E. M.

Kenney

(2023). Strategic directions in preventive intervention research to advance health equity. Prevention Science, 24(4), 577–596. https://doi.org/10.1007/s11121-022-01462-5

Brown

G. K.

Owens

Richards

Di Folco

Schwannauer

(2023). Facilitators and barriers to the implementation of a school-based intervention for anxiety. Child Psychiatry & Human Development, Advance online publication. https://doi.org/10.1007/s10578-023-01615-9

Cabassa

L. J.

Baumann

A. A.

(2013). A two-way street: Bridging implementation science and cultural adaptations of mental health treatments. Implementation Science, 8, 90. https://doi.org/http://www.implementationscience.com/content/8/1/90.

Costello

A. B.

Osborne

(2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research, & Evaluation, 10(7), 1–9. https://doi.org/10.7275/jyj1-4868

Damschroder

L. J.

Aron

D. C.

Keith

R. E.

Kirsh

S. R.

Alexander

J. A.

Lowery

J. C.

(2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(50), 1–15. https://doi.org/10.1186/1748-5908-4-50

Damschroder

L. J.

Reardon

C. M.

Widerquist

M. A. O.

Lowery

(2022a). Conceptualizing outcomes for use with the consolidated framework for implementation research (CFIR): The outcomes addendum. Implementation Science, 17(7), 1–10. https://doi.org/10.1186/s13012-021-01181-5

Damschroder

L. J.

Reardon

C. M.

Widerquist

M. A. O.

Lowery

(2022b). The updated consolidated framework for implementation research based on user feedback. Implementation Science, 17(75), 1–16. https://doi.org/10.1186/s13012-022-01245-0

10.

Eisman

A. B.

Kiperman

Rupp

L. A.

Kilbourne

A. M.

Palinkas

L. A.

(2022). Understanding key implementation determinants for a school-based universal prevention intervention: A qualitative study. Translational Behavioral Medicine, 12, 411–422. https://doi.org/10.1093/tbm/ibab162

11.

Eisman

A. B.

Palinkas

L. A.

Brown

Lundahl

Kilbourne

A. M.

(2022). A mixed methods investigation of implementation determinants for a school-based universal prevention intervention. Implementation Research & Practice, 3, 1–14. https://doi.org/10.1177/26334895221124962

12.

Else

I. R. N.

Andrade

N. N.

Nahulu

L. B.

(2007). Suicide and suicidal-related behaviors among indigenous Pacific Islanders in the United States. Death Studies, 31(5), 479–501. https://doi.org/10.1080/07481180701244595

13.

George

Mallery

(2003). SPSS for Windows step by step: A simple guide and reference, 11.0 update. Allyn & Bacon.

14.

Glanz

Maskarinec

Carlin

(2005). Ethnicity, sense of coherence, and tobacco use among adolescents. Annals of Behavioral Medicine, 29(3), 192–199. https://doi.org/10.1207/s15324796abm2903_5

15.

Glanz

Mau

Steffen

Maskarinec

Arriola

K. J.

(2007). Tobacco use among Native Hawaiian middle school students: Its prevalence, correlates and implications. Ethnicity & Health, 12(3), 227–244. https://doi.org/10.1080/13557850701234948

16.

Gorsuch

R. L.

(1983). Factor analysis (2nd ed.). Lawrence Erlbaum Associates.

17.

Haynes

S. N.

Smith

G. T.

Hunsley

J. D.

(2019). Scientific foundations of clinical assessment (2nd ed.). Routledge.

18.

Helm

Okamoto

S. K.

(2013). Developing the Ho’ouna Pono substance use prevention curriculum: Collaborating with Hawaiian youth and communities. Hawai‘i Journal of Medicine & Public Health, 72(2), 66–69.

19.

Hishinuma

E. S.

Else

I. R. N.

Chang

J. Y.

Goebert

D. A.

Nishimura

S. T.

Choi–Misailidis

Andrade

N. N.

(2006). Substance use as a robust correlate of school outcome measures for ethnically diverse adolescents of Asian/Pacific Islander ancestry. School Psychology Quarterly, 21(3), 286–322. https://doi.org/10.1521/scpq.2006.21.3.286

20.

Howard

M. C.

(2016). A review of exploratory factor analysis decisions and overview of current practices: What we are doing and how can we improve? International Journal of Human-Computer Interaction, 32(1), 51–62. https://doi.org/10.1080/10447318.2015.1087664

21.

Hudson

K. G.

Lawton

Hugh-Jones

(2020). Factors affecting the implementation of a whole school mindfulness program: A qualitative study using the consolidated framework for implementation research. BMC Health Services Research, 20(133), 1–13. https://doi.org/10.1186/s12913-020-4942-z

22.

Kaholokula

J. K.

Okamoto

S. K.

Yee

B. W. K.

(2019). Special issue introduction: Advancing Native Hawaiian and other Pacific Islander health. Asian American Journal of Psychology, 10(3), 197–205. https://doi.org/10.1037/aap0000167

23.

Kim

S. S.

Ziedonis

Chen

(2007). Tobacco use and dependence in Asian American and Pacific Islander adolescents: A review of the literature. Journal of Ethnicity in Substance Abuse, 6(3), 113–142. https://doi.org/10.1300/J233v06n03_05

24.

Konan

Agiroglu

B. A.

(2022). A meta-analysis: School administrators’ and teachers’ views on the level of administrators’ instructional leadership behaviours. International Journal on Livelong Education and Leadership, 8(2), 23–38. https://doi.org/10.25233/ijlel.1186444

25.

Lawson

G. M.

Owens

J. S.

Mandell

D. S.

Tavlin

Rufe

Power

T. J.

(2022). Barriers and facilitators to teachers’ use of behavioral classroom interventions. School Mental Health, 14(4), 844–862. https://doi.org/10.1007/s12310-022-09524-3

26.

MacCallum

R. C.

Widaman

K. F.

Zhang

Hong

(1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99. https://doi.org/10.1037/1082-989X.4.1.84

27.

McLean

K. L.

Hata

Marshall

S. M.

Okamoto

S. K.

(2021). Deep-structure adaptations and culturally grounded prevention interventions for Native Hawaiians: A systematic review of the literature. Journal of Racial and Ethnic Health Disparities, 8(3), 570–578. https://doi.org/10.1007/s40615-020-00815-6

28.

Meshkovska

Scheller

D. A.

Wendt

Jilani

Scheidmier

Stratil

J. M.

Lien

(2022). Barriers and facilitators to implementation of direct fruit and vegetable provision interventions in kindergartens and schools: A qualitative systematic review applying the consolidated framework for implementation research (CFIR). International Journal of Behavioral Nutrition and Physical Activity, 19(11), 1–19. https://doi.org/10.1186/s12966-022-01246-8

29.

Mettert

Lewis

Dorsey

Halko

Weiner

(2020). Measuring implementation outcomes: An updated systematic review of measures’ psychometric properties. Implementation Research & Practice, 1, 1–29. https://doi.org/10.1177/2633489520936644

30.

Nigg

C. R.

Anderson

J. K.

Troumbley

Alam

M. M.

Keller

(2013). Recent trends in adolescent alcohol use in Hawai‘i: 2005-2011. Hawai‘i Journal of Medicine & Public Health, 72(3), 92–98.

31.

Nishimura

S. T.

Hishinuma

E. S.

Goebert

(2013). Underage drinking among Asian American and Pacific Islander adolescents. Journal of Ethnicity in Substance Abuse, 12(3), 259–277. https://doi.org/10.1080/15332640.2013.805176

32.

Nunnally

J. C.

Bernstein

I. H.

(1994). Psychometric theory (3rd ed.). McGraw-Hill.

33.

Okamoto

S. K.

(2001). Instrument development: Practitioner fear of youthful clients: An instrument development and validation study. Social Work Research, 25(3), 173–184. https://doi.org/10.1093/swr/25.3.173

34.

Okamoto

S. K.

Helm

Chin

S. K.

Hata

Okamura

K. H.

(2020). The implementation of a culturally grounded, school-based, drug prevention curriculum in rural Hawai‘i. Journal of Community Psychology, 48(4), 1085–1099. https://doi.org/10.1002/jcop.22222

35.

Okamoto

S. K.

Helm

Giroux

Edwards

Kulis

(2010). The development and initial validation of the Hawaiian youth drug offers survey (HYDOS). Ethnicity & Health, 15(1), 73–92. https://doi.org/10.1080/13557850903418828

36.

Okamoto

S. K.

Helm

Pel

McClain

Hill

A. P.

Hayashida

J. K. P.

(2014). Developing empirically based, culturally grounded drug prevention interventions for Indigenous youth populations. The Journal of Behavioral Health Services & Research, 41(1), 8–19. https://doi.org/10.1007/s11414-012-9304-0

37.

Okamoto

S. K.

Kulis

S. S.

Helm

Chin

S. K.

Hata

Lee

(2019). An efficacy trial of the Ho’ouna Pono drug prevention curriculum: An evaluation of a culturally grounded substance abuse prevention program in rural Hawai‘i. Asian American Journal of Psychology, 10(3), 239–248. https://doi.org/10.1037/aap0000164

38.

Okamoto

S. K.

Kulis

Helm

Lauricella

Valdez

J. K.

(2016). An evaluation of the Ho’ouna Pono curriculum: A pilot study of culturally grounded substance abuse prevention for rural Hawaiian youth. Journal of Health Care for the Poor and Underserved, 27(2), 815–833. https://doi.org/10.1353/hpu.2016.0061

39.

Shoesmith

Hall

Wolfenden

Shelton

R. C.

Powell

B. J.

Brown

McCrabb

Sutherland

Yoong

Lane

Booth

Nathan

(2021). Barriers and facilitators influencing the sustainment of health behaviour interventions in schools and childcare services: A systematic review. Implementation Science, 16, 62. https://doi.org/10.1186/s13012-021-01134-y

40.

Smith

G. T.

(2005). On construct validity: Issues of method and measurement. Psychological Assessment, 17(4), 396–408. https://doi.org/10.1037/1040-3590.17.4.396

41.

Spoth

Rohrbach

L. A.

Greenberg

Leaf

Brown

C. H.

Fagan

Catalano

R. F.

Pentz

M. A.

Sloboda

Hawkins

J. D.

, & Society for Prevention Research Type 2 Translational Task Force Members and Contributing Authors. (2013). Addressing core challenges for the next generation of type 2 translation research and systems: The translation science to population impact (TSci impact) framework. Prevention Science, 14(4), 319–351. https://doi.org/10.1007/s11121-012-0362-6

42.

Subica

A. M.

Guerrero

L.-T.

Aitaoto

Iwamoto

Moss

H. B.

(2020). Electronic cigarette use and associated risk factors in U.S.-dwelling Pacific Islander young adults. Substance Use & Misuse, 55(10), 1702–1708. https://doi.org/10.1080/10826084.2020.1756855

43.

Veletic

Price

H. E.

Olsen

R. V.

(2023). Teachers’ and principals’ perceptions of school climate: The role of principals’ leadership style in organization quality. Educational Assessment, Evaluation, and Accountability, Advance online publication. https://doi.org/10.1007/s11092-023-09413-6

44.

von Elm

Altman

D. G.

Egger

Pocock

S. J.

Gotzsche

P. C.

Vandenbroucke

J. P.

(2007). The strengthening the reporting of observational studies in epidemiology (STROBE) statement: Guideline for reporting observational studies. PLoS Medicine, 4(10), e296. https://doi.org/10.1371/journal.pmed.0040296

45.

Waitzfelder

B. E.

Engel

C. C.

Gilbert

F. I.

(1998). Substance abuse in Hawaii: Perspectives of key local human service organizations. Substance Abuse, 19(1), 7–22. https://doi.org/10.1080/08897079809511369

46.

Weiner

B. J.

Lewis

C. C.

Stanick

Powell

B. J.

Dorsey

C. N.

Clary

A. S.

Boynton

M. H.

Halko

(2017). Psychometric assessment of three newly development implementation outcome measures. Implementation Science, 12, 108. https://doi.org/10.1186/s13012-017-0635-3

47.

Wilhelm

A. K.

Schwedhelm

Bigelow

Bates

Hang

Ortega

Pergament

Allen

M. L.

(2021). Evaluation of a school-based participatory intervention to improve school environments using the consolidated framework for implementation research. BMC Public Health, 21(1615), 1–14. https://doi.org/10.1186/s12889-021-11644-5

48.

Wong

M. M.

Klingle

R. S.

Price

R. K.

(2004). Alcohol, tobacco, and other drug use among Asian American and Pacific Islander adolescents in California and Hawaii. Addictive Behaviors, 29(1), 127–141. https://doi.org/10.1016/S0306-4603(03)00079-0