The impact of the Self-Determined Career Development Model on self-determination

2.1 Sample

After receiving IRB approval, an email was to all community support provider organizations in the state of Missouri through a statewide network describing the project, and interested organizations were invited to contact the research team. Twenty-three provider organizations originally contacted the authors, and 22 initially agreed to participate and were randomly assigned to the SDCDM treatment or control (i.e., business as usual) condition. Each provider organization had between 1 and 6 staff people participating as facilitators (described subsequently), with 38 total facilitators participating. Table 1 provides data on the characteristics of the staff facilitators across the control and treatmentgroup.

2.2 Sample of people with disabilities

Across the 21 provider organizations, 197 participants were identified for the study. Participants with disabilities were selected by organizations based on an identified need for supports to enhance self-determination and career development; no criteria about disability label or level of functioning was used in selecting participants leading to a heterogeneous group of participants, representative of those served by provider organizations. The number of participants per provider organization varied, with a range of 1 to 36 participants. Approximately 75% of provider organizations implemented the SDCDM with 10 or fewer participants. Of the 197 participants, 117 were in the SDCDM treatment group and 80 in the business as usual control group.

Slightly more than half of the participants were male (54%), and the majority of participants (87%) reported their race as White. The average age was 34.5 (SD = 13.04) with a range of 17 to 75. Participant disability category was coded into seven categories: intellectual disability (56%), autism spectrum disorder (9%), learning disability (8%), emotional and behavioral disorders (9%), other health impairments (5%), speech and/or hearing disability (2%), traumatic brain injury (2%), and unknown (13%). Of the participants with an intellectual disability, all had mild to moderate impairments in intellectual functioning. A majority of the sample (71%) did not list a secondary disability. Further details on the sample is provided in Table 2.

2.3 Procedures

After agreeing to participate, support provider organizations were randomly assigned to the SDCDM treatment or business as usual control group. Organizations, therefore, were the unit of random assignment, and participants with disabilities were nested within providers and then facilitators within those organizations. Organizations that were randomly assigned to the SDCDM treatment group identified facilitators from their agency, and facilitators received training and began implementing the SDCDM with target participants. Providers randomly assigned to the control group also identified facilitators, but these facilitators only received training on the data collection activities that were to take place. For both the control and treatment groups, baseline data was collected (described subsequently) prior to implementation of the SDCDM, and follow-up data were collected two more times to identify changes in self-determination as a function of the exposure to the SDCDM. Facilitators in the SDCDM treatment group worked with participants on at least four goals over approximately a one-year period. The first follow-up data collection point (follow-up one) occurred after the first two goals set with the SDCDM were attained, and the second follow-up (follow-up two) after the final two goals set with the SDCDM were attained. Goal attainment was defined by participants completing the three phases of the SDCDM and determining during the evaluation phase (phase three) they had met their goal (see SDCDM section below). Participants in the control group worked on goals that were set through their Individualized Service Plan (ISP), and attainment was determined based on the standards set in the ISP. Thus, the timeline for data collection was determined based on goal attainment, which varied based on participant progress. The average time between baseline and the first follow-up was 6.8 months (SD: 1.7; Range 3.7–13.5), and the average time from baseline to the second follow up was 16.3 months (SD: 1.8; Range 12.1–20.8). Members of the research team remained in monthly contact with facilitators throughout the entire implementation and data collection cycle. Specifically, a monthly email was sent to facilitators by project staff offering technical assistance and troubleshooting of problems that emerged in implementation of the model for the treatment group or in data collection for the treatment and control group. Standardized training and support was provided to facilitators and responses to the SDCDM questions generated by participants (described subsequently) were also collected to enhance and ensure fidelity ofimplementation.

2.4 Participant attrition

One organization that was assigned to the treatment group withdrew from the study after training, but prior to data collection because of reported time constraints, leaving 11 organizations in the treatment group and 10 in the control group for a total of 21 provider organizations in the final sample. Prior to the implementation of the SDCDM and data collection, 7 participants exited the study, including 5 participants from a provider organization who withdrew because of time constraints. Thus, the sample available for analysis was 190, with 111 participants in the treatment group, and 79 in the control group.

Baseline data on self-determination was available for all 190 participants. However, there was significant attrition over time; at the second data collection point, only 137 of the original 190 (72% of the sample) provided self-determination data. By the third data collection point, only 54% of the original sample provided data. Thus, a total of 86 people stopped participating in the study at some point. The reported reasons for participant attrition varied. The largest number of participants (n = 29) exited data collection because they were no longer receiving services from the provider, and the second largest group did not contribute data because either the organization ended their participation or the facilitator they had been working with, within the organization, left the organization (n = 25). The remaining reasons for stopping participation were moving (n = 9) and unavailable/uninterested in continuing participation (n = 5), and withdrew consent (n = 2). For the purposes of analyses, if a participant responded to surveys at any time point, his or her data was retained for analysis. A non-parametric chi square was conducted on dropout by primary disability in order to determine if attrition was related to disability, and no difference was found on expected versus observed dropouts (χ² = 5.29, p = 0.52).

2.5 Intervention –   Self-Determined Career Development Model (SDCDM)

As described previously, organizations randomly assigned to the treatment condition trained facilitators to implement the Self-Determined Career Development Model (SDCDM) with participants with disabilities. The SDCDM consists of a three-phase instructional process, and each phase presents a problem to be solved by the person with the disability. The problem relates to some aspect of the job and career development process. In essence, as the person is answering the questions in each phase, she must: (a) identify the problem, (b) identify potential solutions to the problem, (c) identify barriers to solving the problem, and (d) identify the consequences of each solution. The problem the person with the disability must address in the first phase is “What is my goal?” The problem presented in the second phase is “What is my plan?” The third phase addresses the problem “What have I learned?”

Self-directed learning is the foundation of this model. Self-directed learning means that the person with the disability is supported to play a meaningful role in the following steps: 1) setting her own career and job related goals, 2) participating in decisions related to developing a plan of action to meet goals, 3) implementing the action plan, 4) evaluating her actions, and 5) modifying actions or goals to achieve the desired outcome. Even though self-directed learning is the foundation for the model, a facilitator is integral to model implementation. The facilitator is someone who will enable the person with the disability to succeed by: 1) providing support for working through the model to promote model understanding, 2) supporting a nonjudgmental atmosphere where efforts are valued, and 3) acting as an advocate for success. A facilitator may be a rehabilitation counselor, community provider agency staff member, teacher, parent, natural support person, or a peer mentor. In this study, community support providers that delivered employment services acted as facilitators. The level of support that the facilitator provides will vary but for all people with disabilities, the facilitator will use the model to support the person with the disability in learning the problem solving sequence, answering the questions presented in each phase, and moving from one phase to the next within a goal-oriented context.

The problem presented in each of the three phases is solved by the person with a disability by answering four questions. The questions are structured to employ a problem solving process. The person with the disability works with the facilitator to identify a solution that solves the problem or difficulty. Answering the four questions will help to reduce the discrepancy between where the person with a disability is now and where he or she wants to be. While the questions differ from phase to phase, they represent the same self-regulated problem-solving sequence. Associated with each question across the three phases are Facilitator Objectives, which define what the facilitator is trying to support the person to achieve in answering the questions. Each instructional phase also includes a list of Employment Supports that facilitators can use to enable meet the Facilitator Objectives, and enable people with disabilities to learn to self-direct learning.

2.6 Measures

2.7 Missing data

Missingness was first inspected at the item level for each time point, and individual questions showing little missingness. But when summed to create subscale and total scores for SDS and the GAS, missingness ranged from 7% to 25%. The data was examined next to determine the type of missingness due to attrition. The missing data tended to follow a monotone missingness pattern, that is, when participants exited the study they did not return. This pattern can be easily handled with multiple imputation (Van Buuren, 2012).

Once multiple imputation was selected as the means to handle missing data, a decision needed to be made with regard to how to impute given the nested structure of the data. The data collected had four levels of nesting, (organization, facilitator, person, and time), but with so few organizations (n = 21), the decision was made to impute in a wide format to take into account the levels of person and time. The facilitator was then specified as the level 2 identifier. There were too few participants in the study to impute at the item level, so following a method described by (Enders, 2010), subscale totals and subscale means were included in the imputation process. For the totals, subscale items were added together to create a score unless a question was missing. Subscale means were computed across all answered questions so that if the person answered any questions on a subscale, a mean score was available. The subscale means were then used as predictors in the multiple imputation with chained equations process. Theorized interactions were added to the imputation model, and the SDS subscales and GAS were grand mean centered before imputing 100 data sets in R (version 3.2.0; R Core Team, 2014) with the mice package (version 2.22; van Buuren & Groothuis-Oudshoorn, 2011).

2.8 Analysis

As described previously, the data collected was considered multilevel data, because individual participants with disabilities were not completely independent from each other as they were nested within facilitators who were nested within support provider organizations. This created an opportunity to examine our first research question regarding the amount of variability in outcomes explained by the organization and facilitator that implements the SDCDM. Thus, we examined first examined intraclass correlations (ICC) that resulted from these multilevel models. We examined ICCs as they provide an indication of how much variance in the model is explained by each nested level (i.e., participant, facilitator, organization) for the self-determination outcomes. This provides important information about the degree to which variability in outcomes was explained by personal-level factors, facilitator-level factors, and organization-level factors above and beyond examining the impact of the SDCDM on outcomes. Although (because of the restricted sample size) we were limited in our ability to examine specific factors at each level, we could determine the degree to which the various levels impacted variability in outcomes.

To examine change in total and subscale self-determination scores over time as a function of exposure to the SDCDM (Research Question 2), we then examined multi-level models specified for SDS total score and the four SDS subscale scores (autonomy, self-regulation, psychological empowerment, self-realization). To examine the impact of time, treatment (SDCDM or no-SDCDM), and the time by treatment interaction we added each of these terms to the model as predictors. Because of the small sample size and issues with detecting significance, the interactions of time×treatment were also plotted to examine trends in the pattern of results. All multilevel models were estimated in the lme4 package (Bates et al., 2014) in R (R Core Team, 2013).

4.1 Research question one

As described previously, and as is in the case in many community-based research projects, participants were recruited from multiple organizations and within each organization multiple facilitators were trained. Thus, participants and the outcomes they experienced were influenced by multiple levels including each participant’s own personal characteristics and experiences, as well as the characteristics of the facilitator they worked with and the organization that provided services. Participants were also highly diverse, as each support provider organization supported a diverse range of individuals. As is assumed by social-ecological models of disability (Schalock et al., 2010), multiple systems interact to exert an influence on the outcomes experienced by those with disabilities. And researchers have found that organizational factors impact readiness to implement and support implementation of evidence-based practices (Chilenski et al., 2015). However, rarely does research systematically examine the impact of the nested systems within which people live (Shogren, Luckasson, & Schalock, 2014). Thus, our first goal in the present analyses was to examine, empirically, the variability in outcomes predicted by the nesting of the data (e.g., participants within facilitators within organizations) by examining intraclass correlation coefficients. We found, as might be expected, participant-level factors explained the majority of the variance in self-determination outcomes. However, facilitator and organization-level factors also explained variability in outcomes, but in different ways for different outcome variables. Specifically when analyzing the degree of variability explained, the inclusion of both the facilitator and the organization-level in the models explained additional variability. Interesting, the influence of the facilitator and the organization-level varied based on the subdomain of self-determination analyzed.

For example, for autonomy, self-realization and overall self-determination facilitator level factors explained an additional 5–13% of the variability in outcomes, with almost no variance explained by the organization. Self-regulation and psychology empowerment, on the other hand, had different patterns. For self-regulation, 9 and 5% of the variance was explained by the facilitator and organization, respectively, while for psychological empowerment almost no variance was explained by the facilitator but almost 5% by the organization. These patterns suggest complex relationships between the person, facilitator, and organization in impacting self-determination outcomes. Such variability that would be missed if the nested nature of the data was not analyzed (Raudenbush & Byrk, 2002; Snijders & Bosker, 2011), or if total score and subscale analyses were not undertaken. This suggests the importance of considering these factors in future research, as well as more systematically examining them. A limitation of our study was the lack of data collected for analyses at the facilitator and organization level. Future research must collect robust data on facilitator and organization characteristics that can be explored to examine the variability in outcomes. Further, sufficient samples sizes must be obtained at each level of the analyses to ensure there is adequate power for the analyses. Beyond statistical considerations, a wide range of substantive questions could be explored. For example, in terms of autonomy and self-realization, where facilitator factors explained significant additional variance, but organizational factors did not, examining the specific facilitator factors (that are common across organizations) that lead to enhance autonomy and self-realization will be important. For example, does a supportive relationship between the facilitator and person with a disability lead to the development of greater self-awareness (i.e., self-realization) and more initiation of identifying goals and interests (i.e., autonomy)? Relatedly, for self-regulation, given that both facilitator and organization explained significant variabilities, research is needed to examine the organizational factors (i.e., organizational culture, organizational policies) that influence facilitator behavior and supports (i.e., fidelity of implementation, flexibility in providing supports and using resources to go after goals) and therefore influence self-regulation outcomes. Such research has the potential to inform intervention development that enables the consideration of multi-system level factors in the design and delivery of materials and supports.

4.2 Research question two

The results suggest potential impacts of the SDCDM on self-determination outcomes, particularly on the essential characteristic of autonomy. A few caveats must be noted in discussing these findings. First, we were restricted by a small sample size for the analyses, particularly given the nested nature of the data. And, given the significant variability across participants, facilitators, and organizations found in Research Question 1, and our lack of ability (given the sample size) to systematically examine factors at each level that influenced outcomes, it is possible that there was individual level change for participants working with supporting facilitators in supportive organizations, but that differing patterns were present across all participants, limiting the ability to detect overall changes. This issue was further exacerbated by the heterogeneous nature of the sample. Further, when intervening in adulthood, it may be that changes in self-determination as slower to emerge, given the learning history that people have already experienced. Research has found that previous opportunities to make choices, express preferences and engage in self-determined behavior can create learned helplessness and restrict opportunities for self-determination (Neely-Barnes, Marcenko, & Weber, 2008; Stancliffe, 2001) Limited work has explored the developmental trajectory of self-determination past late adolescence, and work is needed to explore the ongoing development of self-determination in adulthood while examining the influence of contextual factors, including supports and experiences with family members, direct support professionals, and education and adult service systems. Researchers have found that in can take up to two years of intervening to promote self-determination (Wehmeyer et al., 2012) to detect changes in self-determination in adolescents with disabilities, and it may be that the developmental trajectory is even further extended in adulthood. This suggests the need for longer-term intervention and data collection; as well as data collection that systematically examines and accounts for the contextual factors (i.e., personal and environmental characteristics) that influence outcomes. The finding that all participants, irrespective of whether they were in the SDCDM treatment group or control group shown slight (albeit non-significant) increases in their overall self-determination scores, suggests the importance of this research.

For those in the treatment group, the only essential characteristic of self-determination that showed greater change, compared to the control group, was autonomy. This may reflect that autonomy is one of the first characteristics of self-determined behavior to change when intervening to promote self-determination, as the first stage of the SDCDM emphasizes identifying preferences and interests and self-selecting goals, critical elements of autonomous functioning (Benitez, Lattimore, & Wehmeyer, 2005; Wehmeyer et al., 2003). If these are new skills for participants – perhaps because they have not previously had these opportunities, it is possible that these changes are the first to emerge, and with longer-term intervention and data collection as participants begin to self-regulate action toward these goals (Phases 2 and 3 of the SDCDM) and feel empowered in their ability to do so, and come to understand their strengths and support needs related to doing so (i.e., self-realization), that changes in these domains would be observed. Despite the small nature of the observed impacts of the SDCDM on autonomy scores, the pattern of findings particularly that changes in autonomy may be observed first, provides important information to guide future research, particularly related to the nature of change in self-determination scores and the intersection of person-level changes with changes in support provider and organizational behavior and policies as it can be assumed that changes in support provider and organizational behavior and policies may impact opportunities for self-determination and resulting self-determination outcomes.

4.3 Limitations and future research directions

As mentioned previously, the sample size was a significant limitation of the study, particularly given the nesting of the data and the differential impacts of the nested data structure on the dependent variables. Although the number of facilitators was technically sufficient to obtain estimates in a three-level model, the number of provider organizations was small, and further, the participants that were nested within organizations and facilitators were highly diverse. While reflective the people served by the community support providers, this also introduced additional variability into the sample that may have influenced the results. It also meant there was not sufficient sample size to examine predictors within the nested structure that likely influenced outcomes (i.e., participant disability label; facilitator experience; organization support for career development), nor did we collect more nuanced data on organization culture and facilitator skills that would have been germane to teasing out the reasons for the variance being explained at different levels of the model (Snijders & Bosker, 2011). We also experience significant attrition in the sample over time. Although this is not unexpected in community-based research, it has implications for future research planning for sufficient sample sizes to address specific research questions related to effects across levels of the model (i.e., participant, facilitator, organization). Future research must consider these issues in generating samples for community-based research designs.

Another issue that emerged in implementing the SDCDM that influenced sample size was the resources available to support facilitators within provider organizations. As mentioned previously, the SDCDM is a modification of the SDLMI, an evidence-based intervention to promote self-regulated problem solving in service of education goals. The SDLMI has been implemented by teachers, and to create the SDCDM the materials developed for the SDLMI were modified for use by direct employment support providers within community-based organizations. However, challenges arose during the implementation and in collecting follow up and fidelity data. First, employment support providers are more likely than teachers to leave their position. As we noted above 12 (31.5%) of the trained facilitators left or changed jobs and dropped out of the study. This contributed to the loss of 25 study participants. Another challenge associated with this study was logistical in that facilitators worked with multiple people in multiple settings, which made it difficult to implement the SDCDM and collect data on a consistent timeline and collect fidelity of implementation data using best practices. A final challenge arose out of the requirements of from the University Institutional Review Board, which required that each facilitator complete online research ethics training as they were collecting data. This delayed data collection and implementation in most cases, and created barriers to the active involvement of direct support professionals. Anecdotally, participating organizations that left the study felt that the combination of the online ethics training, training in the SDCDM and required implementation activities took their employment support providers away from their jobs for up to four hours without contributing to their work skills. Further work, is needed to address these factors, including consideration of the best ways to collect data and ensure meaningful roles for support providers. One recommendation that arose out of these challenges is separating the facilitation role from the data collection role. This would give the facilitators more time to directly provide employment supports using the SDCDM. Thus, the findings must be considered preliminary and used to inform future work that focuses on recruiting larger samples, planning of the nested data structure in generating the sample, and implementing intervention and collecting data over longer periods of time.

Further research is also needed to examine drop-out, the factors that influence it, and its potential impact on outcomes. Additionally, different participants had different lengths of exposure to the SDCDM intervention because of varying goals and implementation/data collection procedures within organizations and facilitators, which likely also influenced the results, particularly given the issues noted above with regard to the speed of change in self-determination outcomes, particularly in adults. Further research is needed over longer periods of time and with more sensitive process measures, such as goal attainment, which has been used in other studies to document shorter-term intervention impacts (Shogren et al., 2012). Finally, we were unable to follow-up to explore changes in career development and actual employment outcomes. Participants set goals related to exploring careers, improving job skills, and identifying and obtaining preferred jobs as part of the SDCDM, and data specific to these outcome variables should be collected in future studies. While it is important to document that interventions designed to promote self-determination actually lead to changes in self-determination scores, a critical next step will be to look at longer-term outcomes related to employment. Generally, these preliminary findings provide information to guide future research development and implementation, and suggest a need for greater attention to multi-level modeling of intervention outcomes, and planning for a sample and data collection procedures that enable analyses to tease out the multiple factors that make a differences and track longer-term outcomes. Only then can the key factors be identified and targeted in interventions and supports to promote career development and employment. Future research also needs to consider, in addition to organization-level factors, community-level factors including economic data and workforce participation as these likely impact opportunities to see changes in career development and employment outcomes, irrespective of individual-level interventions implemented.

4.4 Implications for practice

Despite the limitations, the findings provide several implications for practice. First, facilitators were able to implement the SDCDM with participants, and this had an impact on autonomy for those in the SDCDM treatment group. The slow nature of the growth over time suggests that considering long-term supports related to self-determination in the context of career development may be important, particularly for people who have a long-history of other-directedness in the support and opportunities they have had for career development and employment. Additionally, organization and facilitator factors impact outcomes. While this seems logical, it highlights the importance of organization culture and structure on both staff and people with disabilities receiving supports. Attending to environmental factors that influence consumers and staff within organizations will be critical to enable positive outcomes for all (Keith & Bonham, 2005; Schalock et al., 2014).