The Outcomes of Vocational Interventions

Abstract

The purposes of this article are to (a) summarize the status of career intervention research since Spokane and Oliver’s seminal meta-analysis of vocational intervention outcome and (b) discuss in more detail a recently proposed method for determining whether career interventions make a difference in the lives of clients (clinical significance). Although several excellent meta-analyses of the career intervention outcome literature have appeared since 1983, we briefly summarize the meta-analytic results, discuss future research directions that were suggested by the meta-analyses, and evaluate our progress in addressing suggestions. The bulk of the article focuses on summarizing how clinical significance of career outcomes can be addressed, presenting normative data that can be used to address clinical significance questions with several widely used outcome measures, and illustrating clinical significance calculations using published career intervention studies.

Keywords

intervention research clinical significance meta-analysis

The first major meta-analysis of career intervention outcome appeared over 30 years ago (Spokane & Oliver, 1983). We thank Bruce Walsh, Editor of the Journal of Career Assessment, for asking us to write a 30th anniversary article on this seminal study. Although we missed the 30th anniversary by several years, it is still worth revisiting this study to see how far research on career intervention efficacy has advanced over the ensuing years. Thus, the first purpose of this article is to use the Spokane and Oliver (1983) article as a springboard to summarize the current status of career intervention outcome research.

Spokane’s and Oliver (1983) meta-analysis of the career outcome literature seemed to inspire a number of subsequent meta-analyses. The purposes of these newer meta-analyses were to update Oliver and Spokane’s results (Oliver & Spokane, 1988; Whiston, Sexton, & Lasoff, 1998), to improve on the meta-analytic methodology employed by Oliver and Spokane (Brown & Ryan Krane, 2000; Whiston, Brecheisen, & Stephens, 2003; Whiston et al., 1998), or to take a more focused approach by addressing common sets of client concerns (e.g., choice difficulties; Brown & Ryan Krane, 2000). Collectively, these meta-analyses have suggested that career interventions have demonstrable effects on a variety of client outcomes. Oliver and Spokane’s initial results suggested that career interventions (when considered collectively) had a rather substantial effect on client outcomes (when also considered collectively), yielding an overall d of 1.11. Thus, this early meta-analysis suggested that the average client receiving some form of career intervention improved over a full standard deviation more than the average person receiving no counseling. However, more recent meta-analyses that have used more up-to-date meta-analytic procedures have yielded substantially lower effect size estimates, with the most consistent estimate to be a d of approximately .34 for interventions targeted at career choice difficulties (Brown & Ryan Krane, 2000; Whiston et al., 1998, 2003). Thus, there is consistent meta-analytic evidence to suggest that career interventions (when compared to no- or delayed-treatment controls) are demonstrably, but modestly, effective, especially at promoting outcomes associated with career choice-making (e.g., decidedness, vocational identity, career maturity, and career decision-making self-efficacy beliefs)—the average client receiving career choice interventions improves about a third of a standard deviations more than those receiving no intervention (see also Brown & McPartland, 2005; Miller & Brown, 2005; Whiston & Rahardja, 2008).

Several suggestions have also been offered to improve the apparent modest effectiveness of career interventions, including tailoring interventions to the reasons that clients seek counseling (e.g., not all clients want more options) and developing career problem diagnostic systems (e.g., Brown & McPartland, 2005; Brown & Ryan Krane, 2000; Miller & Brown, 2005; Rounds & Tinsley, 1984; Spokane & Oliver, 1983), engaging in Aptitude × Treatment interaction research (e.g., Fretz, 1981; Spokane & Oliver, 1983), ensuring that interventions include critical ingredients identified by Brown and Ryan Krane (2000), and providing appropriate “doses” of treatment (e.g., Brown & Ryan Krane, 2000; Whiston & James, 2013). More importantly, many scholars have suggested that our outcome research focuses too much on those who might need our services the least—largely college students and affluent members of society (e.g., Blustein, 2006; Brown, 2014; Whiston & James, 2013).

Recent literature (see Brown, 2014) suggests that researchers are paying increasing attention to dose issues (sometimes too rigidly) and to including critical ingredients into the design of their interventions (e.g., Björnsdóttir, Einarsdóttir, & Vilhalmsdóttir, 2011; Chronister & McWhirter, 2006; Hirschi & Lage, 2008). Also several diagnostic systems have been developed (Brown et al., 2012; Gati, Krausz, & Osipow, 1996; Hacker, Carr, Abrams, & Brown, 2013; Saka & Gati, & Kelly, 2008), with Gati and colleagues system used extensively with some success to direct intervention efforts (e.g., Gati & Amir, 2010; Gati & Levin, 2014). Peterson and his colleagues at the Career Center at the Florida State University have been tailoring treatments to client problems for some time via their cognitive information processing theory (Reardon, Lenz, Sampson, & Peterson, 2000), although outcome data on their treatment-matching approach are quite limited.

There has also been increasing attention to studying the effects of career interventions with more diverse clients using both experimental and quasi-experimental designs, including domestic abuse survivors (Chronister & McWhirter, 2006; Davidson, Nitzel, Duke, Baker, & Bovaird, 2012), incarcerated offenders (Fitzgerald, Chronister, Forrest, & Brown, 2013), Asian-American clients (e.g., Kim & Atkinson, 2002), first-generation Chinese immigrants (Shea, Ma, Yeh, Lee, & Pituc, 2009), business students in Taiwan (Peng, 2001; Peng & Herr, 1999), inner-city adolescents (Turner & Conkel, 2010), and highly indecisive Italian adolescents (Nota & Soresi, 2004). Other studies have also showed some promising outcomes for diverse clients in the United States and internationally (e.g., Björnsdóttir et al., 2011; Di Fabio & Kenny, 2011; Grier-Reed, Skaar, & Conkel-Ziebel, 2009; Hirshi & Lage, 2008; Masdonati, Massoudi, & Rosier, 2009; McWhirter, Rasheed, & Crothers, 2000; O’Brien, Dukstein, Jackson, Tomlinson, & Kamatuka, 1999; Plimmer, 2012; Thrift, Ulloa-Heath, Reardon, & Peterson, 2012).

Thus, it appears that at least some suggestions to improve career intervention effectiveness derived from meta-analytic data are being heeded by those who develop career interventions. Although there is much work still to be done, it appears that career interventions have continued to show as at least moderate effectiveness in promoting career behaviors in increasingly diverse client samples. However, despite promising effect size estimates, we still have little evidence that the gains clients demonstrate on common outcome measures make a difference in their lives.

Clinical Significance of Career Interventions

In the innovative spirit of Spokane and Oliver (1983), our second purpose is to summarize a new way to analyze outcome data; namely, how to analyze for the clinical significance of outcomes associated with career interventions. Clinical significance, initially introduced into the psychotherapy outcome literature by Neil Jacobson and colleagues (e.g., Jacobson, Follette, & Revensdorf, 1986: Jacobson & Truax, 1991), addresses the question of whether interventions make a difference in the lives of clients. According to Jacobson and colleagues, standard data analytic strategies (null hypothesis significance testing) and even effect size estimates fail to shed sufficient light on clinical significance because they really do not address the degree to which therapy improved the lives of clients. Jacobson and Truax (1991) provided an excellent example to illustrate the limitations of current data analytic strategies—that an average weight loss associated with a treatment for obesity of 2.00 pounds versus a 0.00 pound weight loss in a control condition would likely be statistically significant and could also yield a large effect size estimate. For example, if the pooled standard deviation was 1.00, then the resulting effect size estimate would suggest that the average treated client lost +2.00 standard deviations more than the average control client. But is a two-pound weight loss likely to make a difference in the lives of significantly over weight people and doesn’t a d of 2.00 overstate the magnitude of the treatment effect (see also Brown, 2014)?

Jacobson, Follette, and Revensdorf (1986) proposed, as a definition of clinical significance, that clinically significant change should represent a change to normal functioning among client samples. In order to operationalize this conceptual definition, they proposed that clients and nonclients (i.e., normally functioning individuals) could be conceived of as residing in two overlapping population distributions—one containing clients and the other containing nonclients. Clinical significance can then be determined by assessing the degree to which clients move from the client to the functional distribution after treatment or the degree to which clients become more like nonclients (and less like clients) on the outcome measures of interest.

Jacobson et al. (1986) proposed several possible ways to assess for clinical significance: (a) does the client move out of the client distribution; (b) does the client move into the functional distributions; and (c) is the client closer to the mean of the functional distribution than the mean of the dysfunctional population after treatment? The strongest evidence for clinical significance is yielded by the third (c) option and requires norms for both functional and nonfunctional populations. If both are available, a cutoff score can be established for each client to estimate the point of entry into the functional population (and out of the dysfunctional population):

c = S D_{0} M_{1} + S D_{1} M_{0} / S D_{0} + S D_{1},

where SD ₀ and SD ₁ are the standard deviations for the functional and nonfunctional populations, respectively. M ₀ and M ₁ are the means of the functional and nonfunctional populations.

Jacobson and colleagues (Jacobson et al., 1986; Jacobson & Truax, 1991) further suggested that this approach could be used to track individual client change if the cutoff score could be complemented by a way to estimate whether each client’s change was statistically reliable. They proposed the following index to address the question of individual change:

R C I = x_{2} - x_{1} / s_{d i f f},

where RCI is called the reliable change index and x ₁ and x ₂ are the client’s pre- and posttreatment scores on the outcome measure. S _diff is the standard error of difference which represents the amount of change that would be expected based on measurement error. RCI values greater or equal to 1.96 suggest that the amount of change exhibited by the client is greater (p < .05) than would be expected on the basis of measurement error.

RCI and c can then be used together to track client improvement and classify each client into one of the following typical categories: (a) recovered (statistically reliable change and movement out of the client into the nonclient distribution), (b) improved (statistically reliable change only), (c) unimproved (no statistically reliable change), and (d) deteriorated (statistically reliable change in the wrong direction). These data can then be used by clinicians to evaluate their practices and by researchers to compare the effectiveness of alternative interventions or to address other research questions. For example, Choi, Buskey, and Johnson (2010) employed the joint RCI and c criteria to track counseling center clients’ progress and explore the relation of progress to academic adjustment and commitment (i.e., do counseling centers have a positive impact on academics and retention?). Results revealed statistically significant differences between clients who were classified as recovered and those who showed improvement only or no changes on the measures of academic adjustment and commitment (the latter two groups showed no significant differences). Choi et al. (2010) interpreted these results as suggesting that counseling centers can be important academic resources in college and university communities if they are provided with the freedom to ensure clients are recovered before terminating treatment.

Brown (2014) illustrated how Jacobson and colleagues’ methods could be employed to test for the clinical significance of career intervention outcomes. For example, Brown (2014) showed how c values could be obtained from norms of college career center clients and nonclients (students not seeking career help) to evaluate clinical significance of client change on the Career Thoughts Inventory (CTI) and its subscales (Sampson, Peterson, Lenz, Reardon, & Sanders, 1996)—do the negative career thoughts (for example) of career clients move closer to the mean of nonclient populations than the mean of client populations after counseling? Brown (2014) also presented an alternative method (Z _cs) for assessing the clinical significance of gains by career clients that may fit better with the positive development philosophy associated with counseling and vocational psychology. The method also quantifies client gains rather than resulting only in dichotomous decisions. Brown (2014) also showed how the alternative index could serve as a stand-alone measure of clinical significance and how it could be used along with Jacobson and colleagues’ method to assess the clinical significance of gains obtained by clients in career interventions.

In the remainder of this article, we summarize Brown’s (2014) method, provide additional normative data for several instruments that are used widely in career intervention outcome research, and illustrate the use of the method with published career intervention outcome studies (see Brown, 2014 for additional illustrations). Brown (2014) proposed the Z_cs for four primary reasons. First, he argued that Jacobson and colleagues method may not be consistent with counseling and vocational psychology’s philosophy of promoting positive development versus simply treating problems. Brown (2014), therefore, sought a method that did not exclusively require norms for clinical dysfunctional populations (but could be employed with such norms). Second, he wanted a method that could be flexibly employed as a stand-alone method for assessing clinical significance of career interventions or as an adjunct to Jacobson and colleagues c. Third, he sought a method that would quantify client movement rather than simply result in a dichotomous decision about whether or not a client has moved from one population distribution to another. Fourth, he wanted a method that could be used to track individual client progress (along with RCI), but also could be useful for estimating the clinical significance of mean gains from group designs.

Z _cs was proposed as a “quantitative estimate of client’s resemblance to the typical or average person in a comparison or norm group chosen to demonstrate the practical or clinical impact of an intervention on the lives of clients” (Brown, 2014, p. 66). It is calculated by the following formula:

Z_{c s} = X - M_{N} / S D_{N},

where M_N and SD_N are the means and standard deviations of the norm or comparison group and X is the pre- and posttreatment score for an individual client (x_i ) or the pre- and posttreatment means of treatment and comparison groups (M_t and M_c ). The former (x_i ) is used (along with RCI) for tracking individual client change, while M_t and M_c are employed for tracking the average movement of experimental and control groups. As a standard score, Z _cs has a mean of 0.00 and a standard deviation of 1.00 and is interpreted as how far the individual’s score (or group mean) on the outcome measure of interest falls from the norm group mean in standard deviations. Lower values of Z _cs are associated with increasing clinical significance until the norm group mean is reached (i.e., Z _cs = 0.00)—indicating that the client’s posttreatment score (in the individual case) fell exactly at the norm group mean. Further clinically significant gains are suggested when the client’s score (or group mean) begins to surpass the norm group mean after reaching 0.00.

Brown (2014) illustrated the procedure via a fictitious example of a career development program designed to promote increased school engagement of students who are at high risk for dropping out of high school. Students are administered a measure of school engagement at pre- and posttreatment (and maybe also at follow-up intervals) and the manual of the school engagement measure is used to identify appropriate norm groups for addressing questions of clinical significance. The manual contains data collected from a large group of highly engaged students who are on track to graduate which provide comparison group data (i.e., M_N and SD_N ) for calculating Z _cs at pre- and posttreatment for each student (or the group as a whole). The question then is to what degree do students in the career intervention (individually or on the average) come to resemble the average engaged, on-track students at posttreatment. A Z _cs of −1.00 at pretreatment and 0.00 at posttreatment suggests clinically significant improvement—that is the student’s feelings of school engagement moved from 1 standard deviation below the norm group mean at pretreatment to the norm group mean at posttreatment. In other words, the student’s feelings of school engagement (as assessed by the school engagement measure) at posttreatment came to resemble the school engagement of the average engaged student who was on-track to graduate. Even better, if the manual included scores of high school dropouts, c scores could also be calculated and used in conjunction with the Z _cs. Z _cs would be used to quantify student movement, while c (along with RCI) would be used for classification purposes. Although the ultimate proof of whether the intervention made a difference in the lives of these students would be whether or not they actually graduated from high school, the clinical significance methodology does provide a proximal, intermediate way to assess clinical significance of career intervention outcomes.

Selecting Norm Groups

The major difficulty involved in implementing the clinical significance methodology is the availability of appropriate norm groups, although Brown (2014) suggested several possibilities, including test manuals, literature searches, and meta-analyses. The fictitious example of the school dropout intervention suggested that appropriate normative data on engaged, on-track students might be available in published test manuals, but this might not always be the case. As an alternative, a literature search on school engagement might identify studies that compared the engagement scores of high school graduates and dropouts. The means and standard deviations reported by these two groups could be used to track clinical significance via c, RCI, and Z _cs. If only high school graduate norms were available, then Z _cs could be used as a stand-alone measure of clinical significance. Should multiple studies be available, the data provided by these studies could be meta-analytically combined to yield less biased normative data (M_N and SD_N ) than would be provided by a single study.

Another issue in selecting appropriate norm groups is ensuring that they represent appropriate comparisons for estimating clinical significance, while being aware of limitations of groups that represent less than adequate matches. For example, although using high school graduates as the norm group in the high-risk student example may be appropriate if norms on engaged students are not available, some students graduate without being highly engaged in school (i.e., there will be more overlap with the dropout norms than if the norm group was composed of only engaged students). Similarly general high school norms, if used for clinical significance analyses, will contain some students who do not graduate creating increased overlap with the at-risk student norms.

As another example, career interventions for college students are often designed to help students decide on a major and/or career. Assessment of clinical significance would address the question of how closely clients resemble students who had declared a major or decided on a career at pre- and posttreatment on the outcome measure of interest. In these cases, the most appropriate norm groups would be composed of students who had declared majors or decided on a career direction. Although general college student norms could also be used in these cases, they would be less than ideal because of distributional overlap (i.e., general college norms will include some students who have not declared a major or decided on a career direction).

Normative Data on Measures Used in Career Intervention Research

Brown (2014) provided some preliminary normative data for clinical significance analysis with the CTI (Sampson et al., 1996) and the Career Decision Difficulty Questionnaire (CDDQ; Gati et al., 1996). The former norms (CTI) used the means and standard deviations obtained from two client and two nonclient samples reported in the CTI manual and c values were calculated for each of CTI total and subscale scores. The latter norms (CDDQ) were derived from two samples of U.S. college students (Gati & Amir, 2010; Gati et al., 1996), but since no client norms were available for this instrument, no c values could be calculated. Also as noted previously, general college student norms, although useful, may be less than ideal for purposes of assessing for the clinical significance of gains on CDDQ scores associated with career interventions designed to help students choose a major or an occupation.

In this section, we provide updated norms on the CTI and CDDQ obtained from a meta-analysis of the published literature. In addition, we provide new norms, also obtained meta-analytically, for the short form the Career Decision-Making Self-Efficacy Scale (CDSE; Betz, Klein, & Taylor, 1996), another widely used measure in career intervention outcome research. We also provide normative data on the Satisfaction with Life Scale (SWLS; Diener, Emmons, Larson, & Griffin, 1985) because we believe this is an important but understudied outcome of career interventions. We hope that our normative data on this scale will stimulate research on the impact of career interventions on clients’ life satisfaction rather than just on their career decision-making difficulties, career thinking, and career decision self-efficacy beliefs.

Another type of outcome that should be addressed more in career intervention research is clients’ levels of psychological distress. For example, Rochlen, Milburn, and Hill (2004) found that the average score obtained by a cluster of career counseling center clients who displayed a high degree of career uncertainty (41% of their sample) was almost a half of a standard deviation higher than the average score of outpatient mental health clients on the emotional distress scale of the Outcome Questionnaire 10.2 (OQ10.2; Lambert, Finch, Okiishi, Burlingame, Mckelvey, & Reisenger, 1998). We provide no normative data on the OQ 10.2 because the OQ 10.2 manual provides excellent norms along with c values that can be used to assess clinical significance with the Jacobson and colleagues procedure. We would urge that the OQ 10.2 or other measures of psychological distress and symptoms be routinely included in career intervention research. Demonstrating that career interventions are associated with clinically significant change in psychological functioning would be strong evidence that our interventions may actually make a difference in the lives of our clients (see also Multon, Heppner, Gysbers, Zook, & Ellis-Kalton, 2001; Whitson, 2001).

In order to collect normative data on the CTI, CDDQ, CDSE, and SWLS, we engaged in comprehensive literature searches on each measure using computerized databases, reference lists from identified articles, and hand searches of tables of contents of relevant journals. To be included, a study had to provide means and standard deviations on the measure and specify the scaling used for each item (e.g., 1–5 scale). Data on sample characteristics were also coded. References that were used in developing clinical significance norms are marked with an asterisk in the Reference section of this article.

Before calculating meta-analytic means and standard deviations, we plotted sample means and standard deviations by date of publication and found no clear linear or curvilinear trends nor evidence of significant spikes (e.g., around 2008–2010 during the economic crisis). Thus, despite rapid changes in employment outlooks and the economy over the last 30 years, these changes do not seem to be related in a clearly discernable way to how people respond to the CTI, CDDQ, CSES, or SWLS. Thus, we used all studies that provided relevant data and calculated sample size weighted means and pooled standard deviations.

Table 1 provides updated normative data on the CTI total and subscale scores published from 1996 to 2014. Separate norms are provided for college students, career course participants, and career center clients. Participants were mostly female (60%) and Caucasian (White = 66%, Black = 15%, Asian American = 3%, Latino/Latina = 5%, Native American = 1%, and other or not reported = 10%). The weighted means and pooled standard deviations for the latter two groups (course participants and clients) were calculated from preintervention data. C values are also provided for college students versus career course participants and college students and career center clients. The former c values can be used along with the RCI to track whether individual course participants demonstrate clinical significant change—enter the distribution of college students on the CTI. The latter c values can be used (along with RCI) to track whether career center clients enter the student population after counseling. Weighted means and pooled standard deviations are also reported for three additional groups—noncollege adults (66% women, 85% White, mean age = 39.0), college students with learning disabilities, and female felons.

Table 1.

Clinical Significance Norms for the Career Thoughts Inventory (CTI).

Sample/scale	N	K	M_w	SD_p	c
U.S. college students
CTI-total	980	4	45.21	21.95
CTI-DMC	980	4	10.09	7.46
CTI-CA	980	4	12.29	5.68
CTI-EC	980	4	3.57	2.43
U.S. career course pretest
CTI-total	680	4	48.15	21.31	46.70^a
CTI-DMC	954	5	11.55	7.63	10.81^a
CTI-CA	954	5	14.42	5.61	13.36^a
CTI-EC	954	5	4.10	2.77	3.82^a
U.S. career center clients
CTI-total	575	2	58.68	17.54	52.70^b
CTI-DMC	656	3	15.21	6.92	12.75^b
CTI-CA	656	3	16.51	4.43	14.66^b
CTI-EC	656	3	4.86	2.74	4.17^b
U.S. high school
CTI-total	396	1	48.78	20.50
CTI-DMC	396	1	11.89	7.03
CTI-CA	396	1	12.19	4.89
CTI-EC	396	1	3.84	2.15
U.S. adult (noncollege)
CTI-total	571	1	36.33	22.09
CTI-DMC	571	1	8.20	7.00
CTI-CA	571	1	9.34	5.51
CTI-EC	571	1	2.95	2.12
Cognitive/LD college students
CTI-total	153	1	41.01	23.85
CTI-DMC	153	1	9.49	6.97
CTI-CA	153	1	11.17	5.39
CTI-EC	153	1	3.32	2.15
Female felons
CTI-total	---	---	---	---
CTI-DMC	92	1	12.30	6.90
CTI-CA	92	1	8.50	3.60
CTI-EC	92	1	6.80	3.80

Note. N = total sample size; K = number of samples; M_w = sample size weighted mean; SD_p = pooled standard deviation; CTI-total = CTI total score; CTI-DMC = Decision-Making Confusion Scale; CTI-CA = Choice Anxiety Scale; CTI-EC = CTI External Conflict Scale.

^a c Value for college student versus career course participant analyses.

^b c Value for college student versus career center client analyses.

Table 2 provides updated normative data (1996–2014) on the CDDQ total score and scores on its three primary subscales (Lack of Readiness, Lack of Information, and Inconsistent Information). Although we have also calculated weighted means and pooled standard deviations for the 10 secondary subscales, they are not included here because of space considerations. They are available from the first author upon request. Separate norms are provided for a variety of different groups—U.S., Israeli, Taiwanese, and Chinese college students; Swiss young adults; and Greek (male and female), Korean, and French high school students. Normative data are also summarized for Israeli career service seekers versus nonclients and Taiwanese decided versus undecided college students. C values are also presented for clinical significance analyses of Israeli service seekers versus Israeli nonclients and Taiwanese undecided versus decided college students.

Table 2.

Clinical Significance Norms for the Career Decision-Making Difficulties Questionnaire (CDDQ).

Sample/scale	N	K	M_w	SD_p	c
U.S. college student
CDDQ-total	2,157	6	3.56	1.24
CDDQ-LR	2,590	7	4.05	1.13
CDDQ- LI	2,590	7	3.46	1.79
CDDQ-II	2,590	7	3.01	1.44
Israeli college student
CDDQ-total	297	1	3.31	1.21
CDDQ-LR	297	1	3.50	1.04
CDDQ-LI	297	1	3.62	1.83
CDDQ-II	297	1	4.26	1.63
Israeli nonclients
CDDQ-total	259	1	3.49	1.36
CDDQ-LR	259	1	3.71	1.23
CDDQ-LI	259	1	3.84	1.78
CDDQ-II	259	1	3.04	1.36
Israeli career service seekers
CDDQ-total	95	1	4.06	1.12	3.80^a
CDDQ-LR	95	1	3.91	1.18	3.81^a
CDDQ-LI	95	1	4.61	1.46	4.27^a
CDDQ-II	95	1	3.59	1.40	3.31^a
Taiwanese college student
CDDQ-total	1026	1	4.79	1.15
CDQ-LR	1026	1	5.09	1.08
CDDQ-LI	1026	1	4.92	1.71
CDDQ-II	1026	1	4.48	1.18
Taiwanese decided college student
CDDQ-total	317	1	4.20	1.17
CDDQ-LR	317	1	4.29	1.16
CDDQ-LI	317	1	4.92	1.71
CDDQ-II	317	1	4.12	1.31
Taiwanese undecided college student
CDDQ-total	197	1	5.36	1.00	4.83^b
CDDQ-LR	197	1	5.27	1.04	4.80^b
CDDQ-LI	197	1	5.64	1.56	4.95^b
CDDQ-II	197	1	5.21	1.06	4.73^b
Chinese college students
CDDQ-total	911	1	---	---	---
CDDQ-LR	911	1	4.78	1.12
CDDQ-LI	911	1	4.63	1.55
CDDQ-total	166	1	3.58	1.44
Greek high school students-male
CDDQ-total	123	1	4.11	1.35
Greek high school students-female
CDDQ-total	166	1	3.58	1.44
Swiss young adults
CDDQ-total	168	2	3.46	.60
French high school
CDDQ-total	575	1	140.21	37.59
Korean high school
CDDQ-total	613	1	129.62	41.75

Note. N = total sample size; K = number of samples; M_w = sample size weighted mean; SD_p = pooled standard deviation; CDDQ-total = CDDQ total score; CDDQ-LR = CDDQ Lack of Readiness Scale; CDDQ-LI = CDDQ Lack of Information Scale; CDDQ-II = CDDQ Inconsistent Information Scale.

^a c Values for Israeli nonclient versus help seekers analyses.

^b c Values for Taiwanese decided and undecided college student analyses.

CDSE-short form normative data are summarized in Table 3 (2005–2014). Different investigators have used different scaling metrics for this scale (e.g., 1–10, 0–9, and 1–5 scales) although the 1–5 scale seems to be the most consistently employed in recent years. Thus, the normative data in Table 3 are for the CDSE-short form with the 1–5 response format. Data are provided separately by race/ethnicity and gender for U.S. college students as well as for participants in career courses and declared versus undeclared majors (with c values). Weighted means and pooled standard deviations are also summarized for several different samples of international high school and college students and for Latino/Latina high school students.

Table 3.

Clinical Significance Norms for the Career Decision-Making Self-Efficacy Scale—Short Form (CDSE).

Sample	N	K	M_w	SD_p	c
U.S. college students	2,995	11	3.85	.58
Career course—preintervention	420	1	3.44	.62	3.65^a
Declared major	262	1	3.55	.62
Undeclared/uncertain	112	2	3.19	.57	3.35^b
White college students	1,469	2	3.80	.60
African-American college students	469	3	3.97	.66
Asian-American college students	71	1	3.80	.55
Latino/Latina college students	238	2	3.90	.73
Male college students	49	1	3.76	.51
Female college students	292	2	3.92	.57
French college students	650	1	3.40	.50
Chinese college students	796	1	3.55	.52
South Korean college students	217	1	3.33	.58
Taiwanese college students	336	1	3.49	.47
Turkish college students	695	1	3.51	.57
Philippine college students	141	1	3.73	.49
Latino/Latina high school students	203	2	3.88	.56
Chinese high school students	384	1	3.45	.55
Hong Kong high school students	532	1	3.75	.53
U.S. high school students	216	1	3.68	.64
Greek male high school students	92	1	3.39	.56
Greek female high school students	108	1	3.54	.60

Note. N = total sample size; K = number of samples; M_w = sample size weighted mean; SD_p = pooled standard deviation; declared major = U.S. college students who had declared a major; undeclared/uncertain = U.S. college students who had not declared or were uncertain about a major.

^a c Value for college student versus career courses analyses.

^b c Value for college declared versus undeclared/uncertain analyses.

Finally, Table 4 provides weighted means and pooled standard deviations on the SWLS (1985–2014) for U.S. college students as well as Mexican American college students (male, female, and total); U.S. high school students; U.S., Swedish, and Swiss young adults; patients with traumatic brain injury; and clients seeking mental health treatment. Most of the mixed U.S. samples were women (76% for college students and 59% for adults) and White (57% for college students and 73% for adults).

Table 4.

Clinical Significance Norms for the Satisfaction with Life Scale (SWLS).

Sample	N	K	M_w	SD_p
U.S. college students	2,420	9	23.88	6.90
Mexican American college students	446	1	25.65	6.15
Mexican American college students (male)	178	1	25.15	5.70
Mexican American college students (female)	438	2	25.85	6.28
U.S. adults	1,779	9	24.02	7.22
U.S. high school students	454	1	24.30	8.01
Swedish young adults	2,966	1	19.77	4.33
Swiss young adults^a	542	2	4.80	.55
Traumatic brain injured	253	1	17.55	7.89
Outpatient clients	43	2	15.85	7.20

Note. N = total sample size; K = number of samples; M_w = sample size weighted mean; SD_p = pooled standard deviation.

^a M_w and SD_p reported for these two samples were based on mean total scores.

Illustrations of Clinical Significance Analyses

Brown (2014) illustrated how Z _cs can be used to assess for clinical significance of outcomes associated with a career development course (Reese & Miller, 2006, 2010) and to track how course changes may affect the clinical significance of the intervention. Brown (2014) also demonstrated how Z _cs can be employed to assess the comparative effectiveness of two interventions that were individually evaluated in two different publications (Grier-Reid, Skaar, & Conkel-Ziebell, 2009; Reed, Reardon, Lenz, & Leierer, 2001). We provide two additional illustrations of using the Z _cs to evaluate intervention outcome—one exploring the comparative effectiveness of two computer-assisted career guidance system (CAGS) interventions and the other evaluating the effectiveness of an individual career counseling intervention.

Betz and Borgen (2009) studied the comparative effectiveness of two CAGS with college students participating in a university career exploration program for students who were undecided about their college majors. One of the CAGS (FOCUS; Career Dimensions, Inc., 2007) generated occupational possibilities, while the other (CAPA) generated college major possibilities for participants. The CDSE-SF was administered at pre- and postintervention and the results revealed a significant main effect for time and a significant Group × Time interaction, suggesting that CAPA participants’ CDSE scores changed more than the FOCUS participants. Using the college student norms provided in Table 3 and pre-course CAPA (3.45) and FOCUS (3.57) means, Z _cs pre-course values were −.69 for the CAPA participants and −.48 for FOCUS. Postintervention Z _cs values were .00 and .05 for the CAPA and FOCUS participants, respectively, suggesting that the changes demonstrated by both groups were clinically significant (post-course means were 3.85 and 3.88 for CAPA and FOCUS, respectively). That is, these data suggested that although participants in both groups started out with CDSE scores that were from three fourths to a half of a standard deviation below the college student mean at pretreatment, they had reached the college mean by postintervention. A better CDSE norm group composed only of college students with declared majors might have revealed less substantial clinical gains, but the results do suggest that participants’ career decision-making self-efficacy beliefs came to resemble closely the self-efficacy beliefs of the average college undergraduate after participating in the course and one of the CAGS.

Masdonati, Massoudi, and Rosier (2009) described the results of a study designed to evaluate the effectiveness of a four- to five-session individual career counseling intervention with Swiss clients. The intervention was based on the theory of work adjustment (Dawis & Lofquist, 1984) and social cognitive career theory (Lent, Brown, & Hackett, 1994), and counselors were trained to incorporate four of the five critical counseling ingredients outline by Brown and Ryan Krane (2000). Outcomes were evaluated with the CDDQ and SWLS as well as with a satisfaction with the intervention measure. The scores on the former two scales obtained by the treatment group were compared to those of a nonequivalent comparison group of nonhelp seekers at pre- and posttreatment. We only present the results of a clinical significance analysis with the treatment group because the comparison group was too nonequivalent to draw any comparative inferences (e.g., their pretreatment means on the CDDQ and SWLS already matched or were better than those obtained via the Swiss norms displayed in Tables 2 and 4). Analyses of pretreatment means of the clients on both the CDDQ (M = 4.44) and SWLS (M = 4.32) using the Swiss norms revealed Z _cs values suggesting substantial career decision difficulties (Z_cs = 1.48) and less than average subjective well-being (Z _cs = −.87). Posttest scores (CDDQ M = 3.64, SWLS M = 5.47) after receiving four to five sessions of counseling suggested substantial clinical gains on the CDDQ (Z_cs = .03) and some gains in subjective well-being (Z _cs = −.27). In other words, participants CDDQ total scores moved from 1.48 standard deviations higher than the norm group mean at pretreatment (higher CDDQ scores indicate increasing career decision-making difficulties) to the norm group mean at posttreatment, although their subject well-being (while improved) was still below the norm group mean.

We hope that these two illustrations as well as those provided in Brown (2014) demonstrate the value of including clinical significance analyses in outcome research. We have primarily showed how Z _cs can be employed to gather data on the clinical significance of average scores obtained in nomothetic research. Stronger (and more useful) evidence of clinical significance would be yielded by using c, RCI, and Z _cs to track individual client change. We think that such methods would be of value to both counselors and researchers. The method provides the former (counselors) with a method to evaluate their practices by allowing them to track each client’s gains, while it provides the latter group (researchers) with a method to analyze data for individual as well as group change. We have provided norms on four outcome measures in this article, but hope to have normative data on several other measures in the future. Finally, we hope that this article proves as innovative as Spokane and Oliver (1983) over the next 30 years.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Meghan Roche was funded with a graduate assistantship from Loyola University Chicago.

References

*Amir

Gati

(2006). Facets of career decision-making difficulties. British Journal of Guidance and Counseling, 34, 483–503.

*Behrens

E. L.

Nota

M. M.

(2014). The Self-Directed Search as a stand-alone intervention with college students. Career Development Quarterly, 2, 224–238.

*Bertoch

S. C.

Lenz

J. G.

Reardon

R. F.

Peterson

G. W.

(2014). Goal instability in relation to career thoughts, decision state, and performance in a career course. Journal of Career Development, 41, 104–121.

Betz

N. E.

Borgen

F. H.

(2009). Comparative effectiveness of CAPA and FOCUS online: Career assessment systems with undecided college students. Journal of Career Assessment, 17, 351–366.

*Betz

N. E.

Hammond

M. S.

Multon

K. D.

(2005). Reliability and validity of a five-level response continuum for the Career Decision Self-Efficacy scale. Journal of Career Assessment, 13, 131–149.

Betz

N. E.

Klein

K. L.

Taylor

K. M.

(1996). Evaluation of a short form of the Career Decision-Making Self-Efficacy scale. Journal of Career Assessment, 4, 47–57.

Björnsdóttir

M. D.

Einarsdóttir

Vilhalmsdóttir

(2011, 9). Two career interventions for upper secondary school students in Iceland: An outcome evaluation. Paper presented at the International Conference on Vocational Designing and Career Counseling: Challenges and New Horizons, Padua, Italy.

Blustein

D. L.

(2006). The psychology of working: A new perspective for career development, counseling, and public policy. Mahwah, NJ: Erlbaum.

Brown

S. D.

(2014). Career intervention efficacy. In Hartung

P. J.

Savickas

M. L.

Walsh

W. B.

(Eds.), APA handbook of career intervention (Vol. 1, pp. 61–77). Washington, DC: American Psychological Association.

10.

Brown

S. D.

Hacker

Abrams

Carr

Rector

Lamp

… Siena

(2012). Validation of a four-factor model of career indecision. Journal of Career Assessment, 20, 3–21.

11.

Brown

S. D.

McPartland

E. B.

(2005). Career interventions: Current status and future directions. In Walsh

W. B.

Savickas

M. L.

(Eds.), Handbook of vocational psychology: Theory, research, and practice (3rd ed., pp. 195–226). Mahwah, NJ: Erlbaum.

12.

Brown

S. D.

Ryan Krane

N. E.

(2000). Four (or five) sessions and a cloud of dust: Old assumptions and new observations about career counseling. In Brown

S. D.

Lent

R. W.

(Eds.), Handbook of counseling psychology (3rd ed., pp. 740–766). New York, NY: John Wiley.

13.

*Bryant

R. M.

Constantine

M. G.

(2006). Multiple role balance, job satisfaction, and life satisfaction among women school counselors. Professional School Counseling, 9, 265–271.

14.

*Bullock-Yowell

Peterson

G. W.

Reardon

R. C.

Leierer

S. J.

Reed

C. A.

(2011). Relationships among career and life stress, negative career thoughts, and career decision state: A cognitive information processing perspective. Career Development Quarterly, 59, 302–314.

15.

*Burns

G. N.

Jasinski

Dunn

Fletcher

(2013). Academic support services and career decision-making self-efficacy in student athletes. Career Development Quarterly, 61, 161–167.

16.

*Buyukgoze-Kavas

(2014). A psychometric evaluation of the Career Decision Self-Efficacy Scale-short form with Turkish university students. Journal of Career Assessment, 22, 386–397.

17.

Career Dimensions, Inc. (2007). FOCUS. Retrieved from http://www.focuscareer2.com

18.

*Chaney

Hammond

M. S.

Betz

N. E.

Multon

K. D.

(2007). The reliability and factor structure the Career Decision Self-Efficacy Scale-SF with African Americans. Journal of Career Assessment, 15, 194–205.

19.

*Cheung

F. M.

Wan

S. L. Y.

Fan

Leong

Mok

P. C. H.

(2013). Collective contributions to career self-efficacy in adolescents: A cross-cultural study. Journal of Vocational Behavior, 83, 237–244.

20.

Choi

K. Y.

Buskey

Johnson

(2010). Evaluation of counseling outcomes at a university counseling center: The impact of clinically significant change on problem resolution and academic functioning. Journal of Counseling Psychology, 57, 297–303.

21.

Chronister

K. M.

McWhirter

E. H.

(2006). An experimental examination of two career interventions for battered women. Journal of Counseling Psychology, 53, 151–164.

22.

*Chung

Y. B.

(2002). Career decision-making self-efficacy and career commitment: Gender and ethnic differences among college students. Journal of Career Development, 28, 277–284.

23.

*Conklin

A. M.

Dahling

J. J.

Garcia

P. A.

(2013). Linking affective commitment, career self-efficacy, and outcome expectations. A test of Social Cognitive Career Theory. Journal of Career Development, 40, 68–83.

24.

Davidson

M. M.

Nitzel

Duke

Baker

C. M.

Bovaird

J. A.

(2012). Advancing career counseling and employment support for survivors: An Intervention Evaluation. Journal of Counseling Psychology, 59, 321–328.

25.

Dawis

R. V.

Lofquist

L. H.

(1984). A Psychological Theory of Work Adjustment. Minneapolis, MN: University of Minnesota Press.

26.

Diener

E. D.

Emmons

R. A.

Larsen

R. J.

Griffin

(1985). The satisfaction with life scale. Journal of Personality Assessment, 49, 71–75.

27.

Di Fabio

Kenny

M. E.

(2011). Promoting emotional intelligence and career decision making among Italian high school students. Journal of Career Assessment, 19, 21–34.

28.

*Dipeolu

Reardon

Sampson

Burkhead

(2002). The relationship between dysfunctional career thoughts and adjustment to disability in college students with learning disabilities. Journal of Career Assessment, 10, 413–427.

29.

Fitzgerald

E. L.

Chronister

K. M.

Forrest

Brown

(2013). Options for preparing inmates for community reentry: An employment preparation intervention. The Counseling Psychologist, 41, 990–1010.

30.

*Flores

L. Y.

Ojeda

Huang

Y. P.

Gee

Lee

(2006). The relation of acculturation, problem-solving appraisal, and career decision-making self-efficacy to Mexican American high school students’ educational goals. Journal of Counseling Psychology, 53, 260–266.

31.

Fretz

B. R.

(1981). Evaluating the effectiveness of career interventions. Journal of Counseling Psychology, 28, 77–90.

32.

*Garcia

P. R. J. M.

Restubog

S. L. D.

Toledang

L. S.

Tolentino

L. R.

Rafferty

A. E.

(2012). Differential moderating effects of student- and parent-rated support in the relationship between learning goal orientation and career decision-making self-efficacy. Journal of Career Assessment, 20, 22–33.

33.

Gati

Amir

(2010). Applying a systematic procedure to locate career decision-making difficulties. Career Development Quarterly, 58, 301–320.

34.

*Gati

Krausz

Osipow

S. H.

(1996). A taxonomy of difficulties in career decision-making. Journal of Counseling Psychology, 43, 510–526.

35.

Gati

Levin

(2014). Counseling for career decision-making difficulties. Career Development Quarterly, 62, 98–113.

36.

*Gati

Osipow

S. H.

Krausz

Saka

(2000). Validity of the Career Decision-Making Difficulties Questionnaire: Counselee versus career counselor perceptions. Journal of Vocational Behavior, 56, 99–113.

37.

*Gati

Saka

(2001). High school students’ career-related decision-making difficulties. Journal of Counseling and Development, 79, 347–364.

38.

*Gaudron

J.-P.

(2011). A psychometric evaluation of the Career Decision Self-Efficacy Scale- Short Form among French university students. Journal of Career Assessment, 19, 420–430.

39.

Grier-Reed

T. L.

Skaar

N. R.

Conkel-Ziebell

J. L.

(2009). Constructivist career development as a paradigm of empowerment for at-risk culturally diverse college students. Journal of Career Development, 35, 290–305.

40.

*Gushue

G. V.

Clark

C. P.

Pantzer

K. M.

Scanlon

K. R. L.

(2006). Self-efficacy, perceptions of barriers, vocational identity, and the career exploration behaviors of Latino/a high school students. Career Development Quarterly, 54, 307–317.

41.

Hacker

Carr

Abrams

Brown

S. D.

(2013). Development of the Career Indecision Profile: Factor structure, reliability, and validity. Journal of Career Assessment, 21, 32–41.

42.

*Hardin

E. E.

Donaldson

J. R.

(2014). Predicting job satisfaction: A new perspective on person-environment fit. Journal of Counseling Psychology, 61,634–640.

43.

Hirschi

Lage

(2008). Increasing the career choice readiness of young adolescents: An evaluation study. International Journal for Educational and Vocational Guidance, 8, 95–110.

44.

*Hsieh

H. H.

Huang

J. T.

(2014). The effects of socioeconomic status and proactive personality on career decision self-efficacy. Career Development Quarterly, 62, 29–43.

45.

*Hubell

Gustvasson

J. P.

(2008). A psychometric evaluation of the Satisfaction with Life Scale in a Swedish nationwide sample of university students. Personality and Individual Differences, 44, 1070–1079.

46.

Jacobson

N. S.

Follett

W. C.

Revensdorf

(1986). Toward a standard definition of clinically significant change. Behavior Therapy, 17, 308–311.

47.

Jacobson

N. S.

Truax

(1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 12–19.

48.

*Jin

Watkins

(2012). The dimensionality of the Career Decision Self-Efficacy Scale-Short Form among Chinese graduate students. Journal of Career Assessment, 20, 520–529.

49.

Kim

B. S. K.

Atkinson

D. R.

(2002). Asian American client adherence to Asian cultural values, counselor expression of cultural values, counselor ethnicity, and counseling process. Journal of Counseling Psychology, 49, 3–13.

50.

*Kim

Jang

S. H.

Jung

S. H.

Lee

B. H.

Puig

Lee

S. M.

(2014). A moderated mediation model of planned happenstance skills, career engagement, career decision self- efficacy, and career decision status. Career Development Quarterly, 62, 56–69.

51.

*Komarraju

Swanson

Nadler

(2014). Increased career self-efficacy predicts college students’ motivation and course major satisfaction. Journal of Career Assessment, 22, 420–432.

52.

*Koumoundourou

G. A.

Kounenoce

Sia Vara

(2012). Core self-evaluations, career decision self-efficacy, and vocational identity among Greek adolescents. Journal of Career Development, 39, 269–286.

53.

*Koumoundourou

G. A.

Tsaousis

Kounenou

(2011). Parental influences on Greek adolescents’ career decision-making difficulties: The mediating role of core self- evaluations. Journal of Career Assessment, 19, 165–182.

54.

Lambert

M. J.

Finch

A. M.

Okiishi

Burlingame

G. M.

Mckelvey

Reisenger

C. W.

(1998). Administration and scoring manual for the OQ-10.2. Stevenson, MD: American Professional Credentialing Services, LTD.

55.

*Lancaster

B. P.

Rudolph

C. E.

Perkins

T. S.

Patten

T. G.

(1999). The reliability and validity of the Career Decision Difficulty Questionnaire. Journal of Career Assessment, 7, 393–414.

56.

*Lease

S. H.

(2004). Effects of locus of control, work knowledge, and mentoring on career decision-making difficulties. Journal of Career Assessment, 12, 239–254.

57.

Lent

R. W.

Brown

S. D.

Hackett

(1994). Toward a unifying social cognitive theory of career and academic interests, choice, and performance. Journal of Vocational Behavior, 45, 79–122.

58.

*Lustig

D. C.

Strauser

D. R.

(2002). The relationship between sense of coherence and career thoughts. Career Development Quarterly, 51, 2–11.

59.

*Masdonati

Massoudi

Rosier

(2009). Effectiveness of career counseling and the impact of the working alliance. Journal of Career Development, 36, 183–203.

60.

*Mau

W. C.

(2001). Assessing career decision-making difficulties: A cross-cultural study. Journal of Career Assessment, 9, 353–364.

61.

McWhirter

E. H.

Rasheed

Crothers

(2000). The effects of high school career education on social-cognitive variables. Journal of Counseling Psychology, 47, 330–341.

62.

*Metheny

McWhirter

E. H.

(2013). Contributions of social status and family support to college students’ career decision-making self-efficacy and outcome expectations. Journal of Career Assessment, 21, 378–394.

63.

Miller

Brown

S. D.

(2005). Career choice interventions. In Brown

S. D.

Lent

R. W.

(Eds.), Career development and counseling: Putting theory and research to work (pp. 441–465). New York, NY: John Wiley.

64.

Multon

K. D.

Heppner

M. J.

Gysbers

N. C.

Zook

Ellis-Kalton

(2001). Client Psychological distress: An important factor in career counseling. Career Development Quarterly, 49, 324–335.

65.

*Navarro

R. L.

Ojeda

Schwartz

Pina-Watson

Lina

L. L.

(2014). Cultural self and personal self: Links with life satisfaction among Mexican American college students. Journal of Latino Psychology, 2, 1–20.

66.

Nota

Soresi

(2004). Improving the problem-solving and decision-making skills of a high indecision group of young adolescents: A test of the “Difficult: No Problem!” training. International Journal of Educational and Vocational Guidance, 4, 3–21.

67.

O’Brien

K. M.

Dukstein

R. D.

Jackson

S. L.

Tomlinson

M. J.

Kamatuka

N. A.

(1999). Broadening career horizons for students in at-risk environments. Career Development Quarterly, 47, 215–229.

68.

Oliver

L. W.

Spokane

A. R.

(1988). Career intervention outcome: What contributes to client gain? Journal of Counseling Psychology, 35, 447–462.

69.

*Osborn

D. S.

Howard

D. K.

Leierer

S. J.

(2007). The effect of a career development course on the dysfunctional career thoughts of racially and ethnically diverse college freshmen. Career Development Quarterly, 55, 365–377.

70.

*Osipow

S. H.

Gati

(1998). Construct and concurrent validity of the Career Decision-Making Difficulties Questionnaire. Journal of Career Assessment, 6, 347–364.

71.

*Pavot

Diener

(1993). Review of the Satisfaction with Life Scale. Psychological Assessment, 5, 164–170.

72.

Peng

(2001). Comparing the effectiveness of two different career education courses on career decidedness for college freshman: An Exploratory study. Journal of Career Development, 28, 29–41.

73.

Peng

Herr

E. L.

(1999). The impact of career education courses on career beliefs and career decision making among business college students in Taiwan. Journal of Career Development, 25, 275–290.

74.

*Peterson

Park

Seligman

M. E. P.

(2005). Orientations to happiness and life satisfaction: The full life versus the empty life. Journal of Happiness, 6, 25–41.

75.

*Pina-Watson

Jimenez

Ojeda

(2014). Self-construal, career decision self-efficacy, and perceived career barriers predict Mexican-American Women’s life satisfaction. Career Development Quarterly, 62, 210–223.

76.

Plimmer

(2012). Adult career counseling using possible selves—A quasi-experimental field study in naturalistic settings. Journal of Career Assessment, 20, 53–70.

77.

*Railey

M. G.

Peterson

G. W.

(2000). The assessment of dysfunctional career thoughts and interest structure among female inmates and probationers. Journal of Career Assessment, 8, 119–129.

78.

Reardon

R. C.

Lenz

J. G.

Sampson

J. P.

Jr Peterson

G. W.

(2000). Career development and planning: A comprehensive approach. Belmont, CA: Brooks/Cole.

79.

*Reed

C. A.

Reardon

R. C.

Lenz

J. G.

Leierer

S. J.

(2001). A cognitive career course: From theory to practice. Career Development Quarterly, 50, 158–167.

80.

Reese

R. J.

Miller

C. D.

(2006). Effects of a university career development course on career decision-making self-efficacy. Journal of Career Assessment, 14, 252–266.

81.

Reese

R. J.

Miller

C. D.

(2010). Using outcome to improve a career development course: Closing the scientist-practitioner gap. Journal of Career Assessment, 18, 207–219.

82.

Rochlen

A. B.

Milburn

Hill

C. E.

(2004). Examining the process and outcome of career counseling for different types of career clients. Journal of Career Development, 30, 263–275.

83.

Rounds

J. B.

Tinsley

T. J.

(1984). Diagnosis and treatment of vocational problems. In Brown

S. D.

Lent

R. W.

(Eds.), Handbook of counseling psychology (pp. 137–177). New York, NY: John Wiley.

84.

Saka

Gati

Kelly

K. R.

(2008). Emotional and personality-related aspects of career decision-making difficulties. Journal of Career Assessment 16, 403–424.

85.

*Sampson

J. P.

Jr Peterson

G. D.

Lenz

J. G.

Reardon

R. C.

Sanders

D. E.

(1996). Career thoughts inventory professional manual. Lutz, FL: Psychological Assessment Resources.

86.

Shea

P. W.

Yeh

C. J.

Lee

S. J.

Pituc

S. T.

(2009). Exploratory studies on the effects of a career exploration group for urban Chinese immigrant youth. Journal of Career Assessment, 17, 457–477.

87.

*Sovet

Metz

A. J.

(2014). Parenting styles and career decision-making among French and Korean adolescents. Journal of Vocational Behavior, 84, 345–355.

88.

Spokane

A. R.

Oliver

L. W.

(1983). Outcomes of vocational interventions. In Osipow

S. H.

Walsh

W. B.

(Eds.), Handbook of vocational psychology (pp. 99–136). Hillsdale, NJ: Erlbaum.

89.

*Strauser

D. R.

Lustig

D. C.

Ciftci

(2008). Psychological well-being: Its relation to work personality, vocational identity, and career thoughts. Journal of Psychology, 142, 21–35.

90.

Thrift

M. M.

Ulloa-Heath

Reardon

R. C.

Peterson

G. W.

(2012). Career interventions and career thoughts of Pacific Island college students. Journal of Counseling and Development, 90, 169–176.

91.

*Tien

H. L. S.

(2005). The validation of the Career Decision-Making Difficulties Questionnaire in a Chinese culture. Journal of Career Assessment, 13, 114–127.

92.

*Tirpak

D. M.

Schlosser

L. Z.

(2013). Evaluating FOCUS-2’s effectiveness in enhancing first-year college students’ social cognitive career development. Career Development Quarterly, 61, 110–123.

93.

Turner

S. L.

Conkel

J. L.

(2010). Evaluation of a career development skills intervention with adolescents living in the inner city. Journal of Counseling and Development, 88, 457–465.

94.

Whitson

S. C.

(2001). Selecting career outcome assessments: An organizational scheme. Journal of Career Assessment, 9, 215–228.

95.

Whiston

S. C.

Brecheisen

B. K.

Stephens

(2003). Does treatment modality affect career counseling effectiveness? Journal of Vocational Behavior, 62, 390–410.

96.

Whiston

S. C.

James

B. N.

(2013). Promotion of career choices. In Brown

S. D.

Lent

R. W.

(Eds.), Career development and counseling: Putting theory and research to work (2nd ed., pp. 565–594.). New York, NY: John Wiley.

97.

Whiston

S. C.

Rahardja

(2008). Vocational counseling process and outcome. In Brown

S. D.

Lent

R. W.

(Eds.), Handbook of counseling psychology (4th ed., pp. 444–461). New York, NY: John Wiley.

98.

Whiston

S. C.

Sexton

T. L.

Lasoff

D. L.

(1998). Career intervention outcome: A replication and extension of Oliver and Spokane (1988). Journal of Counseling Psychology, 45, 150–165.

99.

*Williams

M. W.

Rapport

L. J.

Millis

S. R.

Hanks

R. A.

(2014). Psychosocial outcomes after traumatic brain injury: Life satisfaction, community integration, and distress. Rehabilitation Psychology, 59, 298–305.

100.

*Wright

L. K.

Reardon

R. C.

Peterson

G. W.

Osborn

D. S.

(2000). The relationship among constructs in the Career Thoughts Inventory and the Self-Directed Search. Journal of Career Assessment, 8, 105–117.

101.

*Xu

Zhi-Jin

Tracey

T. J. G.

(2014). Relation of environmental and self-exploration with career decision-making difficulties in Chinese students. Journal of Career Assessment, 22, 654–665.