Abstract
We examined the relative effectiveness of information giving (IG) and therapeutic assessment (TA) models of career assessment feedback in reducing career indecision. Clients initiating career counseling at a large Midwestern University completed measures of career choice anxiety (CCA), vocational identity, and career decision-making self-efficacy prior to and following a two-session intervention. We also administered measures of treatment integrity and session helpfulness immediately following treatment. Clients were randomly assigned to an intervention based upon either an IG or TA model of assessment feedback. Results indicated that TA participants’ vocational identity scores improved significantly more than those of IG participants; a medium-to-large effect size was identified. TA and IG participants’ CCA and decision-making self-efficacy scores significantly increased but not differentially following treatment. Participants of both groups rated their feedback sessions as “greatly helpful.” We discuss implications for career assessment as well as the limitations of the study.
Keywords
According to recent U.S. Department of Education statistics, more than 18 million students attended degree granting postsecondary institutions in 2009 (Aud et al., 2011). It is well established that many students experience college major and career indecision (Gordon, 2007). Indecision can be reflected by self-reports and changes of declared majors. Regarding the latter, Gordon (2007) reported that as many as 75% of college students change their undergraduate major at least once. Further, approximately 7% of college-bound students who took the SAT in 2009–2010 reported that they were undecided about their college major (College Entrance Examination Board, 2010). The American College Testing (ACT) reported that an even larger number—14% of college-bound students—reported indecision about their college major (ACT, 2009). Research has demonstrated that indecision has negative effects on college student persistence, academic performance, and timely degree completion (Allen & Robbins, 2010; Tracey & Robbins, 2006).
Because college major and career indecision are commonly experienced and have negative consequences, higher education institutions have invested significantly in student support services. The U.S. Department of Education reported that 2- and 4-year postsecondary institutions invest approximately 4.7% of their budgets (Aud et al., 2011) in student services, including career counseling. The U.S. higher education system has placed a high priority on efficient college major decision making and expeditious degree completion and career entry. Therefore, those who provide support services to higher education students have much to gain by developing and implementing more effective career counseling interventions.
Career Counseling Outcome Research
Outcome research has demonstrated that career interventions are at least moderately effective. Four significant meta-analyses have reported average overall effect sizes ranging from .34 to .82 (Oliver & Spokane, 1988; Ryan, 1999; Whiston, Brecheisen, & Stephens, 2003; Whiston, Sexton, & Lasoff, 1998). These effect sizes indicate that the average career counseling participant experiences more positive outcomes than 63–79% of control group participants.
Three implications can be drawn from these meta-analyses. First, the weight of evidence indicates the necessity for counselor involvement in career counseling. Both Whiston, Sexton, and Lasoff (1998) and Whiston, Brecheisen, and Stephens (2003) found that counselor-free intervention is the least effective modality. Second, individual counseling and structured workshop/classes appear to be the most effective career counseling modalities (Oliver & Spokane, 1988; Whiston et al., 1998, 2003). Third, Ryan (1999) identified five critical ingredients of effective career interventions: individualized interpretations and feedback, providing world-of-work information, modeling effective career decision-making processes, engaging clients in written exercises, and choice-related support. Accordingly, these treatment components should be the focus of career counseling outcome studies.
Although the aforementioned meta-analyses have produced valuable information that informs practice, there has been a general decline in the volume of career counseling outcome research (Whiston et al., 2003). More recently, Scheel et al. (2011) identified a decrease in career counseling research studies in the top primary counseling psychology publications over the past 30 years. This decrease can be juxtaposed against the multiple calls of researchers for more critical examination of career counseling interventions (e.g., Bernes, Bardick, & Orr, 2007; Heppner & Heppner, 2003; Whiston, 2011). Whiston et al. (2003) specifically highlighted the lack of research comparing career assessment feedback methods. The lack of comparative research regarding assessment feedback is particularly problematic because of the central role of interest assessment in career counseling (Hansen, 2005) and the increase in Internet-mediated career assessment tools available directly to students and counselors (e.g., Tracey, 2010). Three points should be considered. First, interest assessment will remain an essential component of career counseling. Second, counselor-mediated interventions provide superior outcomes. Third, the costs of counseling personnel are increasing in a higher education budget with limited funding. Therefore, it is important for researchers to continue exploring the most effective methods of counselor involvement in career assessment feedback.
Therapeutic Assessment
Claiborn, Goodyear, and Horner (2001) reviewed research on assessment feedback methods and provided the following recommendations. They suggested that assessment feedback is more effective when it (a) involves a collaborative therapeutic relationship, (b) is conducted in a structured environment, (c) involves discussion of feedback comprehension, and (d) follows goal or question setting. Therapeutic assessment (TA; Finn & Tonsager, 2002) is a promising approach to assessment feedback that incorporates these components.
Finn and Tonsager (2002) defined TA as “a semi-structured form of collaborative assessment that uses psychological testing as the centerpiece of brief therapy” (p. 10). The TA process is based on three defining principles. The first principle, self-verification, was derived from Kohut’s (1977) theory of self-psychology that stipulates that many problems of the self, particularly anxiety, result from failed integration of self-view with the view mirrored by one’s environment. A primary counseling focus is to enable clients to attain an accurate and cohesive view of self (Patton, Connor, & Scott, 1982). In TA, self-verification occurs by providing clients with feedback that closely mirrors their self-views. Subsequently, new information is provided to facilitate the formation of a more cohesive and encompassing sense of self.
The second primary principle of TA is self-enhancement, which is derived from social cognitive and self-psychology theories (Finn & Tonsager, 1997). According to Fiske and Taylor (2008), people are motivated to be positively viewed by themselves and others. Combining self-enhancement with the principle of self-verification, clients in TA are given information initially determined to closely match their self-view and then presented with new and potentially discrepant information, which results in an enhanced sense of self for clients. To further self-enhancement, TA incorporates the humanistic principles of respect, empathy, and positive regard of client qualities. With a more cohesive sense of self, clients are able to consider methods for achieving assessment-directed goals.
The third principle of TA is self-efficacy/self-discovery, which is rooted in Bandura’s social cognitive theory. Bandura (1993) referenced the basic human need for mastery and control over one’s environment. He postulated that self-efficacy is developed through personal mastery, vicarious experience, social persuasion, or analysis of physiological state. In gaining mastery and environmental control, people gain increased efficacy to perform a particular set of actions. In TA, clients are actively engaged to form personal goals for assessment and develop personally relevant questions for exploration. Self-efficacy/self-discovery is fostered as clients collaborate in discovering assessment information relevant to their questions and goals (Finn & Tonsager, 1997). The TA process serves to verify a client’s experience, positively reframe and enhance negative self-evaluations, and facilitate client self-discovery.
TA shares some common features with the traditional method of assessment feedback, which we refer to as the information giving (IG) model. The central purpose of the IG approach is for the assessor to provide clinically relevant information to clients to guide decision making and action planning. The IG approach, which has been labeled test-and-tell or delivered assessment, frequently has been employed to provide psychological and career assessment feedback (e.g., Cutler, 2004; Finn & Tonsager, 1997; Goodyear, 1990; Hanson, Claiborn, & Kerr, 1997).
The IG approach has been characterized as “interpreted to the client at a particular fixed point (e.g., during the second interview) in the counseling process; often this is in a ‘data dump’ fashion and with the counselor maintaining responsibility for interpretations and inferences” (Goodyear, 1990, p. 252). Cutler (2004) described the IG approach as consisting of data gathering, interpretation, and client feedback steps. The primary distinction between the two approaches is that IG is test and therapist centered and TA is client and relationship centered. By shifting the focus from the test to the client, TA feedback honors the reality that “how we interpret an event greatly influences how we respond to it” (Finn, 1999, p. 1) The TA model represents small but significant modifications to the IG model (Finn & Tonsager, 1997). These minor adjustments retain validity and objectivity while enhancing the impact of assessment feedback for the client.
The Present Study
We conducted this study to compare TA and IG methods of career assessment feedback. In conducting this study, we incorporated four research design recommendations from previous investigators, including (a) direct counselor involvement in career assessment feedback, (b) intentionality of study outcome measurement selection, (c) inclusion of treatment integrity checks, and (d) use of real career counseling clients. We hypothesized that TA would yield significantly greater reduction of career indecision than the traditional IG approach. We had three general expectations regarding the incremental benefits of TA over the IG approach reflecting the TA tenets of self-verification, self-efficacy, and self-discovery. First, TA should be more powerful than IG in strengthening vocational identity. Second, TA should yield higher career decision-making efficacy (CDSE). Third, TA should decrease CCA.
We had five hypotheses regarding the relative positive effects of TA in comparison to IG immediately following assessment feedback: TA condition participants will demonstrate a significantly greater increase in CDSE. TA participants will report a significantly greater increase in vocational identity. TA participants will indicate a significantly greater decrease in CCA. TA participants will rate their assessment feedback session as significantly more helpful. TA and IG participants together will demonstrate significantly higher CDSE and vocational identity and lower CCA following completion of the respective assessment processes.
Method
Participants
Clients
Twenty-three participants, including 18 men and 5 women, completed the study and outcome measures in full. Participants were actual clients seeking career counseling services at a large Midwestern university counseling center. An additional 22 participants began but did not complete the study due to scheduling problems, choosing not to complete the career assessment or failing to return for their feedback sessions. Sixteen participants reported ethnicity, with 14 identifying as Caucasian/White, 1 as African, and 1 as Asian American. The age range, reported by 20 of the 23 participants, was 17–27 with a mean age of 20.54. Of the 17 students who reported their academic status, 6 were first-year, 6 were second-year, 2 were third-year, and 1 was a fourth-year student. The sample also included one graduate student and one alumni/community member. Note that not all participants reported demographic information.
Counselors
Seven members of the counseling center staff were recruited and trained as counselors for this study. Due to scheduling difficulties, only four counselors had participants who completed the study in full. These four counselors were either currently completing their American Psychological Association (APA)-accredited doctoral internship in psychology or had completed their doctoral degree in counseling or clinical psychology. All four counselors had familiarity and experience conducting career assessment and feedback prior to the study. Each counselor completed a training session covering study protocols. Following protocol training, all counselors conducted assessment sessions for both the TA and IG treatment conditions. Table 1 shows relevant counselor demographics of the four counselors participating in the study.
Counselor Demographics.
Measures
Career Factors Inventory (CFI; Chartrand & Robbins, 1997)
The CFI has a total of 21 items with a 5-point Likert-type response scale; the CFI is intended to measure career needs and potential barriers to decision making. There are four CFI scales: Need for Career Information, Need for Self-Knowledge, CCA, and Generalized Indecisiveness (Chartrand, Robbins, Morrill, & Boggs, 1990). Chartrand et al. (1990) reported respective CFI scale score internal reliability coefficients of .86, .79, .73, and .83 with a total scale reliability of .87. These results indicate sufficient to good internal consistency. The authors also were able to use CFI scores to distinguish between students high and low in academic major choice decidedness. Chartrand, Robbins, Morrill, and Boggs (1990) and Lewis and Savickas (1995) demonstrated that CFI scale scores are significantly correlated with other previously validated, theoretically similar instruments. The CCA scale was designed to measure anxiety experienced in relation to the career decision-making process. Research indicates that the CCA scale is related to similar constructs such as trait anxiety. We used the CCA scale in this study in consideration of the TA tenet of self-verification because anxiety around the decision-making process should be lower for clients with greater self-verification. We calculated pretest and posttest internal reliabilities of .890 and .802, respectively, for the CCA scores of the study sample.
Career Decision-Making Self-Efficacy Scale–Short Form (CDSE-SF; Betz & Taylor, 2001)
Taylor and Betz (1983) developed the CDSE to apply Bandura’s (1977) self-efficacy theory to career decision making, with the specific aim of measuring self-efficacy related to Crites’s (1978) five career choice competencies. Consequently, CDSE reflects confidence in one’s ability to complete five tasks theoretically related to career decision making, including: self-appraisal, occupational information, goal selection, planning, and problem solving (Nilsson, Schmidt, & Meek, 2002). The CDSE-SF has a total of 25 items and the same five scales as the original. Each item is scored on a 10-point Likert-type scale ranging from 1 (no confidence at all) to 10 (complete confidence). In a review of 14 studies employing the CDSE-SF, Nilsson, Schmidt, and Meek (2002) reported scale score reliabilities to range from .71 to .82, .78 to .82, .83, .77 to .83, and .69 to .75, respectively. The mean total reliability coefficient was .94. Nilsson et al. found no statistical differences in the reliability properties of the CDSE and CDSE-SF.
Concurrent validity has been found between CDSE-SF scores and other career indecision measures. Betz, Klein, and Taylor (1996) found positive associations between the total CDSE-SF scores and positive career decision-making outcomes such as higher decision-making certainty and vocational identity and lower career indecision. We used the CDSE-SF total score in consideration of the TA tenets of self-efficacy/self-discovery and self-enhancement. We calculated pretest and posttest internal reliabilities of .902 and .913, respectively, for the CDSE-SF scores of the study sample.
My Vocational Situation (MVS; Holland, Daiger, & Power, 1980)
The MVS is a 26-item measure intended to assess vocational identity, need for career information, and career decision-making barriers. It consists of three separate scales including the 18-item Vocational Identity scale (VIS), 4-item Occupational Information, and 4-item Barriers scales. An additional free response question, which precedes the three primary scales, asks the respondent to “List all the occupations you are considering right now.” Using scores from a college student sample, Holland, Gottfredson, and Power (1980) reported internal consistency reliability estimates for the three MVS scales of .89, .79, and .45, respectively, for men; the estimates were .88, .77, and .65, respectively, for women. The VIS items are negatively scored and stated as true–false responses with higher scores indicating a clearer vocational identity (Holland, Johnston, & Asama, 1993). Example items include I don’t know what my major strengths are and I am uncertain about which occupation I would enjoy.
Holland et al. (1993) summarized the MVS research and concluded that VIS scores are positively correlated with measures of career development, decision making, ego identity, conscientiousness, choice confidence, hope, and self-esteem. They indicated that VIS scale scores were negatively correlated with measures of undecided beliefs, neuroticism, dependent decision-making style, CCA, indecisiveness, and need for career information and self-knowledge. We used the VIS in consideration of the TA tenet of self-verification. Note that the VIS was not negatively scored because the purpose of this study was to detect pre/post change. Thus, lower VIS scores in our results reflect clearer vocational identity. We calculated pretest and posttest internal reliabilities of .838 and .702, respectively, for the VIS scores of the study sample.
Counselor Rating Form–Short Form (CRF-SF; Corrigan & Schmidt, 1983)
The CRF-SF, a 12-item version of the full CRF, is used to assess client perceptions of counselor attractiveness, expertness, and trustworthiness. Each scale consists of positive adjective items rated on a 7-point scale from not very to very. Higher scores reflect more positive perceptions of the counselor. Epperson and Pecnik (1985) reported internal consistency coefficients of .88, .86, and .88, respectively, with an average value of .87.
Several researchers have reported confirmation of a three-factor structure for the CRF-SF (Corrigan & Schmidt, 1983; Epperson & Pecnik 1985; Tracey, Glidden, & Kokotovic, 1988). Tracey, Glidden, and Kokotovic (1988) also reported a global construct of overall counselor positivity perception. Corrigan and Schmidt (1983) reported high intercorrelations between the three scales’ scores ranging from .381 to .900. Further evidence of validity was provided by Kokotovic and Tracey (1987), who found that client CRF-SF scores on the trustworthiness and expertness scales were significantly related to client dropout and continuation behavior. In consideration of the high correlations among the three theoretical constructs and evidence of a global counselor positivity measure, we used the total counselor rating and three separate CRF-SF scales in this study. Using the current study data, we calculated internal reliabilities of .957 for the total scale scores, .945 for expertness, .926 for attractiveness, and .884 for the trustworthiness scale.
Assessment
We used the Strong Interest Inventory (SII; Donnay, Morris, Schaubhut, & Thompson, 2005) to provide clients with feedback. The SII is a career interest assessment tool widely used in both counseling practice and research; it has undergone several revisions. The current SII consists of 291 items that yield four sets of subscale scores. All scale scores are presented as standard scores. The General Occupational Themes (GOT) measure preferences within each of the Holland six interest types (i.e., realistic, investigative, artistic, social, enterprising, and conventional). The Basic Interest Scales (BIS) measure interest in 30 types of activities organized by primary interest type (e.g., performing arts and culinary arts within the artistic theme; counseling and helping and teaching and education with the social theme, sales and management within the enterprising theme). The Occupational Scales (OS) show the respondent’s similarity to groups of satisfied workers of their sex in 244 different occupations. Occupations are organized within each of the six interest types according to the occupation’s primary interest theme. Realistic occupations include electrician, forester, and radiologic technologist. Investigative occupations include dietician, pharmacist, and mathematician. The five dichotomous Personal Style Scales (PSS) reflect preferences for work style (working alone vs. with people), learning environment (practical vs. academic), leadership style (level of comfort), risk taking, and team orientation (alone vs. group). The GOT, BIS, and OS items are rated on a 5-point scale ranging from strongly like to strongly dislike.
The following psychometric data are primarily based on the 2004 revision of the SII. It is significant to note that correlations between scores on the 1994 and 2004 versions of the SII have been reported to range from .90 to .98 for the GOTs and from .80 to .98 between parallel BISs. The internal consistency coefficients for the Realistic, Investigative, Artistic, Social, Enterprising, and Conventional (RIASEC) GOT scale scores were all .91 or greater (Donnay et al., 2005). The BIS scores had a mean internal consistency coefficient of .87. The median test–retest reliability for the OS scores was .86. The internal consistency coefficients for the PSS scores were in the range of .82 to .87. A significant amount of research has demonstrated the concurrent, construct, and discriminant validity of the SII scales (Donnay et al., 2005; Gasser, Larson, & Borgen, 2007).
Treatment Checks and Perception
We used treatment validity check and session helpfulness items similar to those developed by Hanson and Claiborn (2006). The session helpfulness question was modified to a 9-point scale following Elliott’s (1985) rating method. To assess treatment perception each participant responded to the following questions with 5-point response scales after the assessment feedback session: “To what extent was the testing process personalized by … ” (a) “having you develop questions, prior to taking the test, about yourself you wanted answered?” and (b) “being involved actively in the interpretation of the test results?” They then responded to the helpfulness question with a 9-point scale “How helpful or hindering was your test feedback session?” Higher scores indicated greater perceived session involvement and helpfulness.
Procedure
Counselor Training
The recruited counselors were trained in two 2-hr sessions consisting of didactic and role-play learning. The first author conducted the training sessions. The counselors were asked to review the SII manual and the TA and IG protocols developed for the study. Each counselor was asked to memorize both protocols and practice mock sessions, one of which was taped, using each protocol until judged competent by the trainer. Competence was determined based on counselor’s successful adherence to a protocol checklist. At the request of the counselors, we developed a quick reference sheet for the protocols for in-session use. As an additional check, two independent undergraduate students rated the taped mock session of each counselor with the CRF-SF. The CRF-SF scores were compared using an analysis of variance (ANOVA) to determine if the counselors were perceived similarly in terms of their expertness, attractiveness, and trustworthiness.
Test Interpretation and Feedback
Our test interpretation and feedback procedures were similar to those reported by Cutler (2004) in her comparison of TA and IG approaches. Of note, Cutler’s feedback procedures and protocol were adapted from assessment feedback protocols by Finn (1996), Hanson (2001), and Pawlowski (2002). The IG feedback protocol used in this study additionally incorporated the recommended interpretation format detailed in the 1994 SII manual (Harmon, Hansen, Borgen, & Hammer, 1994). The TA protocol was modified to include elements consistent with the principles of self-verification, self-enhancement, and self-efficacy/self-discovery.
Participation was invited by the counseling center reception staff as clients checked in for their intake sessions. Those clients who agreed to participate completed the CFI, MVS, and CDSE-SF pretest measures prior to their intake session with a counselor. Participants then completed their intake session according to the assigned feedback protocol. Each participating counselor alternated between TA and IG protocols for each successive participant. The participants then were scheduled to complete the SII following their intake sessions.
In the first intervention session, participants met with their assigned counselor prior to taking the SII for a 45-min session that was used to either (a) gather information only about the participant’s career concerns (IG condition) or (b) gather information and establish collaborative questions and goals (TA condition). For the second and final intervention session, participants received SII feedback according to their assigned treatment condition. At the conclusion of the second session, participants completed the posttest measures of the CFI, MVS, and CDSE-SF and the treatment check and reaction questions. Career counseling continued at the discretion of the counselor and client following study completion.
Results
Data Screening and Preanalysis Results
All outcome variables were screened for normality and outliers following procedures outlined by Tabachnik and Fidell (2001). To evaluate any potential covariate effects, difference scores were calculated between participants’ pre- and posttreatment scores on the three dependent measures. Correlations were then evaluated between the difference scores and participant age, sex, and academic status. No correlations were significant, indicating that difference scores were not significantly related to demographic characteristics. We also calculated Pearson correlations between participant difference and pretest scores and all three were statistically significant. Three separate ANOVAs were calculated with treatment group as the independent variable and pretest scores as the respective dependent variables. None of the ANOVAs were significant, indicating that treatment group did not affect the dependent measures prior to treatment. In consideration of the significant correlation between participant difference and pretest scores, we decided to use one-way analysis of covariances (ANCOVAs) with pretest scores as a covariate, posttest scores as the dependent variable, and treatment group as the independent variable for Hypotheses 1, 2, and 3.
A total of 45 participants initially completed the pretest measures. Of those 45 participants, 22 failed to complete the SII or return for a feedback session. Twelve participants who did not complete the study either reported some demographics or completed a portion of the pretest measures. Of the 12, 7 were women and 5 were men. There were five first-year, two second-year, two third-year, and two fourth-year students. Their average age was 23.18 with a range of 18–42. Nine reported Caucasian/White as their ethnicity. We conducted three ANOVAs using completion status (completed vs. failure to complete) as the independent variable and pretest scores as the dependent variable. There were no significant effects for completion status. We compared other characteristics of the completed and dropout groups and could not identify any characteristics distinguishing the two groups. Ultimately, 23 participants completed both sessions and the pre- and post measures with one participant failing to complete the pretest CFI and VIS measures and one omitting the posttest CFI measure.
Counselor Check
Two undergraduate students blind to the hypotheses viewed 15-min segments of each counselor’s taped mock session and then completed the CRF-SF for each counselor. We conducted four ANOVAs with counselor as the independent variable and CRF-SF total, expertness, attractiveness, and trustworthiness scores as the dependent variables. The analyses did not reflect a significant counselor effects for CRF-SF total, expertness, attractiveness, or trustworthiness. Therefore, we concluded that the four counselors were perceived as comparably expert, trustworthy, and attractive by the participants. Table 2 below displays relevant F statistics and p values for each CRF-SF dependent variable.
Counselor CRF-SF Scores Analysis of Variance (ANOVA) Results.
Note. CRF-SF = Counselor Rating Form–Short Form.
*Indicates statistical significance at the .05 level.
Treatment Check
To assess treatment integrity, we examined participants’ perceptions of their feedback sessions as measured by their responses to the treatment check questions. We conducted two ANOVAs with treatment as the independent variable and perceived feedback personalization and session involvement as the dependent variables. The ANOVA results are shown in Table 3. There was a treatment effect for session personalization; TA participants perceived a higher level of personalization than IG participants. The session involvement scores of the TA and IG groups did not significantly differ. However, the session involvement scores of TA participants were an average of .45 higher than those of IG participants, with a large effect size of .148. Together these results suggest that the two treatments were perceived as intended.
Treatment Integrity Analysis of Variance (ANOVA) Results.
Note. IG = information giving; SD = standard deviation; TA = therapeutic assessment.
*Indicates statistical significance at the .05 level.
Analysis of Hypotheses
Hypothesis 1 was that TA participants would demonstrate a significantly greater increase in CDSE than IG participants immediately following assessment feedback. We conducted a one-way ANCOVA using treatment group as the independent variable, CDSE-SF pretreatment scale score as the covariate, and CDSE-SF posttreatment score as the dependent variable. All η2 values were calculated following recommendations by Levine and Hullett (2002). There was not a significant treatment group effect, F(1, 21) = .778, p = .388. The results are presented in Table 4. The CDSE scores of TA and IG participants did not differ.
Dependent Measures Analysis of Covariance (ANCOVA) Results.
Note. CCA = Career Choice Anxiety scale; CDSE = Career Decision-Making Self-Efficacy scale; IG = information giving; SD = standard deviation; TA = therapeutic assessment; VIS = Vocational Identity scale.
*Indicates statistical significance at the .05 level.
The second hypothesis was that TA participants would report a significantly greater increase in vocational identity than IG participants following feedback. We conducted a one-way ANCOVA using treatment group as the independent variable, VIS pretreatment as the covariate, and VIS posttreatment as the dependent variable. The results are presented in Table 4. There was a significant treatment group effect, F(1, 20) = 5.32, p = .032. The VIS scores of TA participants were significantly lower than those of IG participants. Note that a decrease in VIS indicates a more positive vocational identity. This outcome provides support for Hypothesis 2.
The third hypothesis was that TA participants would have significantly lower CCA than IG participants following assessment feedback. We conducted a one-way ANCOVA using treatment group as the independent variable, CCA pretreatment as the covariate, and CCA posttreatment as the dependent variable. The results are presented in Table 4. There was not a significant treatment group effect for CCA posttreatment scores, F(1, 19) = 1.43, p = .247. The TA treatment did not differentially affect CCA.
The fourth hypothesis was that TA participants would rate their assessment feedback session as significantly more helpful than IG participants immediately following assessment feedback. We conducted an ANOVA using treatment group as the independent variable and perceived helpfulness as the dependent variable. The results are presented in Table 4. Treatment groups did not affect participant ratings of feedback helpfulness, F(1, 21) = .138, p = .714. It is noteworthy that the average helpfulness scores for both groups were just less than the “Greatly Helpful” scale descriptor, which suggests a possible ceiling effect.
The fifth and final hypothesis was that TA and IG participants together would demonstrate significantly higher CDSE and vocational identity and lower CCA following assessment feedback sessions. Three repeated measures ANOVAs were conducted using pretest and posttest scores as the dependent variable and time as the independent variable. Results are displayed in Table 5. The hypothesis was fully supported. Participants’ scores of CDSE, vocational identity, and CCA did significantly improve.
Dependent Measures Time Analysis of Variance (ANOVA) Results.
Note. CCA = Career Choice Anxiety scale; CDSE = Career Decision-Making Self-Efficacy scale; SD = standard deviation; VIS = Vocational Identity scale.
*Indicates statistical significance at the .05 level.
Discussion
The purpose of this study was to explore the comparative effectiveness of two approaches to career interest assessment feedback. We examined the effectiveness of TA, an alternative method of career assessment feedback, and IG, a traditional method, in increasing CDSE and vocational identity and reducing CCA. All participants were actual clients seeking career counseling at a university counseling center. The results indicated that TA was more effective than IG in increasing vocational identity. In addition, the assessment feedback process (i.e., TA and IG together) was effective in increasing CDSE and vocational identity and at reducing CCA. Overall, participants in both the TA and IG conditions rated their assessment feedback experience as “greatly helpful.”
Discussion of Hypotheses
The first hypothesis was that TA participants would demonstrate a significantly greater increase in CDSE than IG participants immediately following assessment feedback. The results did not support this hypothesis as CDSE scores significantly increased for both groups with a large effect size of .405. Because self-efficacy scores improved in both treatment groups, there was not sufficient evidence in support of the TA principle of self-efficacy/self-discovery. It is important to note that the self-efficacy increase for TA participants was greater than that of IG participants by .48 of a standard deviation (SD) for this measure. However, this was not a statistically significant difference.
There are three potential reasons for the absence of a significant treatment difference for decision-making self-efficacy. First, both treatments were effective; the posttreatment scores were near the ceiling of the CDSE measure. There may have been little room for treatments to significantly differ. Second, the limited number of participants in each treatment group did not yield statistical power (.134) to detect actual difference between the groups. Third, it is possible that both treatments were equally effective in increasing CDSE. Cutler (2004) proposed that if conclusions from psychotherapy outcome research are extended to career counseling outcome research, we should not expect treatment modality to be a critical factor influencing change. In this vein, the results regarding Hypothesis 1 reflect outcomes of minimal-to-no differences between active treatments common in psychotherapy outcome research (e.g., Luborsky et al., 1999).
Hypothesis 2 was that TA participants would report a significantly greater increase in vocational identity than IG participants following assessment feedback. The results supported this hypothesis. The vocational identity scores significantly improved for both treatment groups and to a significantly greater extent for those in the TA condition. A medium to large effect size of .124 was found for group difference, accounting for 12.4% of the variance between treatment groups. On average, TA participants’ vocational identity scores decreased by 2.55 (SD = 3.514) more than those of IG participants. A large effect size of .551 was found for treatment, accounting for 55% of the variance. This finding suggests that the self-verification principle of TA may enhance sense of vocational identity, even following a relatively brief treatment.
Hypothesis 3, that TA would yield a significantly greater decrease in CCA than IG, was not supported. A small to medium effect size of .049 was calculated for the group difference in CCA, accounting for 4.9 % of the variance in group posttest scores. On average, TA participants’ CCA scores decreased by 3.6 more (SD = 3.47 for posttest) than those of IG participants. A large effect size (ES) of .230 was found for the overall treatment effect, accounting for 23% of the variance. The failure to find a significant change in CCA does not substantiate the immediate effect of TA nor its principles.
The lack of a statistically significant finding may be due to three reasons. We first must consider the small sample size and corresponding low statistical power (.205). Second, a brief TA process may not adequately address CCA. In consideration of the smaller relative ES, it is possible that self-verification creates more immediate changes in identity and only affects anxiety in the longer term. A third possibility is that our TA protocol did not emphasize participant anxiety; counselors were not instructed to attend to client anxiety. A TA protocol that specifically invites the sharing of affect related to career choice dilemmas might lead to a greater reduction in anxiety.
The fourth hypothesis was that TA participants would rate their assessment feedback session as significantly more helpful than IG participants immediately following assessment feedback. Hypothesis 4 was not supported. Participants in both groups perceived treatments as helpful with average overall scores of 7.92 and 7.77, which fall just below the maximum value of 9 on the helpfulness rating scale. There are three potential reasons for the absence of a differential treatment effect for helpfulness. First, the use of a single item may not have produced a measure with sufficient spread or dimensionality to adequately measure helpfulness. Second, the small sample size again led to a minimal power value of .065. Third, both treatments may have been genuinely and equally helpful.
Hypothesis 5 was that TA and IG participants together would demonstrate significantly higher CDSE and vocational identity and lower CCA immediately following their assessment feedback session. There was full support for Hypothesis 5 with large treatment effect sizes of .405, .551, and .230 for CDSE, vocational identity, and CCA, respectively. These positive findings indicate that, as a whole, TA and IG assessment protocols were effective. In addition, the findings suggest that the tenets of TA (e.g., self-verification, self-enhancement, and self-efficacy/self-discovery) at least in part lead to the theoretically predicted changes.
Implications
These results both support and contrast with findings from previous career counseling and TA research. With respect to the career counseling literature, the current study provides further evidence for the overall efficacy of career counseling and specifically for the effectiveness of individualized career assessment (cf. Brown & Ryan Krane, 2000; Whiston et al., 1998, 2003). Although not a direct focus of the current study, the results provide further evidence for the effectiveness of career assessment feedback with the SII in increasing decision-making self-efficacy (cf. Luzzo & Day, 1999; Uffelman, Subich, Diegelman, Wagner, & Bardash, 2004). Our findings were also comparable to those of Cutler (2004), who found no significant differences between the outcomes of TA and IG assessment feedback. The current study also adds to career counseling intervention research in demonstrating the comparable effectiveness of TA as an alternative method of assessment feedback. These results represent a response to the call of Whiston et al. (2003) for more comparisons of different methods (e.g., individual, group, computer, stylistic differences) of career assessment feedback.
The results also were consistent with those of several studies in the TA literature. Specifically, the findings are similar to other studies documenting positive outcomes from TA feedback ranging from small to large effect sizes (e.g., Ackerman, Hilsenroth, Baity, & Blagys, 2000; Hanson et al., 1997; Hilsenroth, Peters, & Ackerman, 2004; Houser, 2005; Ingram, 2000; Newman & Greenway, 1997; Peters, 2000; Tharinger et al., 2009). Further, the results are directly similar to Hanson, Claiborn, and Kerr (1997), who also found no difference between delivered and interactive career counseling feedback on ratings of session helpfulness.
These results also have two important implications in light of the significant increase in Internet-based career assessments that has occurred over the last decade. First, Gati and Asulin-Peretz (2011) identified the need for integrating expert knowledge into Internet-based career assessment tools. Our results suggest that the TA principle of self-discovery can be integrated into these online systems. For example, the step of asking respondents to develop personally relevant exploration questions can be integrated into the design of on online career assessment tool. Asking questions can be designed to precede information generation. Elaboration of the insight sought from the tool and the reasons for the relevance of this additional information could enhance self-discovery in counselor-free online assessments and interventions. Second, the TA principles of self-verification and self-enhancement can be important for counselors in working with the increasing number of clients who initiate counseling after already having used online assessments (Gati & Asulin-Peretz, 2011). These clients may not have been able to integrate or synthesize the information generated from these Internet-based tools. In these circumstances, counselors can begin by focusing on assessment feedback that closely mirrors the self-views of clients (self-verification) and then proceed to consider potentially discrepant information. The growth of automated, open access career assessment and intervention systems produce a potentially overwhelming amount of information. The TA principles provide guidance for counselors who work with clients to make meaning of this information.
Finally, this study adds to the TA literature in three ways. It is the first study that has demonstrated the positive effects of TA assessment feedback on career counseling outcomes. Although Hanson et al. (1997) also found positive outcomes from an interactive career counseling assessment feedback intervention, they examined outcomes only related to session and counselor quality and did not examine career counseling outcomes. Second, the results add further support to the findings of Poston and Hanson (2010) that psychological assessment, particularly involving collaboration, is effective. Finally, in consideration of the identified significant difference and medium to large effect size between TA and IG participants’ vocational identity scores, the results of the study provide partial evidence in support of the TA principle of self-verification.
Taken together, these findings suggest that TA is effective in a variety of contexts. The results also suggest that TA is more effective than IG in increasing vocational identity. It remains unclear if TA leads to additional treatment outcomes such as decreased anxiety and resistance and increased goal achievement and motivation. It also is important to consider the limitations of the study.
Limitations
There were several limitations in the present study that can be attributed to the implementation of the study at a university counseling center. The first pertains to the sampling method. Because actual clients seeking career counseling were solicited for participation, only a limited number of potential participants were available. There was a 51% attrition rate, which occurred as a result of scheduling problems, inappropriate fit with study purpose, counselor illness, failure to complete the assessment or outcome measures, and failure to return for a second session. The small sample size reduced the experimental power of all of our analyses and our ability to detect treatment effects. As a consequence, the sample size placed a significant limitation on the interpretability of our results.
It also is important to consider the strengths of the selected sampling method. Using real clients strengthened the external validity of the study—at a cost to internal validity (Gelso & Fretz, 2001). However, we determined this enhancement to be of greater importance given the direct clinical applicability of the study. Heppner and Heppner (2003) noted a concern with the questionable generalizability created by the preponderance of recruited client samples and unnatural settings used in career counseling outcome research. External validity in the current study was improved through use of a sample of individuals who sought counseling on their own and indicated a genuine need for career counseling. The sampling method also helped achieve excellent population and ecological validity. As a result, the findings can be directly generalized to university counseling center client populations and settings.
A second potential limiting factor was that several counselors performed the two study interventions. We attempted to control for counselor differences by comparing perceptions of the counselors’ expertness, attractiveness, and trustworthiness; we also used a protocol checklist during counselor training to monitor treatment integrity. We standardized training in consideration of variability in the counselors’ experience and availability for scheduling and training. Despite all these efforts, the results still may have been affected by counselor differences and protocol adherence.
It is important to note that the counselors were also assessed for protocol preference or treatment allegiance following training and prior to the start of the study. All counselors except counselor Number 1 stated a preference for protocol A (i.e., the TA protocol). Counselor Number 1, who stated a preference for the IG protocol, also saw the majority of participants. It is difficult to determine if and how these preferences may have affected the results. We were not able to statistically evaluate counselor effect due to the small sample size. However the significant influencing effect of treatment allegiance has been well documented (Luborsky et al., 1999). Because counselor Number 1 saw the majority of participants, the results may have been positively skewed toward the IG treatment. This effect presents a significant limitation to the interpretation of the results particularly considering the significant positive outcomes identified from the TA treatment.
A third limiting factor was the lack of longer term follow-up measurement. Participants were allowed to continue with career counseling following study participation; there was no way to maintain comparability between the participants following study completion. Any follow-up measurement results could not have been attributed to differences in study treatments. In consequence, any longer term effects were not detected and the results are limited to the immediate effects of intervention. Whiston (2001) emphasized the importance of determining both the immediate and long-term effects of career interventions. Implementation of follow-up measures in future research will provide better evaluation of treatment effects over time.
In sum, there were three potential limitations pertaining to the design, sampling, and measurement selection of the current study. These factors may have affected the interpretability and generalizability of results and the ability to detect treatment effects, particularly the small sample size. However the study design, sampling, and measurement selection also provided several strengths to the significance of the study results. Although highly challenging, the choice to conduct the study as part of the regular functioning of a university counseling center with unsolicited career counseling clients afforded adequate internal and strong external validity. We believe intentional measurement selection (i.e., theoretically derived and population relevant) also provided a greater ability to detect treatment effects.
Future Directions
In consideration of the previous research, theoretical nature, methodology, and results of the current study, we make the following suggestions for the direction of future research in three primary areas. The first suggestion involves similar research aimed at exploring methods of career assessment interpretation. Multiple researchers in career counseling have identified subtypes of career undecided individuals that present with distinct sets of career decision-making concerns (e.g., Chartrand et al., 1994; Jones & Chenery, 1980; Kelly & Pulver, 2003; Lucas & Epperson, 1988). In order to fully recognize the individual presentation of career decision-making needs, researchers have encouraged the development of differential treatment approaches (e.g., Lent, 2001; Whiston & Oliver, 2005). Kivlighan, Hageseth, Tipton, and McGovern (1981) conducted the only study we identified in which the effectiveness of differential assessment feedback approaches by personality type was directly examined. They found that treatments that matched personality type (i.e., either task or people oriented) were more effective. We echo the sentiment that future researchers continue these steps and in particular examine the utility of TA and IG approaches with clients with distinct clusters of career concerns.
The second suggestion for future researchers is to implement three methodological enhancements related to the limitations of the current study. First researchers should consider implementing studies of middle and high school students, community college students, or adults in career transition. More specialized groups could also be targeted, including first-generation or international college students. Second, longer term outcomes can be measured in a more controlled setting, such as a departmental training clinic. Third, researchers can add process measures (e.g., specific measurement of aspects of the assessment feedback process, expanded session helpfulness ratings, session evaluations). Although the addition of process measures will increase the research time commitment and potentially reduce the overall number of participants, it also can provide greater understanding of why positive changes do or do not occur.
In closing, we encourage researchers to continue critical examination of the career assessment feedback process. Career assessment continues to be practiced by 56% of APA members who identify counseling as their specialty (Watkins, Campbell, & Nieberding, 1994). Holland’s (1997) person environment fit theory remains a dominant theoretical approach to career counseling and assessment. Career assessment is the central mechanism in practical application of Holland’s theory and, according to Hansen (2005), continues to be the construct most prevalently assessed in practice. Meta-analytic research reflects the effectiveness of career counseling assessment. Further, Brown and Ryan Krane (2000) specifically identified assessment as a critical ingredient of effective career counseling. In consideration of these factors, it is imperative that researchers continue to explore the process and impact of, and further develop improved methodological approaches to career counseling assessment.
Footnotes
Authors’ Note
This article is based on a doctoral dissertation conducted by the first author under the supervision of the second author.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by a small summer dissertation grant, Purdue Research Foundation Summer Dissertation Grant, Purdue University, West Lafayette, IN, June 2007 to July 2007.