Validation of an Online Orientation Seminar to Improve Career and Major Preparedness

Abstract

Career planning and development for undergraduates is a goal for our field, yet the majority of psychology departments do not offer a course to meet this goal. This article examines an online psychology orientation course that requires relatively few instructional resources to achieve success. Outcomes for the course are measured by student work along four dimensions: knowledge of the major, planning for the future, career knowledge, and self-assessment of fit. The analysis shows very large gains in student performance for major knowledge and planning for the future and moderate gains in the other areas. This work suggests that the online model is one potentially powerful tool for preparing undergraduate psychology majors to succeed in the major and after graduation.

Keywords

undergraduate curriculum careers

Since Gottlieb’s call 35 years ago to improve undergraduate preparation for the profession (Gottlieb, 1975), psychology departments have developed and used a variety of methods to prepare undergraduate students to succeed in the major and professionally. As Lloyd (2004) pointed out, these methods include “(a) faculty-student contacts, (b) peer advising, (c) psychology clubs, (d) majors’ meetings and/or graduate school/career days, (e) majors’ handbooks, (f) display racks of advising materials, (g) departmental Web pages, (h) departmental library, and (i) departmental bulletin boards” (p. 1). Improvements in the usability of online resources from a student perspective and the availability of sophisticated online course development and management tools for faculty have made models of online education attractive across a variety of domains, including preparation of students for success in the major and beyond. Recent work (Brinthaupt, 2010) suggests students positively receive online career seminars and feel they learn a significant amount in such an environment. However, research has not studied if student self-reports of learning are supported by other measures of learning outcomes. The current work presents evidence that an online course, delivered to numerous students with minimal faculty interaction, can prepare students to succeed in the major and in planning for future success.

Problem and Solutions

Most faculty members can recall instances in which they have been approached by a bright student who is far along in his or her college experience—but seems uninformed about what they need to do to succeed after college. What makes this particularly problematic from the psychology major perspective is that the American Psychological Association (2010) found that 25% of psychology undergraduates will go on to complete graduate degrees in psychology. Other sources suggest that when factoring in other professional programs, such as law school or business school, the number of psychology undergraduate students seeking graduate training may be closer to 40% (Schwartz, 2000). Students have a short time in a 4-year program to develop the background, skills, and record necessary to successfully apply for postgraduate training or to make informed decisions about how to select a career with a bachelor’s degree and prepare themselves for a competitive job market once they graduate.

The American Psychological Association (2007) has recognized the pervasiveness of this issue, and in “Guidelines for the Undergraduate Psychology Major” it called for career planning and development as one of the ten learning goals for undergraduate programs. There is a rich body of work dating back to Gottlieb’s (1975) proposal on what adding career courses and experiences to the psychology major does for student success. Numerous publications have presented both student self-report data for learning and course satisfaction (Brinthaupt, 2010; Dillinger & Landrum, 2002; Dodson, Chastain, & Landrum, 1996; Landrum & Mulcock, 2007; Lattal, 1980; Macera & Cohen, 2006; Zechmeister & Helkowski, 2001) and more objective measures of learning including knowledge surveys (Maynard, Maynard, & Rowe, 2004; Thomas & McDaniel, 2004) or changes standardized career maturity questionnaires (Ware, 1981) that strongly suggests these efforts improve student readiness.

Yet despite the level of awareness of the problem and the long history of efforts to solve it, the last systematic count suggests that a little over one third of psychology programs offer a course designed to directly meet the goal of improving career planning and student development (Landrum, Shoemaker, & Davis, 2003). There are undoubtedly a number of factors that contribute to this low rate. In departments with faculty appointments split between research, teaching, and service, faculty at research-intensive institutions may perceive a disincentive toward developing new courses that concentrate on undergraduate careers. Faculty will also informally report that “the cream rises to the top,” meaning that the “good” students will succeed in any environment bringing into question the necessity for such efforts. The problem of undergraduate success may also be seen as the “elephant in the room”: Everyone is aware of the scope of the problem but many perceive it as too big to tackle. For example, if a faculty member were to consider building such a course and were to read the latest work describing an excellent example of similar efforts (Appleby, 2010), he or she might be daunted by the report of that faculty member that their course has two sections of 35 students with eight or nine teaching assistants per section. In a department with a large number of majors, acquiring the resources needed for such an effort may seem insurmountable.

Developments in online learning may provide one model to overcome some of these issues. Current students have been raised in an era of pervasive Internet access and are used to information delivery via the Internet. Students can experience high levels of satisfaction with online courses, leading to increased motivation and better learning outcomes (Eom, Wen, & Ashill, 2006). A recent meta-analysis of online education suggests that the online model can be as or even more effective than traditional face-to-face instruction (U.S. Department of Education, 2009). The rapid increase in the number of user-friendly course development tools, online learning systems adopted at the university level, and user familiarity with computer resources has also made development of these courses more attractive for faculty members. Because online courses can be delivered with a range of faculty–student interactivity, an online course for career development may present one way to overcome some of the barriers for offering such a course that were previously discussed.

Using online resources in the context of improving undergraduate psychology career planning and development is an area with great promise. Early work (Johnson & Morgan, 2005) reported that a brief (three 45-minute sessions), interactive Web course helped to raise student self-reports of advising satisfaction. However, the Web course was delivered as part of a larger effort to improve undergraduate advising, so it is unclear what effect the Web course had by itself. More recent work (Brinthaupt, 2010) examined an in-depth Web course used to serve a large number of majors (more than 900), which has been awarded a “exemplary course” designation by the hosting university. Data collected from students completing the course strongly suggested that students were satisfied with the course and that they felt they gained valuable knowledge by completing the course.

One aspect that could be added to the limited amount educators know about using online models to foster career and student development for psychology undergraduates is how these courses directly affect student learning. As we have reviewed, relatively few studies offer direct empirical data suggesting students improve as a result of career development courses. Much of the data on the effectiveness of these courses come in the form of student self-reports of learning and/or satisfaction. However, there are ways that the products of a course can be used to directly measure learning outcomes, such as applying rubrics to existing student work to assess proficiency level of students for desires skills (Halonen et al., 2003). In this article, we provide an example of this as applied to an online course designed to foster career planning and student development. To preview, the results of a rubric-based content analysis of precourse and postcourse student work suggested that an online course that is rich in content but that requires little faculty interactivity with individual students can produce measurable, positive outcomes for student success.

The Course

The course (PSYC 102: Orientation Seminar in Psychology) is a 1-credit-hour, credit/no-credit course. It is required before applying to become a psychology major. The course is delivered to approximately 500 students per year. It is self-paced and takes about 15 to 20 hours to complete. It is divided into four sections: Introduction, Know the Basics, Opportunities, and Planning. All content is offered exclusively via online delivery. The current version of the course is available to students through the university’s Web-based course management system.

The Introduction covers course structure, expectations and goals, and course procedures, and it includes a mandatory academic misconduct and plagiarism lesson and quiz that they must complete before moving forward in the course.

Know the Basics explains professional mentorship, major policies and procedures, student self-assessment of career fitness and personal values (e.g., using career assessment tests provided by the university’s Career Center), and an orientation to career options in psychology.

Opportunities helps students think about and find scholarships for undergraduate and graduate work, provides a lesson on happiness and data on the relationship between money and happiness (prior to exposing them to salary information), describes career paths after college for both bachelor-level and graduate-level degrees, covers opportunities in the department beyond the required course load such as research internships and honors projects, raises awareness of external opportunities such as service learning and volunteering, and covers successfully preparing for graduate school admissions.

Planning consists of two capstone exercises: preparing a course plan for the remainder of the students’ college years, including departmental and external opportunities, and making a career plan that brings together work that they have completed in other sections.

In each section, students are encouraged to engage with the material in two ways. First, each subsection has an associated quiz that covers the basic information in that section. Students must achieve 90% on each quiz to pass the course. Second, students complete six tasks to encourage them to work through the most important concepts as well as two capstone exercises to help them bring all of their work together to promote learning how to plan for the future. Students build portfolios of their work at the end of the course. The instructor examines student work across the quizzes and tasks, but the level of individual student feedback is minimal.

Method

Assessment of Course Effectiveness

The course has two mechanisms to gauge effectiveness. The first is an online, multiple-choice quiz delivered both precourse and postcourse that asks 15 questions about the major and career options, selected randomly from a 35-item pool. The second is a set of essays that students complete before and after the course. In the first set of two essays, students are asked to write 250 to 400 words answering the questions, “Why do I want to be a psychology major?” and “Describe all of the things you think people with psychology degrees do for a living.” In the exit essays, they answer the prompts, “Please write a brief essay on what you think psychologists do for a living.” and “Please write a brief essay on why you still want to be a psychology major. If your mind has changed, explain why. In your essay, please elaborate on your understanding of any aspects of psychology that have changed while taking PSYC 102.” These essays are ungraded and only the first attempt is recorded.

To develop a scoring rubric, we collected a random set of essays from 30 students. We examined these essays for content that could be used to develop the coding rubric to classify student success. Initial analyses led to the development of four categories the team felt could be reliably extracted from the essays: knowledge of the major, knowledge about future goals and plans, knowledge of career diversity, and self-assessment of fit for the major and careers. We developed an initial coding rubric that classified essays along these four dimensions on a 3-point scale from novice to advanced performance. The rubrics had descriptions of the categories, definitions of performance in each level of a category, and examples of performance at that level. This coding rubric was applied to the initial set of essays. Based on ratings by three research assistants and checks of interrater reliability, we further refined the rubrics in terms of definitions and examples. In addition, the scale was made to range from 0 to 6 points, so that within a level of performance the rater could capture minimum and maximum performance for that level. The final version of the rubric is represented in Table 1. (The examples included in the actual rubric are available upon request.) To assess course effectiveness, sets of 81 precourse and postcourse essays were randomly selected from an additional set of students and were rated by a pair of raters trained to use the scoring rubric.

Table 1.

Scoring Rubric for the Essays

	Novice (0–2)	Intermediate (3–4)	Advanced (5–6)
Knowledge of the Major: psychology major requirements or the importance of a course plan	Vague mention of psychology courses and shows little knowledge about the admissions and major requirements	Shows knowledge of admissions requirements or acknowledges what they learned or what courses they have left	Mentions specific admissions requirements, specific electives; or core courses; or their course plan and goals; or mentions the desire to take an independent study, graduate-level classes, or do an honors thesis
Future Goals and Plans: graduate study and/or admissions, career plans and goals and ways to achieve them, and the importance of research or volunteering	Mentions the possibility of continuing education but does not mention specifics about graduate admissions Mentions a possible career path but does not mention what needs to be done to achieve that career (other than possibly continuing school) Discusses desire or importance of wanting to help people or mentions possibility of researching	Discusses a career path option plus academic experience needed but does not mention specific plans to achieve that goal or specific degree requirements Expresses interest in doing research or volunteering but does not list a project or a particular area of research interest Discusses a specific postgraduate option but does not mention important factors in admittance	Mentions a career, the qualifications, and illustrates a plan to achieve them Lists specific volunteer sites or projects that he/she would like to be involved in or expresses interest in doing research. Discusses graduate school in specific area of study and lists at least two admissions or degree requirements or suggestions for those applying
Career Diversity: knowledge of career diversity in psychology	Merely says there are a variety of careers in psychology or lists only clinical positions; mentions only clinical, sports, or profiling (settings and fields are not careers)	Mentions one career path that is not directly related to clinical, sports, or profiling but never mentions degree requirements; listing more than one without the degree requirement = 4	In-depth knowledge of career options for clinical psychologist and one other career path or Lists more than 2 possible career paths for psychologists and the degree requirements for at least one career
Fit: a student’s knowledge of how psychology as a major or career fits him/her personally	Wants to be a psychology major because (a) he/she wants to help people, (b) he/she had a life experience where therapy was helpful, (c) he/she is a good listener, or (d) took a psychology class in high school or college; one or no career is listed	Discusses another reason (other than a, b, c, or d) for choosing psychology or lists a career associated with a, b, c, or d; at least one career listed fits with their interests in psychology	Mentions some element of the interest inventory that makes the major or a given career a good fit for them or Shows advanced knowledge in a given area, which shows great interest and, therefore, fit

Results

About two thirds of students were at the first-year or sophomore level and the remainder were advanced students. Precourse and postcourse major and career quiz data were selected from 95 students who had completed the course at the point in the semester that the rubric analysis was completed. Students achieved 61.2% correct on the pretest and 71.0% correct on the posttest. This was a significant and large effect, t(93) = 21.51, p < .05, Cohen’s d = 0.77.

For the 81 sets of essays, the average rating of an essay by the raters for each of the four dimensions was used, yielding 81 data points for each dimension both precourse and postcourse, respectively. Analysis of the essay data from the rubric yielded a similar pattern of effects across the four categories (see Figure 1). The biggest improvement was found in the knowledge of the major; students improved from low novice (M = 0.58, SD = 0.74) to near advanced (M = 4.60, SD = 1.33), t(80) = 23.72, p < .05, d = 3.10. Improvement on knowledge of future goals and plans also showed large and significant gains. Students improved from high novice (M = 1.92, SD = 1.08) to near advanced (M = 4.73, SD = 1.10), t(80) = 20.37, p < .05, d = 2.58. Gains in terms of career diversity knowledge and fit for the major were also significant but more moderate. Students improved within their intermediate level of knowledge for career diversity (pretest M = 3.7, SD = 1.02 to posttest M = 4.04, SD = 0.89), t(80) = 4.13, p < .05, d = 0.36, and fit (pretest M = 3.09, SD = 0.86 to posttest M = 3.54, SD = 0.86), t(80) = 5.14, p < .05, d = 0.52.

Figure 1.

Student performance data in precourse and postcourse essays assessed with a scoring rubric across four factors

Discussion

The popularity of the psychology major and the large proportion of majors who seek postgraduate opportunities reinforce the need for courses that can orient students to succeed in the major and develop professionally. Although many departments of psychology deliberately pursue this goal through curriculum design, a majority do not. There are a variety of perceived obstacles and reasons for this. One reason may be unwillingness to devote what may be seen as a prohibitive level of resources to tackle the problem well. However, this work suggests that this is not necessarily the case. The online orientation course examined here did take significant time to develop and does require the regular maintenance and updating that any other course requires, but the level of daily instructor involvement with the student population is minimal given the size of the student population the course serves. Even so, the current data clearly show that the course is effective, not just in terms of student reports of satisfaction as other work as shown (Brinthaupt, 2010) but in terms of improvements in student work as assessed by better test performance and, more importantly, quantitative analysis of student work.

Student work revealed that, following the course, students were more aware of major requirements, goals, and courses that are optional but very important, such as honors opportunities and working in laboratories conducting research with faculty members. There were also large gains in student planning skills. Students were able to name qualification criteria for postgraduate options and careers as well as opportunities of which they need to take advantage to foster their own success, such as volunteer activities and scholarships. These effects were so large for these categories that almost every single student sampled showed improvement. There was a significant increase in career awareness and self-assessment of fit for the major, though these gains were smaller, due in part to moderately high performance of students on these prior to the course and possibly due to course design in terms of career emphasis and the fact that the students may already have an accurate assessment of their own fit for the major. It is important to note that the samples came from about a two-year period of time and a wide range of years in school. The range of time suggests there was a history confound in these data. Because one third of the responses were from juniors and seniors, the effects may actually be underestimates of how well such a course would work with first-semester students.

Lessons Learned

The course as presented here involved a process of years of planning and implementation. There have been many changes as a result of lessons learned. A few of these are worth sharing to conclude this work. The first lesson is that we, as educators, should not expect new college students to know what they do not know. Faculty status is typically the result of a lifetime of good decisions, but many of us will admit that there was a lot of luck involved. Perhaps we took the right class or an advisor saw potential and pointed us in the right direction. It is unreasonable to expect that method to work for hundreds of students in our departments each year. To that end, courses like the orientation seminar must be both (a) required and (b) timely. Our department has an admissions process for majors. Premajors have to take a set of classes, including the orientation seminar, before they can apply to be a major and take the favored, advanced classes. These requirements help make sure that a young adult does not put off what might be a career-changing event. Another lesson is that the course should probably be structured in the same way other courses are in terms of time to completion and expectations. An early version allowed students to complete the course at their own pace, but the course quickly filled with sophomores that became seniors who put it off until the last semester they could. Students now have one semester to complete it like any other course, and this has helped students complete it in a timely manner.

Footnotes

Acknowledgment

We thank Aziz Sultani, Mary Pitman, and Christopher Cushing for their tireless efforts and wonderful enthusiasm in the development of the course. We also thank Barbara Watkins and Lynne Lipsey at Continuing Education for making the course a reality.

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The authors disclosed receipt of the following financial support for the research, authorship and/or publication of this article: Initial work on the course was supported by a grant from the Center for Teaching Excellence, and a special thanks goes out to its director, Dan Bernstein for his constant encouragement and support .

References

American Psychological Association. (2007). APA guidelines for the undergraduate psychology major. Retrieved March 1, 2010, from www.apa.org/ed/resources.html

American Psychological Association. (2010). Support center: What percentage of undergraduate psychology majors continue to earn graduate degrees in psychology? Retrieved March 1, 2010, from http://www.apa.org/support/education/statistics/continuing.aspx#answer

Appleby

D. C.

(2010). Advising in the classroom: A career exploration class for psychology major. In Dunn

(Ed.), Best practices for teaching beginnings and endings in the psychology major: Research, cases, and recommendations (pp. 49–59). New York, NY: Oxford University Press.

Brinthaupt

T. M.

(2010). Development and implementation of an online careers seminar in psychology. Teaching of Psychology, 37, 58–62.

Dillinger

R. J.

Landrum

R. E.

(2002). An information course for the beginning psychology major. Teaching of Psychology, 29, 230–232.

Dodson

J. P.

Chastain

Landrum

R. E.

(1996). Psychology seminar: Careers and graduate study in psychology. Teaching of Psychology, 23, 238–240.

Eom

S. B.

Wen

H. J.

Ashill

(2006). The determinants of students' perceived learning outcomes and satisfaction in university online education: An empirical investigation. Decision Sciences Journal of Innovative Education, 4, 215–235.

Gottlieb

M. C.

(1975). Introduction to fields of psychology: A course proposal. Teaching of Psychology, 2, 159–161.

Halonen

Bosack

Clay

McCarthy

Dunn

Hill

G., IV

…Whitlock

K. A

. (2003). A rubric for learning, teaching, and assessing scientific inquiry in psychology. Teaching of Psychology, 30, 196–208.

10.

Johnson

E. J.

Morgan

B. L.

(2005). Advice on advising: Improving a comprehensive university’s program. Teaching of Psychology, 32, 15–18.

11.

Landrum

R. E.

Mulcock

S. D.

(2007). Use of pre- and postcourse surveys to predict student outcomes. Teaching of Psychology, 34, 163–166.

12.

Landrum

R. E.

Shoemaker

C. S.

Davis

S. F.

(2003). Important topics in an introduction to the psychology major course. Teaching of Psychology, 30, 48–51.

13.

Lattal

K. A.

(1980). Psychology as a profession—A brief course providing career information for psychology majors [Note]. Teaching of Psychology, 7, 243–244.

14.

Llyod

M. A

. (2004). The case for requiring a “careers” course for psychology majors. In Buskist

Hevern

V. W.

Saville

B. K.

Zinn

(Eds.), Essays from E-xcellence in teaching (Ch. 8). Retrieved January 30, 2012, from http://teachpsych.org/ebooks/eit2003/index.html

15.

Macera

M. H.

Cohen

S. H.

(2006). Psychology as a profession: An effective career exploration and orientation course for undergraduate psychology majors. Career Development Quarterly, 54, 367–371.

16.

Maynard

A. M.

Maynard

D. C.

Rowe

K. A.

(2004). Exposure to the fields of psychology: Evaluation of an introductory psychology project. Teaching of Psychology, 31, 37–40.

17.

Schwartz

S. K

. (2000). Working your degree. Retrieved March 1, 2010, from http://money.cnn.com/2000/12/08/career/q_degreepsychology/

18.

Thomas

J. H.

McDaniel

C. R.

(2004). Effectiveness of a required course in career planning for psychology majors. Teaching of Psychology, 31, 22–27.

19.

U.S. Department of Education, Office of Planning, Evaluation, and Policy Development. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, DC: Author

20.

Ware

M. E.

(1981). Evaluating a career-development course: A 2-year study. Teaching of Psychology, 8, 67–71.

21.

Zechmeister

E. B.

Helkowski

(2001). A career course and fair. Teaching of Psychology, 28, 276–278.