Testing the Efficacy of MyPsychLab to Replace Traditional Instruction in a Hybrid Course

Abstract

Online course-packs are marketed as improving grades in introductory-level coursework, yet it is unknown whether these course-packs can effectively replace, as opposed to supplement, in-class instruction. This study compared learning outcomes for Introductory Psychology students in hybrid and traditional sections, with hybrid sections replacing 30% of in-class time with online homework using the MyPsychLab course-pack and Blackboard course management system. Data collected over two semesters (N = 730 students in six hybrid and nine traditional sections of ∼50 students) indicated equivalent final-grade averages and rates of class attrition. Although exam averages did not differ by class format, exam grades in hybrid sections decreased to a significantly greater extent over the course of the semester than in traditional sections. MyPsychLab homework grades in hybrid sections correlated with exam grades, but were relatively low (66.4%) due to incomplete work—suggesting that hybrid students may have engaged with course materials less than traditional students. Faculty who taught in both formats noted positive features of hybrid teaching, but preferred traditional classes, citing challenges in time management and student usage of instructional technology. Although hybrid students often reported difficulties or displeasure in working online about half of them indicated interest in taking other hybrid classes.

Keywords

Hybrid instruction pedagogy higher education teaching/learning strategies

Introduction

Forms of online education have expanded rapidly as digital technologies have become ubiquitous. As early as the 2000–2001 academic year, 90% of 2-year and 89% of 4-year public institutions offered online education (Tallent-Runnels et al., 2006), although what constitutes online education often varies across institutions and educational contexts as elements of online and traditional instruction may be combined in a variety of ways (Moore, Dickson-Deane, & Galven, 2011). The reasons for the increase in online education are both administrative (e.g., more available seats if physical classrooms are needed for less time), and pedagogical (e.g., students need competence in online technologies/communication for their future careers) (Poirier, 2010; Thor, 2010; Young, 2002). Internet-based course management systems (CMS), such as Blackboard, Moodle, or Canvas, and publisher-provided course-packs, such as MyPsychLab, WileyPLUS, or LaunchPad, have made it feasible to teach courses fully or partially online by providing virtual classroom spaces and ready-made online modules to accompany textbook chapters. Incorporating these digital tools into course design provides students with opportunities to develop technological skills (Gerard, Gerard, & Casile, 2009; Powers, Brooks, McCloskey, Sekerina, & Cohen, 2013). Whether this translates into more effective teaching and learning remains controversial, especially for introductory-level survey courses where students may lack adequate preparation to utilize the tools effectively (El Mansour & Mumpinga, 2007).

Use of Publisher-provided Course-packs to Support Learning

Surveys of undergraduates in Introductory Psychology courses suggest that many students do not know how to study effectively (Gurung, 2005; Gurung, Weidert, & Jeske, 2010). Moreover, students' preferred study methods (i.e., re-reading notes and highlighting textbooks) are largely ineffective in fostering long-term retention of course material (Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013). Publisher-provided course-packs aim to improve student-learning outcomes through structured activities that supplement textbook readings, as hands-on interactive activities have been shown to aid online learning (Koedinger, Kim, Jia, McLaughlin, & Bier, 2015). These course-packs, consisting of multi-media resources (e.g., videos, case studies, experimental simulations, and quizzes) accompanied by an e-book, are aggressively marketed as proven to improve learning outcomes. For example, MyPsychLab (2016) claims that it delivers “consistent, measurable gains in student learning outcomes, retention, and subsequent course success.” Similar claims are made by publishers of other online course-packs, for example, WileyPLUS (2016) and LearningCurve (2016). Despite such publisher claims, no randomized controlled studies exist that systematically compare learning outcomes as a function of student access to one of these course-packs.

Nevertheless, publisher-provided course-packs appear to align with what researchers have identified as effective study habits involving self-testing, distributed practice, and interleaved practice (Brown, Roediger, & McDaniel, 2014). Self-testing promotes the retrieval and long-term retention of information to a greater extent than re-reading notes and highlighting textbooks—the latter methods increasing fluency in processing, but not retrieval, thus leading students to overestimate what they know (Karpicke, Butler, & Roediger, 2009). Indeed, in a study correlating study behaviors and exam scores, various methods of self-testing lagged just behind attending class as the most effective study behavior (Gurung et al., 2010). MyPsychLab heavily utilizes self-testing through quizzes; students may review questions they answered incorrectly by going back to the text then retaking the quiz.

Distributed practice, or spaced exposure to key concepts, has been shown to be more effective than massed practice for long-term retention of material (Cepeda, Pashler, Vul, Wixted, & Rohrer, 2006; Seabrook, Brown, & Solity, 2005). MyPsychLab encourages distributed practice through multiple, short assignments that can be completed over the course period. The MyPsychLab calendar function makes it easy for students to keep track of deadlines while working through assignments at their own pace. Interleaved practice, which involves working on multiple problem types within a single study session, has been shown to enhance retention (Rau, Aleven, & Rummel, 2013; Taylor & Rohrer, 2009). MyPsychLab encourages interleaved practice by providing varied learning activities for each chapter. As students complete homework assignments, they are exposed to information in multiple formats (e.g., interviews, case studies, and experiments) that may lead them to reflect on course topics and concepts in different ways.

Given these features, course-packs like MyPsychLab may help students to study smarter and thus outperform students in traditional classes. However, for such benefits to accrue requires students to engage independently with the course-pack, and be self-motivated to persist when confronted with technical issues or potentially challenging content. Despite the ubiquity of technology in their lives, undergraduates are often unfamiliar with course-packs and may require considerable technical support to succeed (Gerard et al., 2009; Wach, Broughton, & Powers, 2011). This results in additional work for instructors, who must include the task of teaching students to use course-packs in their course objectives. While publishers do provide technical support, students may rely on instructors rather than call-in centers to answer technical questions. For instructors, mastering course-pack features in order to troubleshoot student technical difficulties is time consuming and potentially frustrating (Cowie & Nichols, 2010).

More generally, successful engagement with an online course-pack may crucially depend on self-regulatory skills. Students need to motivate themselves to complete homework by setting goals, managing time, delaying gratification, inhibiting distraction, and self-reflecting on their performance (Bembenutty, 2009). Maintaining one's focus in the face of distraction may be especially challenging in digital environments, where studying is pitted against online forms of entertainment, such as social media, video games, and web surfing (e.g., Jacobsen & Forste, 2011; Levine, Waite, & Bowman, 2007). Students who believe in their capabilities, value homework as a task that enhances learning, and have high self-efficacy are more likely to be persistent in the face of distraction (Ramdass & Zimmerman, 2011). However, many college students lack prior background and interest in academic course material, and have obligations that compete for their attention. For these students, remaining motivated to complete weekly online homework might prove to be an arduous task.

Is Hybrid Instruction Effective?

With the increased adoption of online forms of instruction, the U.S. Department of Education, Office of Planning, Evaluation, and Policy Development (2010) conducted two meta-analyses to evaluate learning outcomes as a function of course format. The first compared fully online courses with traditional courses, and found negligible differences in learning outcomes (N = 27 studies; average effect size g = +0.05). The second compared web-enhanced courses (combining online and face-to-face instruction) with traditional courses and documented slightly better outcomes for web-enhanced courses (N = 23 studies; average effect size g = +0.35). However, as noted in the report, web-enhanced courses typically included additional learning time and instructional units; hence, the observed benefits were likely to stem from added instruction, as opposed to the advantages of online media. The increased workload associated with web-enhanced instruction is referred to as the “course and a half syndrome” (Gerard et al., 2009).

Few studies have directly compared learning outcomes for students in hybrid and traditional courses with equivalent numbers of instructional hours: Cottle and Glover (2011) compared learning outcomes for two hybrid sections of Human Development (each ∼50 students, total N = 99) with one larger traditional section (N = 110), and reported that hybrid students received higher final grades (M = 89.7%) than traditional students (M = 82.4%). Others have reported comparable grades across hybrid and traditional formats: for Computer Networks and Communications (Delialioglu & Yildirim, 2007); and for Introductory Statistics (Utts, Sommer, Acredolo, Maher, & Mathews, 2003). In the latter study, hybrid students (N = 77) were more likely than their peers in a large traditional section (N = 208) to report that the course improved their computer/Internet skills; however, they also reported that the hybrid course was much more work than a traditional course, with less positive evaluations overall (Utts et al., 2003). Note that all of these studies compared classes of unequal size—with small hybrid sections compared to larger traditional sections. Given evidence that learning outcomes are better for students in smaller classes (Chapman & Ludlow, 2010) it is of necessity to compare outcomes for hybrid and traditional sections of comparable size.

In comparing hybrid and traditional courses, no previous study has examined the effectiveness of publisher-provided course-packs, such as MyPsychLab, as a replacement for in-class instruction. However, despite the lack of any randomized controlled study, two correlational studies (within web-enhanced classrooms) claim benefits of MyPsychLab usage (Cramer, Ross, Orr, & Marcoccia, 2012; McKenzie et al., 2013). Each study documented improved exam performance with increased online engagement, and attributed benefits to study aids that allowed students to self-test for mastery of key concepts. However, an alternative interpretation is that better-prepared students were more likely to use MyPsychLab. Indeed, McKenzie and colleagues (2013) reported that only 39–53% of students logged into MyPsychLab for each of five units, which suggested a lack of motivation to use the course-pack.

The Current Study

Our study used a quasi-experimental design to compare learning outcomes in public university students taking Introductory Psychology (PSY100) in a hybrid or traditional format. Although students self-selected into course formats, students in hybrid and traditional sections had similar demographic characteristics and all sections were of comparable size: ∼50 students/section. Data were collected at the College of Staten Island (CSI), a four-year college offering associate and baccalaureate degrees to a diverse student body residing in the New York City metropolitan area and commuting to college.

To compare learning outcomes across hybrid and traditional formats, we used departmental exams for all sections, and examined the relationship between online homework and exam grades in hybrid sections. We surveyed instructors and students about the hybrid course, and their preferences in course formats for teaching and learning. Three hypotheses guided this research.

Despite the potential of MyPsychLab to enhance learning through individualized study and self-testing, we expected students to lack familiarity with instructional technology and motivation to work independently. Thus, we predicted that there would be no overall benefit of hybrid instruction relative to traditional instruction.

We expected students to report challenges using instructional technology and a preference for traditional instruction.

Given the required time for instructors to develop skills in using MyPsychLab and in supporting student usage, we expected instructors to show a preference for traditional instruction.

Method

We compared traditional PSY100 sections that met twice a week for a total of 165 minutes with hybrid sections that met once per week for 115 minutes, with the 30% reduction in time replaced with online modules. We used the textbook Psychology: From Inquiry to Understanding (Lilienfeld, Lynn, Namy, & Woolf, 2011), accompanied by MyPsychLab (www.mypsychlab.com). Hybrid sections and traditional sections had identical enrollment caps (50 students/section), and used the same in-class videos, demonstrations, activities, and exams. Due to time constraints, in-class lectures and discussions in hybrid sections were drastically reduced relative to traditional sections. Hybrid sections were introduced as part of a university-wide initiative to provide web-enhanced instruction, http://hybrid.commons.gc.cuny.edu/. Hybrid sections were listed in the schedule of classes as requiring online work, with students self-selecting into traditional and hybrid sections based on seat availability and convenience. The scheduling of hybrid and traditional sections was arranged by the Psychology Department based on classroom availability.

Participants

Data were collected from 15 sections of PSY100 from two semesters, with eight instructors teaching six hybrid sections and nine traditional sections. Instructors were adjunct faculty with Master's degrees, with comparable levels of teaching experience. All had taught Introductory Psychology for fewer than four years; first-time instructors were assigned to teach hybrid as well as traditional sections. Assignment of instructors to sections was made by the chair of the Psychology Department based on instructor availability. Three instructors taught one or more sections in both formats. All sections enrolled the same population of general education students; see Table 1. Approximately two-thirds of students were women; students were, on average, in their 2nd semester of college, with a range of intended majors (10–14% Psychology).

Table 1

Description of Sample

Format	Number of Students	Number of sections	Class size M (range)	Number of instructors	Average number semesters	Psych Major
Hybrid	291 (63% female)	6	48.5 (46–54)	4	2.3	10.3%
Traditional	439 (66% female)	9	48.8 (46–52)	6	2.2	13.7%

The sample at enrollment comprised 730 students (291 hybrid, and 439 traditional) in 15 sections ranging from 46 to 54 students. While students self-selected into course sections, surveys collected on the first day of class showed that 67% of hybrid students enrolled without awareness of the hybrid status of the course, even though course registration indicated online work. An additional 6% said they knew that they were enrolled in a hybrid course, but did not know what that was. This suggests that ∼73% of hybrid students were unaware of taking a course with online requirements although they did know that their class met only one time per week for approximately 2 hours. The reduced class time potentially attracted students with greater constraints on their time.

Exam grades, exam count, final grades, and attrition rates were gathered with departmental permission from final grade-books, with data de-identified to ensure student anonymity. The study was completed with institutional review board approval, classified as exempt.

Data Collection

Data were collected from hybrid sections and traditional sections taught in parallel and covering the same content. Each week, traditional sections had approximately one hour of class activities in the form of demonstrations, videos, and simulations and two hours of lecture and discussion, whereas hybrid sections had approximately one hour of class activities (identical to the traditional sections), one hour of lecture and discussion, and one hour of online homework. However, in addition to the 30% reduction in class time, one class period in each hybrid section was devoted to training students on MyPsychLab and Blackboard, with additional in-class time devoted weekly to answer questions about online work. Hybrid sections and traditional sections also differed in the format of the textbook: Hybrid students were required to purchase access to MyPsychLab, and were encouraged to use the e-book to minimize costs. Traditional students were instructed to purchase textbooks without MyPsychLab access. All students had access to Blackboard. Traditional sections used Blackboard only for posting grades.

Exam grades and exam count

Instructors reported raw exam scores for each of three departmental exams. These exams were non-cumulative, 25-question, multiple-choice tests, covering 3–4 chapters and administered to hybrid sections and traditional sections. Exam questions were selected from the Pearson test-bank by PSY100 coordinators and modified as needed to improve wording. Exam questions focused on key terms, experiments, and theories. We computed exam count as a dependent variable to provide an index of attendance on exam dates.

Final grades

Given the online requirements for hybrid students, final grades were computed differently as a function of course format; see Table 2. Final course grades were collected as percentages and letter grades. Students who received letter grades of A, B, C, D, or F were compared across formats using final percentage grades. Students who received grades of W or WU (official or unofficial withdrawal, due to non-attendance) were removed from the analysis of final grades. However, we separately analyzed percentages of students who withdrew (W or WU) to provide an index of attrition.

Table 2

Final Grade Calculation for Hybrid and Traditional Sections

Format	In-class participation	Class presentation	Exams	MyPsychLab	Blackboard
Hybrid	15%	15%	30%	25%	15%
Traditional	20%	20%	60%	NA	NA

MyPsychLab homework

All hybrid sections were assigned the same online homework, which included 5-question, quick-review quizzes and 1–2 question quizzes based on multi-media activities (videos and simulations) on MyPsychLab, as well as discussion questions posted to Blackboard; see Appendix A. The quick-review quizzes and multi-media activities reviewed key concepts from the chapters, with immediate feedback provided after each quiz question. We attempted to equate time-on-task across formats by assigning hybrid students ∼45 minutes of videos and simulations and ∼15 minutes of quizzes and discussion questions to replace one hour of in-class instructional time per week.

MyPsychLab grades were collected from hybrid sections to examine the relationship between MyPsychLab homework and exam grades. MyPsychLab grades were based on the percentage correct of assigned work. Because MyPsychLab allows students to retake quizzes without penalty for prior incorrect answers, the percentage correct correlates highly with work completed. We did not use time logged into MyPsychLab as a measure of engagement because students could remain logged onto the site for hours while inactive, and login times included access to the e-text as well as homework.

Student surveys

Paper surveys administered anonymously on the first day of class, asked students to provide demographic information. Additional surveys administered on the first day of class, on the day of the first exam (near midterm), and at the end of the semester, asked hybrid students about course features and their satisfaction with the course. Questions included five-point Likert scale items and open response questions; see Appendices B–D for questions; see Table 3 for coding. Because the student surveys were administered anonymously, these data were not linked to subject ID and grades.

Table 3

Coding Scheme for Survey 2 and Survey 3 Open-Ended Questions

Code	Description	Example
Endorsement of Hybrid Format
ENF	Full Endorsement	“No, it is perfect the say it is and truly aided me in learning the material.”
ENN	Endorsement Qualified (not for me)	“Hybrid class is great, but it is not suitable to me.”
Technology Concerns
TMD	MyPsychLab Difficulties (confusion, glitches, stress)	“I feel that they should make the website easier to use.”
TGD	General Dislike (of MyPsychLab or online work)	“I believe that hybrid courses are tedious and horrible. I barely learned anything online.”
TLO	Less Online (too time consuming, too much work)	“Not so much online work, like less questions and videos.”
TTT	Tech Support & Training (need more technical support, need better training and explanation of online requirements)	“Better help from the technology part of the website.”
TED	E-textbook Dislike (issues regarding textbook format)	“It is very challenging doing the online homework using an online textbook. We would be a lot better off with the hardcopy.”
TPS	Online Promotes Slacking Off (students are lazy, don't learn well online)	“Online gives people the chance to slack off.”
TMR	More Reminders (about online homework deadlines)	“If it could send us an email of a reminder to complete the homework then that would be awesome!”
TMT	More Time (to complete online work)	“You can give us more time to do the work online.”
TGC	Grade Calculation (factor online work less)	“Grades should be less dependent on MyPsychLab.”
TOC	Other Concerns (use only one website; need computers in class)	“Put everything in one website, more simple.”
In-Class Time
CII	Increase and improve in-class (need more hours in class; more discussions/activities)	“Another day of class scheduled in a classroom for understanding difficult tasks.”
CIR	Improve Review (need better test prep)	“Reviews for the test or what we had to do in MyPsychLab for further understanding.”
CBI	Better In-class (prefer in-class to online work)	“I rather have the material done in class than online.”
CWH	Written Homework and Handouts (request more)	“I would prefer written assignments.”
CFP	Fast Paced (need to slow down)	“Teach at a slower pace.”
Other
OOB	Online Better (move more or all online)	“I feel as if I am taking 2 of the same classes, one online and one in class. I'd rather not show up to a class to be honest.”
OTH	Other (e.g., require less reading, offer more extra credit, easier tests)	“I don't want to depend on the textbook that much to do well on tests and participate on class discussions.”

Faculty questionnaire

Three instructors who taught in both formats were asked to list up to three things that were positive about the hybrid format, up to three things that were negative about it, and to indicate which format was preferred and why; see Appendix E. This survey was distributed at the end of data collection for the purpose of comparing their experiences across formats.

Results

Course Outcomes

This study compared hybrid sections and traditional sections of Introductory Psychology. While there were more sections of the traditional format (nine compared to six) and thus more students in this format, the sections were statistically equivalent in the distribution of male/female students, class size, number of semesters of college attendance, and declared Psychology majors, all p-values ≥ 0.33 (see Table 1 for means).

Table 4 shows descriptive statistics for outcome measures as a function of course format. Preliminary analyses, using multilevel Poisson regression with by-class random intercepts, revealed no significant difference between the two groups in number of exams (out of three) taken (B = 0.001, Z = 0.15, p = 0.88). To compare exam scores (number correct out of 25 questions) as a function of class format, a multilevel linear regression was fit in the lme4 1.1-7 (Bates, Maechler, Bolker, & Walker, 2014) package in R, with participants nested within class sections and exams nested within participants. To distinguish the three exams, covering different topics and administered at different points in the semester, exam number was included in the model; note that any effect of exam number might reflect either different content or the time of the test. Course type and exam number were both sum coded. Random intercepts were included at both the participant and section level. While it is logically possible to include random slopes at the class section level, the small number of course sections (N = 15) precluded reliable estimation of these effects. Statistical inference was conducted using the lmerTest package (Kuzntsova, Brockhoff, & Christensen, 2014), with Satterthwaite approximated degrees of freedom. Results revealed a significant main effect of exam number, F(2, 1227.72) = 51.90, p<0.001, a non-significant main effect for format, F(1, 13.03) = 2.53, p = 0.14, and a significant interaction between exam number and format, F(2, 1227.72) = 4.35, p = 0.01. As shown in Figure 1, successive exams were apparently more difficult, with hybrid students showing a steeper decline in performance over time than traditional students.

Figure 1.

Mean grades for each exam as a function of class format. Total number of questions per exam = 25.

Table 4

Mean Class Averages as a Function of Format (N = 6 hybrid sections, 9 traditional sections)

Format	Exam Grade (out of 25) M (SD)	Exam Count (out of 3) M (SD)	Final Grade M (SD)	% Attrition (W or WU)	MyPsychLab Homework M (SD)
Hybrid	16.53 (3.96)	2.55 (0.93)	78.1% (15.7)	16.8%	66.4% (28.9)
Traditional	17.64 (4.10)	2.57 (0.95)	82.9% (12.7)	14.8%	NA

To test for an effect of instructor on course outcomes, we fit an additional model with instructor as a dummy-coded factor variable. The factor for instructor was non-significant, F(7, 5.97) = 2.34, p = 0.16, while the main effect of exam number and the interaction between exam number and format remained significant, F(2, 1228.09) = 51.98, p < 0.001 and F(2, 1228.1) = 4.34, p = 0.01, respectively. However, a likelihood ratio test indicated that including instructor significantly improved model fit, χ² (7) = 19.749, p = 0.006. Given this ambiguous evidence of an instructor effect, an additional model was fit to students of the instructors who taught both traditional sections and hybrid sections (random effects included by-participants as there were very few sections in this sample (N = 6), and instructor was included as a dummy variable). As with the model reported earlier, this model revealed a significant main effect of exam number (F(2, 474.11) = 8.07, p < 0.001), and an interaction between exam number and format (F(2, 474.15) = 12.01, p < 0.001) with a non-significant effect of format (F(1, 256.03) = 2.33, p = 0.13).

To examine the interaction in the full sample, a post-hoc test of interactions was conducted using the phia package (De Rosario-Martinez, 2015). While exam scores decreased significantly from Exam 1 to Exam 3 in both hybrid and traditional classes, χ²(1) = 71.46, p < 0.001, and χ²(1) = 33.16, p < 0.001, respectively, the difference between Exam 1 and 3 was larger for hybrid than for traditional students, χ²(1) = 8.64, p = 0.01. The phia package was also used to test for condition effects. Results revealed that differences between traditional and hybrid students at Exams 1 and 2 were non-significant, χ²(1) = 0.54, p = 0.46, χ²(1) = 2.52 p = 0.22, respectively, while the difference at Exam 3 was marginally significant, χ²(1) = 4.95 p = 0.078.

To test whether the interaction could reflect differential attrition across groups, withdrawal rates were compared: 49 of 291 students (16.8%) in the hybrid classes withdrew before the end of the semester while 65 of 439 students (14.8%) in the traditional classes withdrew. A multilevel, logistic regression with by-class random intercepts revealed that this difference was non-significant (B = –0.21, Z = –0.616, p = 0.54). Students who did not withdraw from the class were compared in final grades. A multilevel linear regression with by-class random intercepts revealed no significant difference between the two groups in final grades (B = 4.9, t(13) = 1.93, p = 0.08).

For hybrid students, the average score on MyPsychLab homework assignments was 66.4% (SD = 28.9). To examine whether MyPsychLab homework correlated with exam scores, an average exam score was calculated for each participant. A multilevel linear model revealed a significant, positive relationship between exam and MyPsychLab homework grades, B = 0.04, t(237.45) = 6.72, p < 0.001; r(240) = 0.40, p < 0.001.

Surveys of Students in Hybrid Sections

Survey 1

Of the 291 students enrolled in hybrid sections, 252 (87%) turned in Survey 1 (Appendix B), where they provided reasons for taking the course; see Table 5. When asked why they enrolled in PSY100, the majority (61%) responded that it was required for their major. When asked about their reasons for choosing a hybrid course, the most popular reason was “better fit with work schedule.” Surprisingly, 30% responded false/very false to the statement “I am interested in online learning”—despite their having enrolled in a hybrid course!

Table 5

Survey 1 (Intake)—My main reason for taking a hybrid class is that:

Reason	Very False	False	No Opinion	True	Very True
I am interested in online learning (N = 248)	9.3%	21.0%	48.4%	19.4%	2.0%
It requires less time in class (N = 244)	5.7%	23.4%	36.1%	27.9%	7.0%
I think it should be easier to get a good grade (N = 247)	9.8%	33.7%	42.3%	13.4%	.8%
I prefer to work alone (N = 248)	6.3%	25.4%	37.1%	27.4%	3.6%
I have childcare issues making it difficult to take classes on campus (N = 247)	52.6%	29.6%	14.2%	1.2%	2.4%
It saves me money on gas and commuting (N = 247)	13.8%	26.3%	28.3%	25.1%	6.5%
It fits better with my work schedule (N = 247)	7.3%	15.4%	28.7%	33.6%	15.0%

Note: modal response indicated in bold.

Survey 2

At midterm, 216 students turned in Survey 2 (i.e., 74% of the original sample; 89% of those who completed the course); see Table 6 for a summary of responses and Appendix C for questionnaire. Responses suggest that over half of the students were positive about the hybrid format at midterm, but showed a stronger preference for in-class over online work.

Table 6

Survey 2 (Midterm)

Question	Strongly Disagree	Disagree	No Opinion	Agree	Strongly Agree
I enjoy the online activities through MyPsychLab (N = 215)	9.8%	17.7%	17.2%	47.4%	7.9%
MyPsychLab helped me to learn the textbook material (N = 216)	4.6%	13.9%	21.8%	45.4%	14.4%
I enjoyed the in class discussions and activities (N = 216)	0.5%	2.3%	6.5%	61.1%	29.6%

Note: modal response indicated in bold.

At midterm students were asked “Is there anything that could change to make the hybrid section better?” which provided an opportunity for them to voice their concerns. Of the 146 students who answered the open-ended question (i.e., 68% of students who completed Survey 2), 29% (N = 42) provided uncritical endorsement of the hybrid format, as in “No, it is perfect the way it is and truly aided me in learning the material” whereas 71% (N = 104) provided one or more suggestions or concerns; see Table 7. Student criticism fell into broad categories of “Technology”, “In-class Time”, and “Other”, with inter-rater percentage agreement of 86%.

Table 7

Qualitative Responses to Open-Ended Questions of Surveys 2 and 3 (percentages for each column exceed 100% due to multiple codes per response)

Code	Survey 2 (Midterm) (N = 146)	Survey 3 (Final) (N = 89)
Endorsement of Hybrid Format
ENF	28.8% (N = 42)	24.7% (N = 22)
ENN	1.4% (N = 2)	0
Technology Concerns
TMD	23.3% (N = 34)	13.5% (N = 12)
TGD	6.8% (N = 10)	11.2% (N = 10)
TLO	8.2% (N = 12)	5.6% (N = 5)
TTT	6.8% (N = 10)	9.0% (N = 8)
TED	3.4% (N = 5)	0
TPS	2.7% (N = 4)	2.2% (N = 2)
TMR	1.4% (N = 2)	2.2% (N = 2)
TMT	1.4% (N = 2)	4.5% (N = 4)
TGC	1.4% (N = 2)	1.1% (N = 1)
TOC	3.4% (N = 5)	2.2% (N = 2)
In-Class Time
CII	12.3% (N = 18)	15.7% (N = 14)
CIR	6.8% (N = 10)	4.5% (N = 4)
CBI	3.4% (N = 5)	2.2% (N = 2)
CWH	2.7% (N = 4)	4.5% (N = 4)
CFP	0	1.1% (N = 1)
Other
OOB	2.7% (N = 4)	9.0% (N = 8)
OTH	10.3% (N = 15)	7.9% (N = 7)

Note: See Table 3 for code abbreviations.

Survey 3

At the end of semester, 220 students turned in Survey 3 (i.e., 76% of the original sample, 91% of those who completed the course); see Table 8 for a summary of responses and Appendix D for questionnaire. Overall, feedback was positive: 37% of students agreed to “I think my grade is better in this section than it would have been in a traditional section”, which suggests that they viewed online homework as aiding their success. Nevertheless, when asked if they “would have preferred a traditional lecture PSY100 course”, 41% agreed.

Table 8.

Survey 3 (Exit)—Overall experience

Reason	Strongly Disagree	Disagree	No Opinion	Agree	Strongly Agree
My experience in PSY100 makes me interested in taking other courses in Psychology (N = 219)	5.9%	4.1%	16.0%	51.1%	22.8%
I would recommend a hybrid course of PSY100 to my friends (N = 219)	11.4%	16.0%	13.2%	36.5%	22.8%
If available I would take other Psychology courses in the hybrid format (N = 220)	12.3%	23.2%	16.4%	30.5%	17.7%
I would have preferred a traditional lecture PSY100 course (N = 220)	7.3%	25.5%	26.4%	22.3%	18.6%
I think my grade is better in this section than it would have been in a traditional section (N = 219)	15.1%	18.3%	30.1%	23.3%	13.2%

Note: modal response indicated in bold.

At the end-of-semester survey, students provided critical feedback through the open-ended question “What can we do to improve the course in the future?” Student feedback reflected the same themes as expressed at midterm. Of the 89 students who answered the open-ended question (i.e., 40% of those who completed Survey 3), 25% (N = 22) provided positive feedback, whereas 75% (N = 67) made one or more recommendations for improvement; see Table 7. Student recommendations fell under the broad categories of “Technology”, “In-Class Time”, and “Other”, with inter-rater percentage agreement of 87%.

Instructor Perceptions of Hybrid and Traditional Classes

The three instructors who taught in both formats were asked to provide up to three positive and three negative statements about the hybrid course format; see Appendix E for questionnaire. Although all three instructors mentioned positive features of the hybrid course, they unanimously expressed a preference for traditional sections. Positive comments acknowledged the flexibility of the format, as in “The hybrid course offers students an opportunity to work around their own schedules by limiting the amount of time they have to be in class.” Online homework was viewed as a means of exposing students to course materials prior to class and encouraging independent learning, as reflected in comments like “The hybrid format better encouraged students to activate their critical thinking skills by engaging with the material on their own rather than strictly relying on the traditional professor-as-expert” and “Students instantly received feedback after taking (online) tests and were able to check for content mastery or go back to the assigned chapter readings to self-correct their answers.”

Negative comments highlighted recurrent problems with instructional technology, homework completion, and time management, as in “Some students experienced difficulty setting up the online portion and continued to have difficulties throughout the semester”, “I experienced an increase in student excuses for why an assignment had not been completed on time”, and “I sometimes felt overloaded with managing both environments. I had to make sure the technical problems were resolved, assign and correct online assignments, constantly remind students to complete their work on time, read and bring the online responses and discussions into the classroom. Back in the classroom, I had to pay close attention to class-time management.”

Discussion

Given the growing popularity of hybrid teaching and the aggressive marketing of publisher-provided online course-packs as learning tools, we compared students in 15 sections of Introductory Psychology (six hybrid and nine traditional) using exam averages, final grades, and attrition rates as measures of learning outcomes. Although the self-selection of students into sections created a quasi-experimental design, we were able to compare students' performance on departmental exams in course sections of comparable size, taught in the same semesters, and covering the same content, but differing in the format of 30% of instructional time. Overall, our results failed to support publisher claims that course-packs improve learning outcomes (e.g., Cramer, 2013; Hudson et al., 2014), as there was no evidence of superior performance for hybrid students utilizing MyPsychLab in comparison to students receiving traditional instruction.

Additionally, we observed a significantly larger decrease in exam grades over the course of the semester for students in the hybrid sections relative to their peers in the traditional sections. The worsening exam performance over the course of the semester might be due to progressively more difficult course material in Exams 2 and 3 and/or to student burnout or difficulties in keeping up with their coursework over time (cf. Alarcon, Edward, & Menke, 2011). Given that the drop in exam grades was greater in the hybrid sections, this suggests that hybrid students may have struggled to learn more difficult concepts independently through MyPsychLab, and may have benefited from more explicit teaching (Alfieri, Brooks, Aldrich, & Tenenbaum, 2011). On average, hybrid students completed only 66% of their required MyPsychLab homework, which suggests that they may have spent less time engaged with the course materials than their peers in the traditional sections. Students may have procrastinated in completing MyPsychLab assignments, which resulted in their skipping assignments—underscoring the need for instructors to motivate students to work online (cf. Osborne, Kriese, Tobey, & Johnson, 2009; Pan, Gunter, Sivo, & Cornell, 2005). Given that MyPsychLab homework was a course requirement and contributed substantially to final grade calculations, low homework grades suggest resistance to using online course-packs (see also Cramer et al., 2012; McKenzie et al., 2013; Vigentini, 2009).

In line with prior correlational studies, as well as studies demonstrating a link between the amount of time logged into a required CMS and student grades in hybrid courses (DeNeui & Dodge, 2006; Forte & Root, 2011), MyPsychLab grades correlated with exam grades. This correlation suggests benefits of self-testing for learning (cf. Roediger & Butler, 2011; Roediger & Karpicke, 2006, for reviews), but might also be interpreted as evidence that more capable students are more likely to complete assigned homework than their less capable peers. As suggested by prior research (Ramdass & Zimmerman, 2011), students with stronger self-regulation skills, motivation, and perceived responsibility for learning are more likely to complete homework assignments and be more effective in engaging in self-testing and assessment.

Overlapping Student and Faculty Concerns about Instructional Technology

Both instructors and students voiced concerns about the technology component of the hybrid course, specifically with respect to time management and frustration with technical difficulties. As many students had little-to-no grasp of the online course requirements at the outset of the course, they required considerable technical support and encouragement to succeed. Whereas students requested that more in-class time be devoted to online work, instructors felt that online support took time away from covering course topics. Students also suggested that more class time was needed for hands-on activities, discussions, and exam review—with some expressing a more general concern that instructors needed to teach at a slower pace. Slowing down, however, was not possible if the hybrid sections were to cover the same topics as the traditional sections.

Technical difficulties were exacerbated as MyPsychLab underwent revision over the course of the study, as new features were added and the website was expanded and re-configured. Unfortunately, optimal web-browsers changed across MyPsychLab versions, and publisher representatives sometimes failed to have up-to-date information about compatibility. A frequent comment was that the videos failed to load, which might account for comments that fewer videos be used. As we have no comparative data, we simply do not know whether students would have fared any better with a different course-pack (e.g., WileyPLUS, and LaunchPad). Furthermore, it should be emphasized that many students caught on quickly and apparently thrived, as indicated by end-of-semester comments like “The format of the PsychLab was a little confusing at first, but overall the class was great and I would definitely take another hybrid class.”

Matching Instructional Formats to Student Preferences

Given the range of student engagement with the online work and recurrent complaints about MyPsychLab, it would seem advantageous for students to self-select into course formats that match their interests in using instructional technology. Online coursework offers students the flexibility of completing their studies at their convenience, while aiding institutions in meeting increased enrollment demands, especially for courses at the introductory level (Powers et al., 2013; Tallent-Runnels et al., 2006). However, hybrid courses require personal responsibility and motivation to pursue learning independently; students with these qualities may like the flexibility of working online in combination with reduced in-class instruction. In contrast, students who work better in highly structured environments tend to prefer traditional courses (Jensen, 2011).

As indicated by the survey taken on the first day of class, the majority of students self-selected into hybrid sections without noticing online requirements. Such lack of awareness—reflected in statements like “It's not the course that bothered me. I just didn't know what I was getting myself into when I joined this hybrid course”—precludes students' careful consideration of how varying course demands might fit their expectations. In this regard, several students commented that more information regarding the hybrid format was needed before registration. Results of surveys taken at midterm and at the end of the semester indicate that the hybrid format was not a good fit for a substantial number of students: 28% of hybrid students who took the survey at midterm disagreed with “I enjoy the online activities through MyPsychLab”, and 19% disagreed with “MyPsychLab helped me to learn the textbook material”. This level of satisfaction with MyPsychLab cannot be directly compared to other studies (e.g., Cramer et al., 2012; McKenzie et al. 2013), as none have required course-pack usage and rates of voluntary usage have been low. Our study appears to be the first to examine student satisfaction with a publisher-provided course-pack when its usage was required.

In addition to the varying online requirements, students in hybrid and traditional sections used different textbook formats—with hybrid students encouraged to purchase the e-text, and traditional students encouraged to purchase paper copies. Although we did not collect data on student textbook usage, we are fairly certain that most hybrid students purchased e-texts. Notably, several students reported that they disliked the e-text in the surveys, which fits with other reports of student preferences for paper textbooks (Shepperd, Grace, & Koch, 2008; Woody, Daniel, & Baker, 2010). Students may require more time to read material in e-texts as compared to standard textbooks, perhaps due to tendencies to multi-task while engaged with digital media (Daniel & Woody, 2013). Taken together with student comments that the e-text was confusing, hybrid instructors may need to teach reading skills specific to the use of e-texts, such as how to look up unfamiliar vocabulary and use links to access supplementary materials, as well as techniques for reducing multi-tasking to encourage focused attention when reading online.

Limitations

Our study had a number of limitations that should be addressed in future work. First, because students self-selected into course sections based on scheduling and seat availability, we cannot be certain that the students who enrolled in hybrid sections were fully comparable to their peers in traditional sections. While the majority of hybrid students seemed unaware they had enrolled in a section with online requirements, they did know that their class met only once per week for two hours. Given that the traditional sections met twice per week, the hybrid sections may have attracted students with greater constraints on their time due, for example, to employment, family obligations, or heavier course loads. Future research should include a broader range of questions, as well as a pre-test to assess prior knowledge of course material, to ascertain how student characteristics might influence the self-selection process for enrolling in hybrid versus traditional sections. Although the demographic information we collected did not distinguish the students as a function of instructional format, it is impossible to know whether other factors, such as students' self-regulation skills, study habits, motivation to learn, and overall grade point average, might have distinguished the groups.

Second, we acknowledge other instructional features besides the number of in-class hours that distinguished the two groups, including the final grade calculations and the textbook formats. The use of different formulae for computing final grades across formats—necessitated by the need to credit hybrid students for completing online homework—may have increased the perceived stakes of exams for traditional students relative to their peers in hybrid sections. Likewise, the adoption of an e-text, intended to reduce expenses for hybrid students required to purchase MyPsychLab access, may have influenced study habits, for example, by increasing the tendency to engage with the Internet while reading.

A third limitation is that we collected midterm and end-of-semester surveys only from hybrid students, and we interviewed only the instructors who had taught in both formats. Thus, we do not know to what extent students and instructors in traditional sections had course concerns (e.g., too fast pace, and need for more time for in-class activities) that mirrored the concerns of hybrid students and instructors. Although rates of class attrition were similar in hybrid and traditional sections (averaging ∼16%), students' reasons for dropping out may or may not have been the same. (Note that none of the hybrid students who dropped the course had logged into MyPsychLab, which suggests that the online requirements may have deterred them, although other explanations may be possible as well.) Whereas our decision to use anonymous surveys may have encouraged students to express frank views about the structure of the hybrid course, it prevented our linking survey responses with course outcome measures. Thus, we do not know how students' attitudes towards MyPsychLab, the e-text, and other aspects of online instruction impacted their exam and homework grades and attrition, nor could we control for the demographic information collected through the surveys (e.g., gender, and major) in the statistical analyses.

Finally, as our study looked specifically at MyPsychLab as the implementation for the online component of hybrid instruction, we cannot be certain that our results would generalize to other course-packs, nor do we know which course-pack features were most beneficial for learning. Many publisher-provided course-packs are similar in providing quick review quizzes based on textbook material as well as integrated multimedia activities, but differ in details. For example, MyPsychLab provides links to the e-book to assist the student in relating each activity with what they are reading. LearningCurve, part of Macmillan's LaunchPad, provides “adaptive” quizzes tailored to each student's knowledge level, with questions leading up to a target level of mastery. McGraw-Hill's Connect also uses adaptive quizzes, but with a focus on meta-cognition. Given the observed correlations between online homework completed and exam grades in our study and others (e.g., Cramer et al., 2012; McKenzie et al., 2013), the extent to which students are motivated to engage with the online course materials may prove to be the limiting factor in their utility, as all of the publisher-provided course-packs share features designed for effective self-testing, assessment, and review.

Conclusions

Instructional technology has been promoted as a means of creating interactive, learner-focused classrooms for diverse students (Collins & Halverson, 2009; Moeller & Reitzes, 2011). Publisher-provided course-packs have been designed to promote good study habits with ready-made study modules and direct links to an e-text. However, at large public universities like ours, students tend to have poor preparation to engage with instructional technology, and many find online work to be a burden (Wach et al., 2011). By assisting students in navigating online requirements in an introductory-level course, we were able to promote technological literacy as a learning objective. Nonetheless, we observed costs to learning in the hybrid environment over the course of the semester in terms of exam grades. For the hybrid format to reach its potential may require better integration of online and in-class work as well as further supports to increase student engagement. With increased usage of instructional technologies, researchers need to investigate how specific course-pack features and modules impact learning so that online work can be assigned with careful thought to its purpose.

Footnotes

Acknowledgements

First authorship is shared by Powers and Brooks. Preliminary results were presented at the Annual Meeting of the Eastern Psychological Association, March 2013, and the College of Staten Island Undergraduate Research Conference, April 2013. The authors would like to thank Kenny Zeng and Naomi J. Aldrich for their help with data collection and tabulation.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research was supported by a Teaching with Technology mini-grant from the College of Staten Island, City University of New York.

Author biographies

Kasey L. Powers is a PhD student in Psychology at The Graduate Center, CUNY, and in the Instructional Technology and Pedagogy certificate program. Her research focus is on pedagogy and instructional technology as well as media literacy and how people understand and use digital technology.

Patricia J. Brooks is Professor of Psychology at the College of Staten Island, CUNY, where she directs the Language Learning Laboratory. She serves as the Deputy Executive Officer of the CUNY PhD Program in Psychology (Area: Pedagogy) and Faculty Advisor to the Graduate Student Teaching Association of the American Psychological Association. Her research interests are in two broad areas: (1) individual differences in language learning; and (2) development of effective pedagogy to support diverse learners.

Magdalena Galazyn is a PhD student in Psychology at The Graduate Center, CUNY. Her research focus is working with immigrant populations and non-native English speakers. She is a dedicated instructor, teaching multiple courses each year.

Seamus Donnelly is a PhD student in Educational Psychology at the CUNY Graduate Center. His primary research interests are psycholinguistics and the use of generalized linear mixed models in psycholinguistic research.

Appendix A. Descriptions of online assignments

References

Alfieri

Brooks

P. J.

Aldrich

N. J.

Tenenbaum

H. R.

(2011) Does discovery-based instruction enhance learning? Journal of Educational Psychology 103(1): 1–18.

Alarcon

G. M.

Edwards

J. M.

Menke

L. E.

(2011) Students burnout and engagement: A test of the conservation of resources theory. The Journal of Psychology 145(3): 211–227.

Bates, D., Maechler, M., Bolker, B., & Walker, S. (2014). lme4: Linear mixed-effects models using Eigen and S4. R package version 1.1-7. Retrieved from https://cran.r-project.org/web/packages/lme4/index.html.

Bembenutty

(2009) Academic delay of gratification, self-regulation of learning, gender differences, and expectancy-value. Personality and Individual Differences 46(3): 347–352.

Brown

P. C.

Roediger

H. L.

III McDaniel

M. A.

(2014) Making it stick: The science of successful learning, Cambridge, MA: Harvard University Press.

Cepeda

N. J.

Pashler

Vul

Wixted

J. T.

Rohrer

(2006) Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin 132(3): 354–380.

Chapman

Ludlow

(2010) Can downsizing college class sizes augment student outcomes? An investigation of the effects of class size on student learning. The Journal of General Education 59(2): 105–123.

Collins

Halverson

(2009) Rethinking education in the age of technology: The digital revolution and schooling in America, New York, NY: Teachers College Press.

Cottle

N. R.

Glover

R. J.

(2011) Teaching human development: A case for blended learning. Teaching of Psychology 38(3): 205–208.

10.

Cowie

Nichols

(2010) The clash of cultures: Hybrid learning course development as management of tension. Journal of Distance Education 24(1): 77–90.

11.

Cramer, K. (2013). MyPsychLab case study. Retrieved from http://www.pearsonmylabandmastering.com/northamerica/results/files/Case_Study_MyPsychLab_UWindsor_Ken_Cramer_2010.pdf?v1388436593.

12.

Cramer

Ross

Orr

E. S.

Marcoccia

(2012) Making the grade: Evaluating the construct validity of MyPsychLab as a measure of psychology mastery. Creative Education 3(3): 293–295.

13.

Daniel

D. B.

Woody

W. D.

(2013) E-textbooks at what cost? Performance and use of electronic v. print texts. Computers & Education 62: 18–23.

14.

De Rosario-Martinez, H. (2015). phia: Post-Hoc Interaction Analysis. R package version 0.2-0. Retrieved from http://CRAN.R-project.org/package=phia.

15.

Delialioglu

Yildirim

(2007) Design and development of a technology enhanced hybrid instruction based on MOLTA model: Its effectiveness in comparison to traditional instruction. Computers & Education 51(1): 474–483.

16.

DeNeui

D. L.

Dodge

T. L.

(2006) Asynchronous learning networks and student outcomes: The utility of online learning components in hybrid courses. Journal of Instructional Psychology 33(4): 256–259.

17.

Dunlosky

Rawson

K. A.

Marsh

E. J.

Nathan

M. J.

Willingham

D. T.

(2013) Improving students' learning with effective learning techniques promising directions from cognitive and educational psychology. Psychological Science in the Public Interest 14(1): 4–58.

18.

El Mansour

Mupinga

D. M.

(2007) Students' positive and negative experiences in hybrid and online classes. College Student Journal 41(1): 242–248.

19.

Forte

J. A.

Root

(2011) To ITV or not to ITV: A comparison of hybrid and web-enhanced approaches to teaching a macro-course in human behavior in the social environment. Journal of Human Behavior in the Social Environment 21(1): 82–96.

20.

Gerard

J. G.

Gerard

R. L.

Casile

(2009) Course crash in hybrid space: An exploration and recommendations for virtual course space. Journal of Educational Technology Systems 38(2): 165–180.

21.

Gurung

R. A.

(2005) How do students really study (and does it matter)? Teaching of Psychology 32: 239–241. .

22.

Gurung

R. A.

Weidert

Jeske

(2010) Focusing on how students study. Journal of the Scholarship of Teaching and Learning 10(1): 28–35.

23.

Hudson, D. L., Whisenhunt, B. L., Shoptaugh, C. F., Rost, A. D., Fondren-Happel, R. N., & Cathey, C. (2014). MyPsychLab case study. Retrieved from http://www.pearsoned.com/wp-content/uploads/MyPsychLab_MissouriStateU_Hudson_Whisenhunt_CS_Update.pdf.

24.

Jacobsen

W. C.

Forste

(2011) The wired generation: Academic and social outcomes of electronic media use among university students. Cyberpsychology, Behavior, and Social Networking 14(5): 275–280.

25.

Jensen

S. A.

(2011) In-class versus online video lectures: Similar learning outcomes, but a preference for in-class. Teaching of Psychology 38(4): 298–302.

26.

Karpicke

J. D.

Butler

A. C.

Roediger

H. L.

(2009) Metacognitive strategies in student learning: Do students practice retrieval when they study on their own? Memory 7(4): 471–479.

27.

Koedinger

K. R.

Kim

Jia

J. Z.

McLaughlin

E. A.

Bier

N. L.

(2015) Learning is not a spectator sport: Doing is better than watching for learning from a MOOC. In: Kiczales

Russell

D. M.

Woolf

(eds) L@S '15: Proceedings of the second (2015) ACM conference on learning @ scale, New York: ACM, pp. 111–120.

28.

Kuznetsova, A., Brockhoff, P., & Christensen, R. (2014). lmerTest: Tests in Linear Mixed Effects Models. R package version 2.0-20. Retrieved from http://CRAN.R-project.org/package=lmerTest.

29.

LearningCurve (2016). Retrieved from http://www.macmillanhighered.com/Catalog/elearningbrowsebymediatype/LearningCurve.

30.

Levine

L. E.

Waite

B. M.

Bowman

L. L.

(2007) Electronic media use, reading, and academic distractibility in college youth. CyberPsychology & Behavior 10(4): 560–566.

31.

Lilienfeld

S. O.

Lynn

S. J.

Namy

L. L.

Woolf

N. J.

(2011) Psychology: from inquiry to understanding, Plus NEW MyPsychLab, 2nd Edition, New York, NY: Pearson Education, Inc.

32.

McKenzie

W. A.

Perini

Rohlf

Toukhsati

Conduit

Sanson

(2013) A blended learning lecture delivery model for large and diverse undergraduate cohorts. Computers & Education 64: 116–126.

33.

Moeller

Reitzes

(2011) Integrating technology with student-centered learning, Quincy, MA: Nellie Mae Education Foundation.

34.

Moore

J. L.

Dickson-Deane

Galyen

(2011) e-Learning, online learning, and distance learning environments: Are they the same? The Internet and Higher Education 14(2): 129–135.

35.

MyPsychLab (2016). Retrieved from http://www.pearsonmylabandmastering.com/northamerica/mypsychlab/educators/results/results-library.php?product[]=MyPsychLab&hpd.

36.

Osborne

R. E.

Kriese

Tobey

Johnson

(2009) And never the two shall meet? Student vs. faculty perceptions of online courses. Journal of Educational Computing Research 40(2): 171–182.

37.

Pan

Gunter

Sivo

Cornell

(2005) End-user acceptance of a learning management system in two hybrid large-sized introductory undergraduate courses: A case study. Journal of Educational Technology Systems 33(4): 355–365.

38.

Poirier

(2010) A hybrid course design: The best of both educational worlds. Techniques: Connecting Education and Careers 85(6): 28–30.

39.

Powers

K. L.

Brooks

P. J.

McCloskey

D. P.

Sekerina

I. A.

Cohen

(2013) Hybrid teaching of psychology. In: Hamada

(ed.) E-learning: New technology, application and future trends, Hauppauge, NY: Nova Science Publishers, pp. 147–166.

40.

Ramdass

Zimmerman

B. J.

(2011) Developing self-regulation skills: The important role of homework. Journal of Advanced Academics 22(2): 194–218.

41.

Rau

M. A.

Aleven

Rummel

(2013) Interleaved practice in multi-dimensional learning tasks: Which dimension should we interleave? Learning and Instruction 23(1): 98–114.

42.

Roediger

H. L.

Butler

A. C.

(2011) The critical role of retrieval practice in long-term retention. Trends in Cognitive Science 15(1): 20–27.

43.

Roediger

H. L.

Karpicke

J. D.

(2006) The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science 1(3): 181–210.

44.

Seabrook

Brown

G. A.

Solity

J. E.

(2005) Distributed and massed practice: From laboratory to classroom. Applied Cognitive Psychology 19(1): 107–122.

45.

Shepperd

J. A.

Grace

J. L.

Koch

E. J.

(2008) Evaluating the electronic textbook: Is it time to dispense with the paper text? Teaching of Psychology 35(1): 2–5.

46.

Tallent-Runnels

M. K.

Thomas

J. A.

Lan

W. Y.

Cooper

Ahern

T. C.

Saw

S. M.

Liu

(2006) Teaching courses online: A review of the research. Review of Educational Research 76(1): 93–135.

47.

Taylor

Rohrer

(2009) The effects of interleaved practice. Applied Cognitive Psychology 24(6): 837–848.

48.

Thor

(2010) The right mix. Community College Journal 81(1): 38–43.

49.

U.S. Department of Education, Office of Planning, Evaluation, and Policy Development (2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, DC. Retrieved from http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf.

50.

Utts

Sommer

Acredolo

Maher

M. W.

Matthews

H. R.

(2003) A study comparing traditional and hybrid internet-based instruction in introductory statistics classes. Journal of Statistics Education 11(3): np. Retrieved from www.amstat.org/publications/jse/v11n3/utts.html.

51.

Vigentini

(2009) Using learning technology in university courses: Do styles matter? Multicultural Education and Technology Journal 3(1): 17–32.

52.

Wach

Broughton

Powers

(2011) Blending in the Bronx: The dimensions of hybrid course development at Bronx Community College. Journal of Asynchronous Learning Networks 15(1): 87–94.

53.

WileyPLUS (2016). Retrieved from https://www.wileyplus.com/WileyCDA/Section/id-407326.html.

54.

Woody

W. D.

Daniel

D. B.

Baker

C. A.

(2010) E-books or textbooks: Students prefer textbooks. Computers & Education 55(3): 945–948.

55.

Young

J. R.

(2002) “Hybrid” teaching seeks to end the divide between traditional and online instructions. Chronicle of Higher Education 48(28): A33–A34.