The Effects of Survey Timing on Student Evaluation of Teaching Measures Obtained Using Online Surveys

Abstract

Teaching evaluations are an important measurement tool used by business schools in gauging the level of student satisfaction with the educational services delivered by faculty. The growing use of online teaching evaluations has enabled educational administrators to expand the time period during which student evaluation of teaching (SET) surveys can be completed by students. This added benefit increases the complexity of data collection and introduces new questions related to the time window during which SET survey administration should commence and stop. This article examines the role of the timing of SET survey completion on student satisfaction measures in the context of online marketing courses. The results indicate that there are significant differences between those students who respond early to survey invitations and those who respond late. Early responders and late responders reflect different segments of the student body, have different course evaluation formation dynamics, and exhibit different grade expectations. The findings suggest the existence of systematic biases in SET scores related to response rate, requiring educators to closely examine policies related to the timing of SET survey administration.

Keywords

course evaluation student satisfaction faculty assessment

Teaching evaluations have become a central measure used by business schools for a range of purposes. Student evaluation of teaching (SET) surveys are the primary means by which most schools assess their faculty’s teaching skills (Kozub, 2010; Simpson & Siguaw, 2000). In addition, SETs have become an integral part of the accreditation process used by the major accreditation agencies that evaluate the quality of business education worldwide (Roller, Andrews, & Bovee, 2003). With the growth of electronic means of information exchange and social media, teaching evaluations are now also becoming a publicly visible reflection of a university’s quality of educational services. In addition to their benefits from an assessment perspective, successful business schools utilize teaching evaluations as a means for providing feedback to their faculty with the objective of improving their teaching skills. From an organizational perspective, teaching evaluations can be viewed as a reflection of customer satisfaction, which decades of research has shown to be a good indicator of the long-term sustainable existence of an organization (Heskett, Sasser, & Wheeler, 2008; Maritz & Nieman, 2008). It is no surprise, therefore, that SET measures are of significant importance to a range of stakeholders in business education.

Given the continuing need for the use of SETs by business schools, and the needs of the various constituencies that they serve, it is important to understand the dynamics by which students interact with teaching evaluation instruments. Furthermore, as higher education undergoes a revolution in delivery form, and transforms into online forms of content delivery, understanding the dynamics of how students process and respond to online teaching evaluation surveys is of growing importance. What is unique about online teaching evaluations is that they allow students to complete their course evaluations privately and at their own convenience rather than being bound to the setting and time of the class session during which the survey is administered, as would be the case for traditional pencil-and-paper in-class SET surveys. Online survey delivery modes are heavily used in distance and hybrid online courses and are increasing in popularity for traditional face-to-face classes as well due to their efficiency and cost advantages. The changeover to online administration of SET surveys can drastically affect the dynamics by which students interact with the survey instrument. It is therefore of growing importance to understand the dynamics by which students react to online course evaluations.

In this article, we will examine the effects of the timing of survey administration on student satisfaction ratings provided by business students in the specific context of online marketing courses. Data from 178 students registered in eight sections of marketing courses are used. The effects of survey completion timing, as measured by the length of time it took a student to respond to the invitation to complete the SET survey, is examined and contrasted with other measures related to student satisfaction. Results from the study indicate that student satisfaction ratings are affected by survey timing and that early responders are different in the way they evaluate a course when compared to late responders. The article concludes with a discussion of the findings and implications for faculty assessment practices.

Measurement of Teaching Evaluations

The use of measurement processes to assess quality is a long-held tradition in higher education. For more than half a century, universities have gauged student satisfaction utilizing SET survey instruments, and the various measures produced by these surveys have provided administrators with the students’ perspectives on their faculty (Rosa & Amaral, 2014). This is despite the fact that research has shown that the correlation between teaching evaluation measures obtained through SETs and objective learning outcomes is in fact low, putting into question the very use of SET measures for gauging teaching quality (Clayson, 2009). Nevertheless, these measures are widely relied on by various stakeholders, each with their own perspectives and uses. For business school administrators, teaching evaluations are used as an indicator of faculty performance, with significant impact on a range of organizational practices such as the granting of tenure, the promotion of faculty, renewing faculty contracts, determining the recipients of teaching awards, as well as other outcomes and rewards that institutionally are expected to emerge from quality teaching. From a regulatory perspective, teaching evaluations are central to the accreditation process and relied on by the major accreditation agencies both in the United States and abroad (Gaston & Ochoa, 2013; Roller et al., 2003). Externally, teaching evaluations conducted by third parties such as RateMyProfessors.com and RateMyTeachers.com provide a publically visible perspective on the institution as a whole as well as the individual faculty who contribute to its teaching mission.

While debates continue regarding the validity of teaching evaluation surveys as a means for assessing teaching quality, similar debates exist with respect to the use of such surveys in guiding teaching improvements and faculty development. One view asserts that teaching evaluations can have corrective uses by helping calibrate pedagogical approaches (Kozub, 2010; Madden, Dillon, & Leak, 2010; Simpson & Siguaw, 2000). For example, changes over time in an instructor’s teaching evaluations can signal whether modifications of teaching practices have resulted in improvements in teaching ratings. However, research has also demonstrated that SET measures (and associated shifts in such measures) may not be reflective of students’ actual learning (or associated changes in objective learning outcomes) and may therefore be inaccurate measures of teacher effectiveness (Clayson, 2009; Sitzmann, Ely, Brown, & Bauer, 2010). Nevertheless, student evaluations of teaching continue to be a widely used indicator of faculty performance, and for this reason it is essential to understand the factors which influence such evaluations.

Online Teaching Evaluations

Online administration of teaching evaluations is a data gathering practice that has grown in recent years. While at the turn of the century there were very few universities that administered teaching evaluations using online means, today a significant number of institutions of higher education utilize such an approach to collect SET data (Nevo, McClean, & Nevo, 2010). Research indicates that several benefits are associated with online administration of teaching evaluations (Guder & Malliaris, 2010; Popham, 2013). The primary benefit is cost savings, as online administration of teaching evaluations costs a fraction of traditional pencil-and-paper methods. This is because the costs associated with the physical distribution of paper questionnaires as well as the costs of scanning completed questionnaire forms for data input would no longer be incurred in the online format.

Additional benefits are also associated with online teaching evaluations. Online administration enables students to respond to the survey questions at a time and location of convenience to them. This can clearly benefit the students, but it can also improve the accuracy of the collected data and thereby improve the quality of the SET measures obtained (Nevo et al., 2010; Popham, 2013). This latter benefit is due to the fact that in contrast to traditional pencil-and-paper teaching evaluations, which are typically administered within the confines of a classroom, students can take their time and respond to the questions in the privacy and convenience of their own physical settings and may therefore have the opportunity to be more insightful in their responses. In addition, since the survey invitations are distributed using electronic means to all students, every student would have the opportunity to respond if he/she chooses to do so. In contrast, when pencil-and-paper survey instruments are used, students who are not able to attend the class session in which the SET survey is being administered would not have the opportunity to express their views. Despite the above-mentioned benefits, online administration of SET surveys introduces new challenges to the ways by which teaching is assessed. One of the central questions is regarding the effects of the timing of the distribution of these surveys, a topic that will be discussed below.

Effects of Teaching Evaluation Timing

In contrast to face-to-face teaching evaluations administered in a class setting at the end of the term, online teaching evaluations are administered to students using online means with greater flexibility. Online survey administration allows students to respond during a window of time at their own convenience. The emerging challenge from the increased degree of flexibility relates to when one should invite students to complete online teaching evaluation questionnaires. This is a sharp contrast to survey administration approaches used in classroom settings in which class time at the end of the term is used for students to fill out teaching evaluation questionnaires. In contrast, when an online teaching evaluation instrument is administered, the exact timing of when the student completes the questionnaire is no longer controllable, nor is there an assurance that all students will respond in a timely manner. This is because students who receive the survey invitation link can choose to respond at time most convenient to them. This increased flexibility complicates the decision of when one should commence the administration of teaching evaluation surveys and when data collection should cease.

Relationship Between Survey Completion Timing and Student Satisfaction Measures

Research on survey methods has shown that the length of time that a respondent takes in responding to survey invitation can be correlated with the polarity of measures provided by the respondent. Typically, early responders are different from late responders in terms of specific underlying measures such as involvement level, economic resources, time availability, and other segmentation criteria (Fowler, 2008; Tian & Tang, 2013). Respondents who choose to immediately participate tend to be more involved and engaged with the survey topic area when compared to late responders.

Differences in response polarity as a function of the time delay in responding to survey invitations have been observed across a range of research and marketing contexts. For example, in the field of consumer research, it has been shown that consumer attitude and preference measures obtained can vary between those consumers who complete surveys early versus those who respond late during the data collection time window (Feld, Frenzen, Krafft, Peters, & Verhoef, 2013). Late responders tend to be less engaged in the product category and exhibit differences in demographic and psychographic measures (Parasuraman, 1982). Such differences also contribute to declining response rates, such that overall response rates for surveys drop over time, to the extent that beyond certain time thresholds receiving reliable responses becomes less likely (Fowler, 2008).

The negative relationship between the delay in respondents’ survey completion and the polarity and quality of responses received has been noted for a range of survey administration media, including surveys administered through mail, phone, and online means (Dillman, Smyth, & Christian, 2008; Fowler, 2008). The same response biases have been observed in fund raising campaigns, whereby late responders are less likely to donate in larger amounts when compared to early responders and tend to be less motivated in supporting the campaigns’ causes (Basil & Herr, 2006; Olsen, Pracejus, & Brown, 2003; Weinstein, 2009). It is therefore likely that in the context of teaching evaluation surveys, similar effects exist, such that more positive responses are expressed by those students who respond early to SET survey invitations.

Hypothesis 1: The length of time between SET survey invitation and survey completion by a student is inversely related to the student’s satisfaction level.

Relationship Between Survey Completion Timing and Student Grades

Research on survey methods has shown that there are differences associated with survey completion timing, such that early responders to survey requests are typically demographically and psychographically different from the late responders. For example, in mail surveys of consumers’ brand preferences, it has been found that late responders are typically less involved consumers and are prone to complete a survey questionnaire due to financial incentives rather than underlying interests in the product (Aaker, Kumar, Leone, & Day, 2012; Goetz, Tyler, & Cook, 1984). Similarly, social research has shown that late responders to surveys regarding social and political matters are typically less attuned to the topic, less favorable toward the views of the survey sponsor, and distinct from early responders in terms of their underlying motivations, ideals, and opinions (Feld et al., 2013; Hayes, 1997).

In a similar manner, in the context of teaching evaluations, the timing of completion of an SET survey by a student may be indicative of the degree of student involvement with the course and hence the nature of responses provided by the student. The degree by which a student reflects positively or negatively about his/her experiences in a course may vary between those students who respond early to the survey invitation versus those who do not respond in a timely manner. This is because students who respond early may be more disciplined, benefit from better study skills, have greater motivation to perform well in the course, and experience a higher degree of self-efficacy in engaging with the course content (Borg, Mason, & Shapiro, 1989; Horn, Bruning, Schraw, Curry, & Katkanant, 1993; Pintrich & Degroot, 1990), resulting in timely completion of the SET survey. This increased degree of engagement with course content is often associated with academically driven segments of the student population (Michaels & Miethe, 1989) who are typically also more receptive to stricter grading policies (Bacon & Novotny, 2002) and aspire to higher academic standards. It is therefore possible that students who respond late to SET survey invitations are those who are less attentive to the course material, less responsive to completing course tasks, and less engaged in the course. These late responders, as a result of their lower degree of engagement with the course may also have lower performance scores in the course. It is therefore expected that

Hypothesis 2: The length of time between SET survey invitation and survey completion by a student is inversely related with the student’s grade expectations.

Differences Between Early and Late Responders in the Drivers of Teaching Evaluations

In addition to differences in overall impressions of the course between early versus late responders, there may also be differences in the underlying psychological dynamics of these two groups. The mindset of early responders may be such that their overall impressions of the course are driven by a different set of factors characterizing the course, when compared to the factors affecting late responders. As suggested previously, early responders may be more academically driven and consist of a different segment of the student population with greater levels of academic discipline and higher grade expectations (Abrami, Dickens, Perry, & Leventhal, 1980; Bacon & Novotny, 2002).

It is therefore possible that the course characteristics that are valued by early responders are different from the course characteristics valued by late responders. Differences in the students’ involvement levels between the two groups as well as differences in their academic profiles may result in one group placing greater emphasis on certain aspects of the course. The overall evaluations of these two groups with respect to the instructor and the course may therefore be driven by distinctly different sets of factors. As a result it is expected that the criteria based on which the two groups form their overall evaluations are different.

Hypothesis 3: The factors influencing student satisfaction are different between early responders and late responders.

Relationship Between Survey Completion Timing and Task Involvement

Variations in student characteristics evident in the delay in responding to an SET survey invitation may also be evident in the amount of effort exerted by the student in completing the survey. Since late responders to an SET survey request may be students who are less involved with the course, their lack of involvement may also carry over to the survey instrument itself. Late responders may be less engaged with the survey process that may have resulted in their delay in responding in the first place. To the same extent they may exert less care in answering the various SET survey questions, filling out the survey in a rushed manner when compared to early responders. It would therefore be expected that the amount of time expended by late responders in completing an SET survey would be less than that of early responders.

Such effects have long been observed by survey research methodologists, as evident by the challenges that researchers face in obtaining accurate answers from late responders in social policy studies, public opinion polls, and consumer surveys. Late responders in these contexts have been found to be less involved in completing the survey instrument (Jones & Lang, 1980; Yaveroglu, Donthu, & Garcia, 2003). Similar effects may therefore exist in the context of SET surveys, whereby students who are late responders may exhibit lower levels of task involvement with the survey instrument, for example, by filling out their teaching evaluations in a shorter time interval or not providing detailed open-ended comments on the SET questionnaire. It is therefore expected that

Hypothesis 4: Task involvement with the SET instrument will be lower for late responders than for early responders.

Methodology

The teaching evaluation questionnaire in this study was the standard teaching evaluation instrument used for end-of-semester teaching evaluations used at an educational institution in northeastern United States. This survey instrument has been used for several years for graduate business courses as a standard instrument for assessment of faculty performance. In this instrument, two overall measures of student satisfaction are used. The first is the student’s overall evaluation of the course, measured by the question, “Overall, my rating of this course is . . ., ” and the second is the student’s overall evaluation of the instructor, measured by the question, “Overall, my rating of this instructor is . . .” For both questions, responses were captured using a 5-point scale: 1 = very poor; 2 = poor; 3 = neutral; 4 = good; 5 = very good. These two questions are highly correlated (r = .74). For each student, the average of the two questions is used as the measure of student satisfaction—the primary dependent variable in this study—consistent with earlier studies on student satisfaction with instructional quality (e.g., Colburn, Sullivan, & Fox, 2012; Gruber, Lowrie, Brodowsky, Reppel, & Voss, 2012).

Additional questions measuring student perceptions of individual aspects of the course are also part of the questionnaire. These items and their exact wording are listed in Table 1. They measure the degree by which the course content increased students’ interest levels, the relevance of the readings and assignments, and the responsiveness, preparedness, and accessibility of the instructor. Students expressed their degree of agreement on questions related to these items using a 5-point response scale, with 1 representing strongly disagree and 5 representing strongly agree. In addition to these measures, students provided what they expect their grades to be at the end of the term, using an ordinal scale (A, B, C, D, and F), and reported how they perceived the workload and amount of learning associated with the online course when comparing them to face-to-face courses, using a 1-to-3 scale (1 = less than; 2 = equal to; 3 = greater than). The questionnaire ended with an open-ended question in which students were able to write down any thoughts they had about the course or the instructor.

Table 1.

Survey Instrument Questions and Basic Statistics.

	Average	Standard deviation
Interest Growth: “My interest in the subject has increased as a consequence of this class”	3.97^a	0.84
Assignments: “Readings and assignments contributed to my appreciation and understanding of the subject”	4.22^a	0.78
Instructor Responsiveness: “The instructor was responsive to the students’ questions and comments”	4.62^a	0.83
Instructor Preparedness: “The instructor was well prepared”	4.65^a	0.73
Instructor Accessibility: “The instructor was accessible and available to students”	4.57^a	0.86
Perceived Work Load: “Compared to other similar courses that are not offered in an online format, would you say the level of workload for this course was . . .”	2.32^b	0.50
Perceived Learning: “Compared to other similar courses that are not offered in an online format, would you say the level of learning for this course was . . .”	2.26^b	0.57

1 = strongly disagree; 2 = disagree to; 3 = neither agree nor disagree; 4 = agree; 5 = strongly agree.

1 = less than; 2 = equal to; 3 = greater than.

The SET data were collected from 182 students registered in eight different course sections of two different MBA-level marketing courses, namely, marketing research and marketing of financial services. Of these students four did not provide complete answers, and as a result the final sample consisted of 178 completed responses. All eight course sections were taught by the same instructor and were delivered in a 15-week trimester timeline. The use of the same instructor across all sections reduces measurement error that may hinder the precision of the analyses due to additional variation that would otherwise be introduced into the data as a result of differences across instructors in terms of their teaching methods and pedagogical approaches. The SET surveys were completed anonymously and distributed online via e-mail 14 days in advance of the end of the term (final exam), and data collection ceased the day before the final exam with a reminder e-mail sent 1 week before the end of the term. The survey was administered anonymously, and responding to the invitation was optional and unforced, consistent with the policies and practices of the educational institution in which this study was conducted. In terms of the student background, since the survey was anonymous it was not possible to obtain individual-level information. However, summary data from a separate anonymous survey instrument administered at the start of the term revealed that the average age of the students was 29.4 years, the average number of years of full-time work experience was 6.6 years, and the average GMAT entrance score was 600. Sixty-eight percent of the students worked full-time, 17% had children living at home, 56% were male, and 15% had taken one or more online courses in the past. The overall survey response rate across the eight course sections was 58%.

A critical measure, focal to the research objectives of this article, was the time gap between survey invitation and survey completion by each student. This measure, referred to as the Survey Response Lag (SRL) in the remainder of this article, is quantified by computing the difference (in number of days) between the day the invitation was sent out and the day a given student completes the SET survey. The greater the SRL, the longer the student has taken to respond to the survey invitation. Figure 1 provides a visual representation of the distribution of SRL.

Figure 1.

Distribution of survey response lag.

As can be seen in Figure 1, the distribution of the responses across the days following the survey invitation is generally evenly distributed. However, a substantial proportion of responses (23%) are captured within 2 days of survey invitation. There is also a moderate rate of growth in response rates as one gets closer to the end of the data-collection period, as 20% of the responses were captured within the last 3 days of the survey administration time window.

Analysis

To examine the relationship between students’ satisfaction and SRL, a graphic plot of student satisfaction across SRL levels was conducted and is shown in Figure 2. As can be seen from Figure 2, there is a drop in student satisfaction levels as the SRL measure increases. Students that take longer to respond to survey invitations on average report lower levels of satisfaction.

Figure 2.

Plot of student satisfaction versus survey response lag.

To determine if the drop observed in Figure 2 is statistically significant, analysis of variance (ANOVA) was applied, with SRL as the independent variable and student satisfaction as the dependent variable. The ANOVA yielded significant results (F_12,165 = 2.53; p < .01). To further examine this relationship, a regression analysis was done with student satisfaction as the dependent variable and SRL as the independent variable. Consistent with the ANOVA results, the regression analysis is statistically significant (F_1,176 = 24.92; p < .01), and the coefficient for SRL is negative and statistically significant (b = −.067; p < .01). Furthermore, the correlation between student satisfaction and SRL is found to be negative (−0.35) and statistically significant (p < .01). These results support Hypothesis 1, which had proposed that a negative relationship exists between students’ delay in completing teaching evaluation surveys and their reported levels of satisfaction.

To test Hypothesis 2, regarding the relationship between SRL and students’ grade expectations, a split-sample analysis was conducted. In this analysis, students who responded within the first 6 days of survey invitation (referred to as “early responders” in the remainder of the article) were analyzed separately from respondents who took longer (“late responders”). Of the total sample, 44% were found to be early responders, and the remaining 56% were late responders. Analysis was then conducted to determine if the percentage of students who expect a grade of A is statistically different between early versus late responders. Of the early responders, 84% expected to receive a grade of A, whereas only 53% of the late responders expected to receive an A. The difference in the percentages was statistically tested using chi-square analysis and found to be significant (p < .01; χ² = 18.9; Φ = 0.33). This result confirms Hypothesis 2 as early responders seem to have higher grade expectations and may be more academically driven, which given the online format of the course may be correlated with their quicker response to the online SET survey invitation.

To test Hypothesis 3, with respect to differences between early and late responders in terms of the factors associated with student satisfaction, regression analysis was conducted. To contrast the dynamics by which student satisfaction is determined for early responders versus late responders, regression analyses were conducted separately for each group. Student satisfaction was used as the dependent variable. In the regression analyses, five of the independent variables (interest growth, assignments, instructor responsiveness, instructor preparedness, and instructor accessibility) are measured using interval scales. However two measures (perceived workload and perceived learning) are measured using an ordinal scale whereby students report if they perceive the course to be equal to, less than, or greater than a similar face-to-face class on these two measures. Considering the distribution of these measures, dummy variable coding had to be used in order to transform these two variables, such that if they are perceived to be less than or equal to a face-to-face class, the corresponding dummy variable takes on a value of 0, and if they are perceived as being greater than a face-to-face course, the dummy variable takes on a value of 1. The results of the two regression analyses are shown in Table 2.

Table 2.

Student Satisfaction Regressions (Late Responders Versus Early Responders).

Predictor	Early responders	Late responders
Intercept^a	3.75*** (8.22)	0.17 (0.62)
Interest growth^a	0.21*** (2.81)	0.37*** (5.82)
Assignments^a	0.02 (0.20)	0.06 (0.81)
Instructor responsiveness	−0.13 (−0.84)	0.14 (1.37)
Instructor preparedness^a	0.02 (0.12)	0.18** (2.01)
Instructor accessibility	0.11 (0.99)	0.15 (1.45)
Perceived work load^a	0.04 (0.37)	0.21** (2.35)
Perceived learning^a	0.12 (1.10)	0.16* (1.69)
R ²	0.21	0.77
F value	2.63	44.45
Sample size	79	99

Note. Numbers in parentheses are t values.

Indicates regression coefficients between the two groups are different at the p < .05 level.

***

p < .01. **p < .05. *p < .1.

As can be seen, both regression analyses are statistically significant (p < .01). For early responders, student satisfaction is positively related to interest growth (p < .01). While interest growth also affects student satisfaction among late responders, the variables that related to student satisfaction for late responders are different. For late responders, student satisfaction is also related to instructor preparedness (p < .05), work load (p < .05), and perceived learning (p < .1).

What is especially notable in Table 2 is the varying dynamics by which the two groups’ satisfaction levels are determined. Tests of differences in the regression coefficients between the two groups indicate systematic variations at statistically significant levels. t Tests were conducted to assess if the regression coefficients for the late responders are different from regression coefficients for early responders, and the differences were found to be statistically different at the p < .01 level for interest growth, assignments, instructor preparedness, work load, perceived learning, and SRL. This suggests that, consistent with Hypothesis 3, which relates to the varying dynamics of satisfaction formation for early and late responders, the satisfaction formation dynamics of these two student segments are indeed different.

To determine if the level of involvement of early responders is different from that of late responders, in terms of the effort they exert in completing the SET survey, two proxy measures for effort were used. The first is the length of time (in minutes) that took the student to complete the teaching evaluation questionnaire. This is measured (in minutes) in terms of the length of time between when the student clicks on the SET survey link to start the survey and when he/she submits the completed survey. The second measure was whether or not the student provided a written response for the open-ended question at the end of the questionnaire. On average, students took 2.5 minutes to complete the SET survey and 36% provided written comments. To test Hypothesis 4, regarding task involvement differences between early and late responders, a t test was conducted on the completion time between the two groups. Early responders completed the SET survey in 2.83 minutes, while late responders completed the survey in 2.27 minutes. Although the greater amount of time expended by early responders is consistent with Hypothesis 4, this difference does not reach statistically significant levels (p = .32; t = 1.01), and the effect size computed using Cohen’s d formula is only 0.15. With respect to the percentage frequency of comments, of the early responders, 32% provided a comment, compared to 39% for late responders. This difference also does not reach statistical significance (χ² = 1.15; Φ = −0.08; p = .28). While directionally various measures of task involvement seem to indicate that there might be some difference in task involvement between the two groups, the statistical tests cannot confirm this difference, which had been hypothesized in Hypothesis 4, and additional data points (e.g., larger sample size) would therefore be needed to support or refute this hypothesis.

Discussion

The fundamental question of interest in this article was to determine if the timing of completion of course evaluation surveys is related to the ratings provided by the students. The results, in the context of online marketing courses, clearly indicate that early responders on average report more positive responses than late responders. Furthermore, the results indicate that the dynamics by which these two groups form their overall evaluations vary. These results are important from administrative and measurement perspectives, as they suggest the possibility of significant nonresponse biases (Guder & Malliaris, 2013; Sax, Gilmartin, & Bryant, 2003) that may influence online student satisfaction ratings, making it essential to seek out student satisfaction ratings from those students who may otherwise be nonresponders.

To determine the effect of full participation of students in completing SET surveys, regression analysis was conducted with the rolling average satisfaction level for the sample (at time of a given student completing the SET survey) as the dependent measure. The independent variable used was the representation of the student in terms of the cumulative percentage of all registered students who have completed the survey (at the time of the student completing the survey). This cumulative percentage could range from a theoretical low of zero (where no student would reply to the SET survey invitation) to a theoretical high of 100% (where all students complete survey). The coefficient for the cumulative percentage from this regression is then used to extrapolate what the rolling average for student satisfaction would be, if all of registered students complete the SET survey. Based on this analysis, the estimated overall average student satisfaction measure would decline from 4.5 to 4.3 if all registered students were to participate in the SET survey. If in fact, student satisfaction ratings can change as a result of when the survey instrument is completed, then administrators must make an effort to reduce such a bias or eliminate it altogether by encouraging full participation from the students.

Full participation may be achieved by embedding registration procedures that, for example, mandate that students complete online teaching evaluation surveys before they can access the online course registration system for subsequent semesters, or other means to motivate student response. It is important however to recognize that such requirements may also create inconveniences for students, reduce the validity of the SET measures in the long run if students find the process cumbersome, and pose legal and policy challenges to the educational institution. It is therefore important that measures deployed to secure 100% participation be evaluated closely for all possible concerns, before implementation is undertaken.

It is also important to recognize that late responders and early responders vary in terms of their self-reported grade expectations. This indicates that the two groups constitute two different segments of the student population. Since the end-of-semester course evaluations were anonymous, it was not possible to know which student completed which survey and to confirm actual grade differences. However, students’ expectations of their grades were in fact significantly higher for the early responders. This finding confirms the view that late responders are generally those who have lower course performance profiles as reflected by their lower grade expectations. Their lower relative performance standing compared to early responders may contribute to their lower ratings of the course and the instructor. To further investigate this relationship, a regression analysis was done with student satisfaction as the dependent measure, and SRL and grade expectations as the independent variables. The results of this analysis are shown in Table 3. The overall regression was found to be significant at the p < .01 level (F_2,174 = 17.9; R² = 0.17). Furthermore, SRL and grade expectations both have statistically significant effects (p < .01) on student satisfaction. SRL has a negative effect and the effects of grade expectations on student satisfaction ratings is positive. The individual effects of these two variables on student satisfaction, assessed using the regression analysis t values, are roughly equal, but clearly in opposite directions (t_SRL = −3.70; t_{Expected Grade} = 3.11).

Table 3.

Effects of SRL and Expected Grade on Student Satisfaction.

Predictor	Parameter estimate	t Value	Significance level
Intercept	3.53	7.57	p < .01
SRL	−0.05	−3.70	p < .01
Expected grade	0.28	3.11	p < .01

Note. SRL = survey response lag. F_2,174 = 17.9; R² = .17.

There are also differences in the dynamics by which the two student segments form their satisfaction levels, as was evident in the regression analysis results that were shown in Table 2. The higher R² figure associated with the regression analysis for late responders may be due to halo effects (Bagozzi, 1993; Beckwith & Lehmann, 1975; Han, 1989), which occur when initial perceptions formed in the early encounters with the instructor (e.g., during the orientation session at the start of the term) may have a great effect on the end-of-semester ratings provided by the students. Such initial impressions not only can exceed the effects of other diagnostic variables, but they tend to affect those with lower levels of involvement (Ambady & Rosenthal, 1993; Bagozzi, 1993; Feldman, 1989; March, 1984). The result is a higher degree of correlation among SET survey measures, which may also result in higher levels of model fit, as also noted in earlier studies of the varying impact of halo effects depending on individuals’ motivation and arousal levels (see, e.g., Bagozzi, 1993; Cooper, 1981; Holbrook, 1981; Murphy, Jako, & Anhalt, 1993). In addition to the halo effect, which may result in the differences in regression model results for early versus late responders, the differences in the results between the two groups may be driven by the higher grade expectations associated with early responders. Students receiving higher grades may not scrutinize the different aspects of the course closely as they may be satisfied with their overall course experience. In contrast, late responders (for whom the analysis in this study shows to have lower grade expectations) may be more scrutinizing of the various course characteristics, some of which they may attribute to their low course performance. As a result, late responders may show a stronger relationship between their overall course satisfaction and the individual components of the course measured through the SET survey, resulting in a higher level of R² in the regression analysis for this subgroup.

To further assess possible differences between early and late responders, the coefficient alpha among all the predictors was computed for each of the two groups and found to be higher for the late responders (α = .84) than for the early responders (α = .75) at statistically significant levels (p < .1). It appears that for late responders, their overall impressions of the professor may guide how the teaching evaluation questions (i.e., individual items on the SET survey) are answered. This is notable in the context of the online course format used in this study, whereby an initial face-to-face orientation meeting at the beginning of the term was followed with several video recordings of the professor, providing the students with brief virtual encounters with the professor during the term.

The results of this study indicate that different student segments respond to online SET survey invitations at different points in time, and that potential nonresponse biases may affect the quality of the collected data. It is important to recognize that the overall magnitude of this bias is limited as indicated by the analyses reported in the earlier discussions, whereby the magnitude of the bias was estimated to be about 0.2 scale points on a 1-to-5 rating scale. While the very existence of this bias limits the usability teaching evaluation data as reliable measures for faculty performance assessment, it is important to recognize that such data can still be used to track the effects of changes in pedagogical approaches used by the professor. If specific changes in teaching methods or course delivery are implemented, associated shifts in overall average SET measures can be expected despite the potential existence of nonresponse biases. For this reason, despite the limitations of SET measures, shifts in these measures associated with specific changes in pedagogical practices can still have meaningful diagnostic benefits to educators.

Future Research

It is important to recognize that this research can be extended in many different directions. For example, the data used in this study included a time window preceding the final exam, but does not include the period of time following the final exam, as SET data collection protocols used in the institution where the data were collected prohibited data collection after the final exam period. However, given that online course evaluation surveys do not require the physical presence of the students, educational institutions can in fact (as some do) entertain the possibility of postexam course evaluation data collection. This clearly brings about many complexities, including the possibility that students who respond after the exam period may be biased by their final exam experience as well as their final course grade and may respond differently in light of these inputs which would not exist had data collection terminated prior to the final exam. Nevertheless, understanding the nature of responses for students who provide postexam SET measures in contexts where such data are available may result in an interesting avenue for future research. In addition, future research can focus on examining the effects of timing of SET survey administration in traditional face-to-face class settings, using pencil-and-paper survey instruments. In such a context since all students are completing the survey at the same time and nonresponse for students attending class may be minimal, the survey timing effects identified in this study may be less pronounced. Testing this proposition may be a promising area for future research.

Another relevant question with respect to online course evaluations is regarding the influence of respondent incentives. The data reported in this study allowed students to complete the survey at their own convenience, and was unforced in the sense that students did not have to complete the survey and participation was optional. However, some universities provide explicit incentives for online course evaluation survey completion, for example, by allowing only those students who have completed the course evaluation survey to view their final course grades or to register for future courses through the online course registration system. As discussed earlier, these forced mechanisms for increasing response rates may influence the results, in that many of the students who may otherwise choose to not respond in an unforced survey administration mode would now be required to participate. With forced response mechanisms, students who are uninterested in the course or the survey completion process will be required to provide inputs into the SET survey. Their lower engagement level with the course will most likely reflect in the form of the downward trend in student satisfaction ratings, which had been observed between early responders and late responders in this study. As a result, it is likely that forced responses will result in further depletion in overall student satisfaction ratings measured by SET survey instruments. Future research can test this proposition and examine the effects of forced versus unforced online SET administration, and contrast the dynamics uncovered in this article, in these two different contexts.

Given that the subjects of this study were graduate students, it would also be interesting for future research to examine if the relationships uncovered here would extend to the undergraduate student population, where student involvement and motivation levels are generally different. Future research can also identify the characteristics for the segment of students who respond early to SET survey invitations. While indications for academic superiority of this group were found in this study, future research can examine if demographic differences (measures such as age and gender which were not collected in this study) may exist between early and late responders. Furthermore, since the teaching evaluation instrument used in this study was administered toward the end of the term, it would be interesting for future research to explore potential effects on student satisfaction ratings if the timing of instrument administration was moved, perhaps to earlier points during the term.

Future research can also examine other contexts where the effects of online SET survey timing on the collected data can be studied. For example, it would be important to further explore the relationships uncovered in this study in the context of other marketing courses. In addition, the effects of variations in professors’ teaching styles on the relationships uncovered in this study may be a fruitful line of inquiry for future research. This would be an important undertaking since the results of this study are based on SET data collected from course sections taught by one instructor with multiple teaching awards and teaching ratings substantially above the institution’s average. It is possible that in contexts where student satisfaction levels with an instructor are low, the relationship between SRL and student satisfaction would vary from what was observed in this study. For example, students who feel disenfranchised due to poor teaching by an instructor may be more responsive to completing SET surveys early. As a result, it is possible that for instructors with poor overall SET ratings, higher response rates would exist and the earlier responses would reflect more negative views of the instructor. Future research can therefore explore this possibility using SET data for a large number of faculty with a range of pedagogical practices. Ideally, such an inquiry would take place across multiple disciplines (not just marketing), such that the findings could inform policies and practices related to improving the quality of SET data in all subdisciplines (e.g., accounting, finance, management, etc.) of business education.

Future research can also focus on the effects of student psychographics, socioeconomic status, and technology access, on online SET survey completion timing. While addressing this question would require the inclusion of additional background demographic and psychographic information in SET questionnaires, it may shed additional light on the relationships identified in this study. Furthermore, one can examine the effects on student satisfaction ratings resulting from the widening or narrowing down of the time window during which SET surveys are administered online. It would be expected that widening the time window would reduce the bias, and testing this proposition can be a practically relevant line of inquiry for future research.

Conclusion

The results of this study demonstrate the relationship between the timing of online course evaluations and the ratings and characteristics of student respondents. They highlight the fact that SET survey response timing can influence the obtained measures and raise questions about the adequacy of survey administration methods and policies that allow for nonresponders, as the possibility of having nonresponders may bias the SET measures. The results have direct implications for educational administrators in terms of the selection of an optimal time window associated with online course evaluation data collection. Specifically, the results indicate that significant response biases exist, such that late responders are more likely to report lower evaluations. Therefore, with the goal of inclusiveness in mind, data collection time windows need to enable late responders to also partake in course evaluation surveys and ideally eliminate nonresponse biases by encouraging all students to participate.

It is important to recognize that while the context of this study was online marketing courses, the results observed in this study are likely to generalize to traditional face-to-face courses where end-of-semester course evaluation surveys are administered using online means (rather than pencil-and-paper questionnaires). As technology increases its impact on business education, not only in terms of how we teach through distance learning methods but also how we assess our own performance as educators, improved understanding of the online mechanisms by which students evaluate faculty is becoming more critical. It is therefore hoped that this article has helped inspire increased understanding and future research in this area of growing significance to business educators.

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

References

Aaker

Kumar

Leone

Day

(2012). Marketing research. New York, NY: Wiley.

Abrami

Dickens

Perry

Leventhal

(1980). Do teacher standards for assigning grades affect student evaluations of instruction? Journal of Educational Psychology, 72, 107-118.

Ambady

Rosenthal

(1993). Half a minute: Predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness. Journal of Personality and Social Psychology, 64, 431-441.

Bacon

Novotny

(2002). Exploring achievement striving as a moderator of the grading leniency effect. Journal of Marketing Education, 24, 4-14.

Bagozzi

(1993). The role of arousal in the creation and control of the halo effect in attitude models. Psychology & Marketing, 13, 235-264.

Basil

Herr

(2006). Attitudinal balance and cause-related marketing: An empirical application of balance theory. Journal of Consumer Psychology, 16, 391-402.

Beckwith

Lehmann

(1975). The importance of halo effects in multi-attribute attitude models. Journal of Marketing Research, 12, 265-275.

Borg

Mason

Shapiro

(1989). The case of effort variables in student performance. Journal of Economic Education, 14, 308-313.

Clayson

D. E.

(2009). Student evaluations of teaching: Are they related to what students learn? A meta-analysis and review of the literature. Journal of Marketing Education, 31, 16-23.

10.

Colburn

Sullivan

Fox

(2012). An examination of the impact of team teaching on student learning outcomes and student satisfaction in undergraduate business capstone courses. American Journal of Business Education, 5, 149-158.

11.

Cooper

(1981). Ubiquitous halo. Psychological Bulletin, 90, 218-244.

12.

Dillman

Smyth

Christian

(2008). Internet, mail, and mixed-mode surveys: The tailored design method. New York, NY: Wiley.

13.

Feld

Frenzen

Krafft

Peters

Verhoef

(2013). The effects of mailing design characteristics on direct mail campaign performance. International Journal of Research in Marketing, 30, 143-156.

14.

Feldman

(1989). The association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies. Research in Higher Education, 30, 583-645.

15.

Fowler

(2008). Survey research methods. New York, NY: Sage.

16.

Gaston

Ochoa

(2013). Higher education accreditation: How it’s changing, why it must. Chicago, IL: Stylus.

17.

Goetz

Tyler

Cook

(1984). Promised incentives in media research: A look at data quality, sample representativeness, and response rate. Journal of Marketing Research. 21, 148-156.

18.

Gruber

Lowrie

Brodowsky

Reppel

Voss

(2012). Investigating the influence of professor characteristics on student satisfaction and dissatisfaction: A comparative study. Journal of Marketing Education, 34, 165-173.

19.

Guder

Malliaris

(2010). Online and paper course evaluations. American Journal of Business Education, 3, 131-137.

20.

Guder

Malliaris

(2013). Online and course evaluations response rates. American Journal of Business Education, 6, 333-338.

21.

Han

(1989). Country image: Halo or summary construct? Journal of Marketing Research, 26, 222-229.

22.

Hayes

(1997). Measuring customer satisfaction: Survey design, use, and statistical analysis methods. Tucson, AZ: ASQ Quality Press.

23.

Heskett

Sasser

Wheeler

(2008). Ownership quotient: Putting the service profit chain to work for unbeatable competitive advantage hardcover. Cambridge, MA: Harvard Business School Publishing.

24.

Holbrook

(1984). Using a structural model of halo effect to assess perceptual distortion due to affective overtones. Journal of Consumer Research. 10, 247-258.

25.

Horn

Bruning

Schraw

Curry

Katkanant

(1993). Paths to success in the college classroom. Contemporary Educational Psychology, 18, 464-478.

26.

Jones

Lang

(1980). Sample composition bias and response bias in a mail survey: A comparison of inducement methods. Journal of Marketing Research, 17(1), 69-76.

27.

Kozub

(2010). Relationship of course, instructor, and student characteristics to dimensions of student ratings of teaching effectiveness in business schools. American Journal of Business Education, 3(1), 33-40.

28.

Madden

Dillon

Leak

(2010). Students’ evaluation of teaching: Concerns of item diagnosticity. Journal of Marketing Education, 32, 264-273.

29.

March

(1984). Students’ evaluation of university teaching: dimensionality, reliability, validity, potential biases and utility. Journal of Educational Psychology, 76, 707-754.

30.

Maritz

Nieman

(2008). Implementation of service profit chain initiatives in a franchise system. Journal of Services Marketing, 22(1), 13-23.

31.

Michaels

Miethe

(1989). Academic effort and college grades. Social Forces, 68, 308-320.

32.

Murphy

Jako

Anhalt

(1993). Nature and consequences of halo error: A critical analysis. Journal of Applied Psychology, 78, 218-225.

33.

Nevo

McClean

Nevo

(2010). Harnessing information technology to improve the process of students’ evaluations of teaching: An exploration of students’ critical success factors of online evaluations. Journal of Information Systems Education, 21(1), 99-109.

34.

Olsen

Pracejus

Brown

(2003). When profit equals price: Consumer confusion about donation amounts in cause-related marketing. Journal of Public Policy & Marketing, 22, 170-180.

35.

Parasuraman

(1982). More on the prediction of mail survey response rates. Journal of Marketing Research, 32, 261-269.

36.

Pintrich

Degroot

(1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82(1), 33-40.

37.

Popham

(2013). Evaluating America’s teachers: Mission possible? Thousand Oaks, CA: Corwin Press.

38.

Roller

Andrews

Bovee

(2003). Specialized accreditation of business schools: A comparison of alternative costs, benefits, and motivations. Journal of Education for Business, 78(4), 197-204.

39.

Rosa

Amaral

(2014). Quality assurance in higher education: Contemporary debates. New York, NY: Palgrave.

40.

Sax

Gilmartin

Bryant

(2003). Assessing response rates and nonresponse bias in web and paper surveys. Research in Higher Education, 44, 409-432.

41.

Simpson

Siguaw

(2000). Student evaluations of teaching: An exploratory study of the faculty response. Journal of Marketing Education, 22, 199-213.

42.

Sitzmann

Ely

Brown

Bauer

(2010). Self-assessment of knowledge: A cognitive learning or affective measure? Academy of Management Learning & Education, 9, 169-177.

43.

Tian

Tang

(2013). Incomplete categorical data design: Non-randomized response techniques for sensitive questions in surveys. New York, NY: Chapman & Hall.

44.

Weinstein

(2009). The complete guide to fundraising management. New York, NY: Wiley.

45.

Yaveroglu

Donthu

Garcia

(2003). Antecedents of survey response bias in business-to-business services. Journal of Business & Industrial Marketing, 15, 366-375.