The Use of Collaborative Midterm Student Evaluations to Provide Actionable Results

Abstract

Midterm student evaluations have been shown to be beneficial for providing formative feedback for course improvement. With the purpose of improving instruction in marketing courses, this research introduces and evaluates a novel form of midterm student evaluation of teaching: the online collaborative evaluation. Working in small teams, students comment on their course using an online collaborative document creation tool. Compared with a standard individual evaluation, the online collaborative evaluation was rated significantly higher by students in enjoyment, ease, and ability to provide useful feedback. In addition, comments yielded from the collaborative evaluation provided formative information that could be used to improve student learning. In a marketing class that emphasizes teamwork, the collaborative evaluation of teaching can reinforce the benefits of functioning well as a team, while providing useful information to the instructor to improve the course.

Keywords

collaboration general multivariate statistics instruction development midterm evaluations student evaluations of teaching

Introduction

Midterm student evaluations of teaching (SETs) have repeatedly shown to be beneficial as a tool for eliciting useful student feedback (Abbott, Wulff, Nyquist, Ropp, & Hess, 1990; Cohen, 1980; Overall & Marsh, 1979; Spencer & Schmelkin, 2002). In contrast to conducting evaluations at the end of the semester, administering midterm evaluations allows instructors an opportunity to manage the course expectations of the students, with the potential to increase student satisfaction (Appleton-Knapp & Krentler, 2006). Nevertheless, compared with the voluminous research on end-of-term ratings-based SETs, very little evidence-based research has examined methods for conducting midterm evaluations.

This article introduces and assesses a novel method for soliciting feedback from students that is particularly well-suited for marketing classes: the online collaborative midterm evaluation. With this method, students are asked to comment on their instructor via a standard evaluation format, but using an online collaborative document creation tool—Google Docs—to provide their comments synchronously. Google Docs is free, does not require software to be downloaded, and can be accessed on any computer or mobile device (see http://learn.googleapps.com). Any comments that students type into the document are visible to all other students, allowing students not only to answer the questions themselves but also to comment on other students’ comments in real-time (Blaschke, 2014; Chu & Kennedy, 2011). Many university students are already familiar with Google Docs, as it is a common tool to use when working on group papers and projects.

A collaborative evaluation could potentially have several distinct advantages over traditional SETs:

Collaborating with peers could increase the engagement of students in the evaluation process (Chad, 2012), leading to more thoughtful comments (Kuh, Cruce, Shoup, Kinzie, & Gonyea, 2008).

Collaborating on an evaluation could evoke the interactive and synergistic advantages associated with focus groups (Kitzinger, 1995), leveraging the quality of the feedback to a higher level.

The public nature of the evaluation could increase the accountability of the process, both for the students providing the feedback and for the instructor in the use of the feedback—thus embracing the public visibility inherent in RateMyProfessor-style evaluations of instructors via social media.

This technique of evaluation would seem to be ideally suited for the modern classroom, since millennials are purported to be collaborative (e.g., Elam, Stratton, & Gibson, 2007; Meister & Willyerd, 2010; Raines, 2003), as well as avid consumers of social media (Lenhart, Purcell, Smith, & Zickuhr, 2010). As such, a collaborative teaching evaluation should provide both a more engaging experience for the students and improved feedback for instructors.

This article reports on a trial of the online collaborative midterm evaluation of teaching, administered to marketing courses. The online collaborative evaluation was compared to a traditional, individual, paper-and-pencil form. This article reports on the theory behind the collaborative midterm evaluation, the methodology used to assess the evaluation, and the lessons learned from this experiment.

Midterm Evaluations of Teaching

This research focuses on the formative function of faculty evaluations, in which the intention of the evaluation instrument is to continuously improve and develop courses and instruction (George & Cowan, 2004; Laverie, 2002). Faculty face end-of-term course evaluations with mixed feelings, perhaps because instructors often perceive that the main purpose for universities to conduct student evaluations is not to support faculty in their teaching function but to serve as a convenient, although imperfect, tool to judge faculty for administrative purposes (Kember, Leung, & Kwan, 2002; Penny, 2003; Simpson & Siguaw, 2000; Spooren, Brockx, & Mortelmans, 2013). Student ratings continue to be the primary means by which chairs of marketing departments evaluate their faculty (Honeycutt, Thelen, & Ford, 2010), despite the fact that the AACSB standards for accreditation explicitly discourages the use of end-of-term student evaluations, as well as other indirect measures of quality, as the sole means to judge faculty (AACSB International, 2013).

For formative purposes, evidence suggests that end-of-term ratings-style evaluations by students are deeply flawed (Clayson, 1999, 2009; Dolnicar & Grün, 2009; Madden, Dillon, & Leak, 2010; Simpson & Siguaw, 2000). The students who complete the evaluations are themselves skeptical that their feedback will have any effect on teaching effectiveness (Chen & Hoshower, 2003; Dunegan & Hrivnak, 2003; Gaillard, Mitchell, & Kavota, 2011; Spencer & Schmelkin, 2002), potentially leading to inattentive completion of evaluations. Substantiating this concern, a study by Ahmadi, Helms, and Raiszadeh (2001) found students to self-report that they completed teaching evaluations in a mean of 2.5 minutes, an indication that many students may not put as much thought into their responses as instructors might wish. In contrast, midterm evaluations by students do not share these weaknesses. In fact, the use of midterm student evaluations to elicit feedback on courses and teaching may mitigate many of the problems inherent to end-of-term evaluations and provide improved guidance for faculty. Literature has noted the advantage and benefits for improving and developing courses and instruction derived from midterm evaluations including the improvement of student satisfaction and the improvement of teaching.

First, midterm student evaluations of teaching effectiveness improve student satisfaction with the process (Abbott et al., 1990; Hunt, 2003; Snooks, Neeley, & Williamson, 2004). Due to their timing, midterm evaluations by students have the obvious benefit of allowing the instructor time to make course corrections based on student feedback, thereby improving the course and teaching methods in a direct manner that benefits the very students who supplied the feedback. As such, students have higher levels of satisfaction with midterm evaluations than end-of-semester evaluations. Naturally, this effect implies action on the part of the instructor to analyze, reflect, and act on the midterm feedback submitted. Taking this one step further, the manner in which feedback is collected also has a direct impact on student satisfaction. Student satisfaction increases when a collective, versus individual, feedback process is used and is followed up by detailed instructor response to their feedback (Cohen, 1980; Finelli, Wright, & Pinder-Grover, 2010; Overall & Marsh, 1979; Snooks, Neeley, & Revere, 2007; Wickramasinghe & Timpson, 2006). In a study by Abbott et al. (1990), one collective method of midterm student evaluation that is particularly effective for eliciting superior feedback for faculty is a Small Group Instructional Diagnosis (SGID). With a facilitator in charge, students first work in small groups to develop feedback related to the strengths and weaknesses of a course. The facilitator then leads a class discussion to reach consensus on the main strengths and weaknesses of the course. The facilitator subsequently reports the main findings from the class discussion to the instructor. The obvious drawback of this method of evaluation is that it requires a great deal of resources, both in the time required away from course material by the class and in the need for a third party to moderate the class discussion.

Second, midterm evaluations make a significant contribution to the improvement of teaching (Cohen, 1980; Marsh, 1987; Marsh & Roche, 1997; Overall & Marsh, 1979; Wickramasinghe & Timpson, 2006). Whether it is increased satisfaction with a midterm evaluation, or instructors creating stronger evaluation instruments for the collection of feedback for teaching improvement, research shows that the collection of midterm feedback directly affects teaching improvements, particularly when the instructor provides follow-up to students (Cohen, 1980). Because midterm evaluations generally have no administration-mandated requirements related to procedures and instruments, instructors have a great deal of latitude in designing their own course evaluations often resulting in more open-ended questions on midterm evaluations. Herein lies the diagnostic strength of midterm evaluations—midterm evaluations can be used to elicit open-ended qualitative responses, versus the bubble-in ratings that tend to dominate end-of-term evaluations.

One qualitative questioning approach that is commonly cited in the literature is the Bare Bones Questioning technique (Snooks et al., 2004), which asks an iteration of three questions:

What (if anything) is interfering with your learning? (STOP)

What suggestions do you have to improve your learning? (START)

What is your instructor doing that helps you to learn? (CONTINUE)

While scant research has evaluated the best practices for collecting student feedback via qualitative methods, the Stop/Start/Continue style of evaluation has been found by Hoon, Oliver, Szpakowska, and Newton (2014) to be preferred by students and to elicit higher quality feedback in comparison to a free form type of student evaluation. The Stop/Start/Continue method can also be applied effectively in a corporate environment (Delong, 2011). The benefit of the use of this type of format for eliciting feedback is that it specifically requests developmental feedback as opposed to judgmental feedback.

While it is true that student evaluations of teaching almost always include some form of open-ended questions, the qualitative responses are often given short shrift when the evaluations are administered at the end of the term (Brennan & Williams, 2004). However, research that directly compares students’ perceptions of ratings-style teaching evaluation with more qualitative-based feedback methods has found that students perceive qualitative feedback to be less superficial and more informative (Braskamp, Ory, & Pieper, 1981; Huxham et al., 2008; Ory, Braskamp, & Pieper, 1980). This may be because the content covered by the standard quantitative questions in teaching evaluations does not always reflect the issues that students consider most important (Grebennikov & Shah, 2013; Kabanoff, Richardson, & Brown, 2003). Still, limited research addresses the use of qualitative feedback to improve teaching, in comparison to the large amount of research conducted on the reliability and validity of ratings scales that measure teaching effectiveness.

Fundamentally, the formative function of faculty evaluations is intended to continuously improve and develop courses and instruction (George & Cowan, 2004; Laverie, 2002). Because feedback from midterm student ratings can be used effectively to inform the instructor and lead to improved teaching, many benefits result from the use of midterm evaluations.

The Collaborative Imperative

As the workplace has become more global and complex, with pressures to reduce inefficiencies, make faster decisions, and employ a wide range of skills and competencies, team work has become increasingly valued. Collaborative work—that is, requiring synchronous work of all team members for completion of a specific task or tasks (Arnould, Ducate, & Kost, 2012)—has been demonstrated to outperform the results of activities by individuals on many types of tasks (Katzenbach & Smith, 1993). Yazici (2004) asserts, “As global teams emerge to adapt quickly to competitive changes, companies depend heavily on collaborative work environments” (p. 110).

As a result of an increased respect for the strength of collaboration, ability to work well in a team is a top skill that employers currently look for when hiring graduates for marketing jobs (Finch, Nadeau, & O’Reilly, 2012; Schlee & Harich, 2010). The 2013 AACSB accreditation standards mandate that individual faculty members “actively involve students in the learning process” and “encourage collaboration and cooperation” (AACSB International, 2013, p. 56). Consequently, many marketing classrooms now incorporate collaborative activities into their lessons in diverse ways (see, e.g., Aylesworth, 2008; Cronin, 2009; Duus & Cooray, 2014; Ferrell & Keig, 2013; Granitz & Koernig, 2011). A number of benefits have been found to correspond to introducing teamwork into marketing classes, including increased satisfaction and positive motivation of students (Chad, 2012; Gruber et al., 2012; Krishen, 2013), which in turn can lead to higher quality work (Kuh et al., 2008). Even short, collaborative exercises, when assigned throughout a term, have been shown to improve learning in higher education (Barkley, Cross, & Major, 2014).

Still, successful teamwork requires groundwork. Despite the perception that millennials are a collaborative generation, students need to be taught proper team building techniques, just as they need to be taught other essential business skills (e.g., communication, problem solving, creativity skills). Collaboration can create stress and frustration for some students when team members’ goals are not compatible (Neu, 2012). The proper environment needs to be created, goals for the exercise should be set, and rules must be established (Chapman, Meuter, Toy, & Wright, 2010; Duus & Cooray, 2014; Freeman & Greenacre, 2011; Loughry, Ohland, & Woehr, 2014; Nunamaker, Briggs, Mittleman, Vogel, & Balthazard, 1997; Thompson, 2013). Both the instructor and the students need to be knowledgeable about the advantages (e.g., division of labor, leveraging of skills, synergistic idea generation) and disadvantages (e.g., power struggles, “groupthink,” time inefficiencies) of teamwork to have an optimal group experience.

Collaborative, Formative Feedback

Based on the extensive body of research in this area, we can summarize both the preferences of students and instructors regarding students’ evaluations of teaching effectiveness (Table 1). As discussed previously, students value the opportunity to supply feedback mid-semester when changes can directly impact their course and learning. Second, they prefer a collective small group process that allows them to provide more in-depth qualitative feedback and the chance to receive a response to their input. Instructors, on the other hand, value a process that allows them to capture formative feedback designed to inform and improve their teaching effectiveness while allowing them to offer diverse collaborative activities in their classrooms. Based on these preferences, this study attempts to address each preference via an online evaluation that borrows the collaborative elements of the SGID method (Abbott et al., 1990), but uses a novel technology that takes less time, does not require a facilitator, and allows real-time response to students offering input.

Table 1.

How Proposed Online Collaborative Midterm Evaluation Aligns With Student and Instructor Preferences.

Preference	Alignment
Students
Midterm evaluation (Abbott et al., 1990; Hunt, 2003; Snooks et al., 2004)	Implemented at mid-semester
Small group process (Abbott et al., 1990)	Small groups of 5-7 students
Collaborative, collective approach (Barkley et al., 2014; Chad, 2012; Chapman et al., 2010; Duus & Cooray, 2014; Freeman & Greenacre, 2011; Gruber et al., 2012; Krishen, 2013; Kuh et al., 2008; Loughry et al., 2014; Nunamaker et al., 1997; Thompson, 2013)	Students work collaboratively to complete online instrument.
Qualitative approach (Grebennikov & Shah, 2013; Huxham et al., 2008; Kabanoff et al. 2003)	Bulk of feedback instrument is structured around Bare Bones Questioning technique (Snooks et al., 2004) using Stop/Start/Continue structure.
Desire for a response (Cohen, 1980; Finelli et al., 2010; Overall & Marsh, 1979; Snooks et al., 2007; Wickramasinghe & Timpson, 2006)	Collaboration of Google doc allows students to see responses to their comments from other students for consensus building and feedback. Recommend that Faculty discuss findings with students following completion.
Instructors
Receive formative feedback for teaching improvement (Wulff & Nyquist, 1986)	Qualitative comments can be content analyzed to create feedback around teaching materials, style, and support needed for teaching improvement (Smimou & Dahl, 2012).
Aligns with collaborative paradigm (Barkley et al., 2014; Chad, 2012; Chapman et al., 2010; Duus & Cooray, 2014; Freeman & Greenacre, 2011; Gruber et al., 2012; Krishen, 2013; Kuh et al., 2008; Loughry et al., 2014; Nunamaker et al., 1997; Thompson, 2013)	Provides a new and novel activity requiring collaborative skill-building knowledge and hands-on practice.

Development of the Online Collaborative Midterm Evaluation

The research goal guiding this inquiry was to develop and assess an online collaborative midterm course evaluation. The collaborative evaluation was created over the course of two terms, with improvements made to the procedure following a pilot study. The evaluations were conducted at the midpoint of the term, using students from four sections of marketing classes in the pilot study (n = 110) and from five sections in the official study (n = 140). The sections of the classes encompassed three courses: sales management (taught by one instructor), and principles of marketing and marketing research (taught by a second instructor). In all three courses, the instructors emphasized collaborative work. The marketing research course and the sales management course are composed principally of upper class marketing majors; the introductory marketing class is composed of students of all class standings and diverse majors.

Two forms of evaluation—a traditional, individual paper-and-pencil form (hereafter called the “individual evaluation”) and an online collaborative evaluation instrument (called the “collaborative evaluation”)—were directly compared, using Bare Bones Questioning technique (Snooks et al., 2004; see also George & Cowan, 2004). On both forms, three questions, plus one additional catchall question, were asked: (1) What should your instructor STOP doing (with respect to teaching)? (2) What should your instructor START doing? (3) What should your instructor CONTINUE doing? (4) Other comments? (George & Cowan, 2004). In each class, the students were randomly divided into two groups: (1) a paper-and-pencil individual evaluation group (hereafter called the “individual group”) and (2) an online collaborative evaluation group (the “collaborative group”). The purpose of the individual group was to serve as a control for assessing the value of the collaborative evaluation.

The collaborative group was brought to the computer lab and asked to complete an online evaluation, using a link that was provided on the web-based course management software. The online evaluation was created using Google Docs, with the settings on the Google Docs altered so that anybody with the link could edit the document. In other words, students could type simultaneously into the document, with each student’s comments visible to all other students. This allowed students not only to answer the questions but also to respond to and elaborate on other students’ comments. To provide anonymity, the instructor left the room for 10 minutes while both groups completed the evaluations.

Following the completion of the evaluations, the students who had been in the computer lab returned to the classroom. The instructor returned to the classroom briefly to distribute another measurement instrument: the “evaluation of the evaluation.” The “evaluation of the evaluation” was an anonymous paper-and-pencil questionnaire designed to assess how students felt about the course evaluation they had just completed. The instructor left the room for 5 minutes to provide the students with anonymity to complete this questionnaire.

The “evaluation of the evaluation” included nine items to measure the performance of the two types of evaluation instruments. The measures were meant to capture three qualities of the evaluation process. The first set of items related to the extent that students find it easy and enjoyable to participate in the process (satisfaction). The next set evaluated how honest students feel they can be and how anonymous they perceive the process to be (honesty; Nunamaker et al., 1997). A third set measured student perception of how accountable the instructor will feel in terms of responding to the feedback (accountability). The “evaluation of the evaluation” concluded with a simple open-ended question: “Comments?”

Pilot Study

Conducted with 110 students in three marketing courses, the pilot study served as an initial trial of the procedures used to administer the online collaborative evaluation and to refine the scales. The scales tested fairly well and all of the measures were retained. However, in the pilot study, on the “evaluation of the evaluation” the students rated the individual evaluation significantly higher than the collaborative evaluation on all three characteristics of satisfaction, honesty, and accountability. The final, open-ended question (i.e., “Comments?”) of the evaluation provided diagnostics as to why the collaborative evaluation was rated lower. Almost all the students’ comments were directed toward the collaborative evaluation. The comments could be described through three themes: (1) Enjoyable/fun, (2) Confusion, and (3) Irrelevant/not taken seriously. The third, and most prevalent, type were comments referring to other classmates using the medium to make jokes and off-topic responses, with a number of references to peers not taking the task seriously. Clearly, the confusion and irreverence of some students compromised the results from the collaborative evaluation. The themes, with examples of verbatim comments, are displayed in Table 2.

Table 2.

Students’ Open-Ended Evaluation of the Online Collaborative SET (Pilot Study).

Theme	Example verbatim comments
Enjoyable/fun	“Fun to complete online and see other responses. However, may encourage groupthink.”
Confusion	“It was a mess! Everyone was messing with other people’s comments.”
	“There were so many people typing at one time that I was unable to type anything. Also other users would delete comments and put useless things on it.”
	“Am I remembering incorrectly, or didn’t we also get comments about not being comfortable with using Google Docs in general?”
Irrelevant/not taken seriously	“I think it was hard to take it seriously while everyone else was typing jokes.”
	“Too many immature responses and had trouble taking it seriously.”

Note. SET = student evaluations of teaching.

Revised Online Collaborative Midterm Evaluation

The next semester the two forms of midterm evaluations were again administered in the same three marketing courses, this time to 140 students. The pilot study provided important feedback for how to improve the online collaborative evaluation for future use. Based on our observations and from student feedback from the pilot study, we introduced several changes for administering the collaborative evaluation during the next semester.

Prior to being given the evaluation form, students were provided a sample Google Docs document on which to practice typing synchronously with their peers to allow them to become comfortable using a collaborative document. For example, one of the instructors gave students a Google Docs survey that queried them on their favorite Halloween candy. Most students find Google Docs intuitive to use, so just a couple minutes of practice with the document raises their proficiency (Blaschke, 2014; Chu & Kennedy, 2011).

Students in the collaborative groups were specifically instructed that they should be respectful of each other’s responses (e.g., do not delete other students’ answers).

Group size for the collaborative evaluation was restricted to 4 to 8 students (the optimal size for focus groups; Kitzinger, 1995). To accomplish this, the students who had been randomly selected for the collaborative evaluation group in each class were further randomly subdivided into two or three groups in each class.

Both groups were given brief instructions to remind them that the purpose of the evaluation was to provide feedback that would lead to improved learning in the classroom.

Evaluation of the Revised Midterm Evaluation

Table 3 displays the results of the individual scale items that were measured on the “evaluation of the evaluation.” As Table 3 shows, the students evaluated the collaborative evaluation significantly higher on three measures: “easier to complete,” “enjoyed completing the evaluation,” and, importantly, “could provide useful feedback.” Medium to large effect sizes were found for “I enjoyed completing the evaluation” (Cohen’s d = .845) and “I could provide useful feedback” (Cohen’s d = .607). The individual evaluation did not score significantly higher on any item.

Table 3.

Students’ Perceptions of Individual Paper Evaluations Versus Online Collaborative Evaluations (N = 140)^a.

Factor, Cronbach’s α	Item	Individual mean	Collaborative mean	t	p	Cohen’s d
Satisfaction (α = .637)	I enjoyed completing the evaluation.	3.51**	4.39**	−4.98	.001	.845
	The evaluation was easy to complete.	4.36**	4.74**	−2.89	.005	.488
	The time it took to complete the evaluation was reasonable.	4.60	4.77	−1.82	.071	.308
Accountability (α = .730)	I could provide useful feedback.	3.75**	4.31**	−3.59	.001	.607
	The instructor will feel accountable for acting on the feedback from the evaluation.	4.03	4.29	−1.69	.094	.286
	The instructor is interested in my feedback.	4.64	4.76	−1.09	.276	.185
Honesty (α = .778)	I am confident that the responses to the evaluation are anonymous.	4.28	4.49	−1.33	.187	.224
	I provided honest responses to the evaluation.	4.65	4.57	0.698	.486	.118
	I felt free to express myself on this evaluation.	4.54	4.56	−0.12	.903	.020

Level of agreement mean rating with Likert-type scale where 1 = strongly disagree and 5 = strongly agree. Shaded mean is significantly higher.

p < .05. **p < .01.

Confirmatory factor analysis was conducted to assess the reliability and validity of the multi-item scales (Gerbing & Anderson, 1988) in the “evaluation of the evaluation.” The model fit was found acceptable on the basis of a battery of fit indexes (Hu & Bentler, 1999). The means of the items were calculated to provide composite measures for the scales. Comparison of the three factors (satisfaction, honesty, and accountability) was conducted for the two forms of evaluations using one-way multiple analysis of variance with covariation (MANCOVA), with course (sales management, marketing research, and principles of marketing) treated as a covariate. It was not necessary also to treat the professor as a covariate, since there was no overlap assignments among courses. MANCOVA was used to accommodate the correlations among the dependent measures as well as to control for the overall Type I error rate (Bray & Maxwell, 1985). For the overall model, the MANCOVA found a statistically significant difference in evaluation of the instruments on the combined dependent variables, F(3, 135) = 7.540, p < .001; Wilks’s Λ = .985; partial η² = .144, with the collaborative evaluation rated higher. Satisfaction was rated significantly higher for the online collaborative evaluation than for the individual paper evaluation, F(1, 137) = 21.840, p < .001. Honesty, F(1, 137) = .302, p = .583, and accountability, F(1, 137) = 3.093, p = .081, were not rated significantly different between the two types of evaluations.

Actionability of Comments

As a further comparison of the two types of evaluation, we evaluated the quality of the feedback that the students provided in response to the Stop/Start/Continue questions. A two-step approach was used to evaluate the quality of the comments that the students provided on both the individual evaluation and the collaborative evaluations. First, we counted those comments related to teacher performance, and second, we identified which of the related comments were also actionable. “Actionable” comments were defined as comments that could be used for improvement and generally fell in the areas of teaching, student support, and/or classroom materials. An example of an unrelated comment is “Obama is the reason America is turning into NORTH KOREA.” An example of a related commented is “I think you are doing everything a marketing teacher should be doing and teaching”; and an example of a related and actionable comment is “Maybe incorporate more concepts/vocab throughout our activities. More instruction on assignments would help.” The percentage of related and actionable comments were compared between individual and collaborative evaluations for each type of course. As Table 4 shows, 70.6% of the comments produced by the individual evaluation and 72.0% of comments produced by the collaborative evaluation were actionable. A chi-square analysis found no statistically significant differences between the types of evaluation forms.

Table 4.

Quality of Student Comments—Individual Versus Collaborative Evaluations (Percent Related and Percent Actionable)^a.

	Individual evaluation				Collaborative evaluation
Course	n	Total # of comments	Related comments (% of total)	Actionable comment (% of total)	n	Total # of comments	Related comment (% of total)	Actionable comments (% of total)
Marketing principles	28	92	95.65	65.22	27	99	83.84	61.62
Marketing research	15	53	90.57	77.36	16	53	100	92.45
Sales management	28	86	96.51	72.09	26	84	94.05	71.43
Total	71	231	94.81	70.56	69	236	91.01	72.03

No significant differences at p < .05 (chi-square analysis).

Closing the Loop

Most important for judging the ability of the midterm evaluations to provide formative feedback is whether the comments guided the instructor to make changes to improve learning in the classroom. We believe that the collaborative evaluations yielded different, but equally, or even more useful, types of comments than that from the individual evaluations. Importantly, when several students agreed with an initial comment, the content of the feedback gained credibility. The synergistic comments, in fact, led to substantive changes in teaching methods by the instructors in each of the courses—that is, “closing the loop” between having issues identified by students and taking measures to mitigate those issues. While these examples highlight the advantages of a collaborative evaluation for improving learning, they also, in a broader sense, illustrate the value of midterm evaluations of all kinds to allow instructors to respond to feedback while a course is still in session. To illustrate, we provide, as follows, an example of each of the main components of the Stop/Start/Continue evaluations.

STOP

Figure 1 displays the unedited comments of a team of eight students in a marketing research class in response to the first question of the collaborative, online document, “1. What should your instructor STOP doing (with respect to teaching)?” From this feedback, the instructor learned that, in her haste to return papers quickly without using too much class time, she “whips” and “flings” them at students. This type of comment would be unlikely to appear on an individual evaluation. The instructor was surprised to read those comments, and since then she has been trying to return papers to students more gently. The last set of responses to that question begins with one student stating, “Grades the SPSS projects pretty strictly,” and four other students agreeing with the response (i.e., five out of eight students), including one student who comments, “Yeah, marked down for some pretty insane things.” If the instructor had read that comment on an individual evaluation, she might have dismissed the comment as that of a disgruntled student who had received a low grade. With a sizeable number of students concurring, the instructor realized that she needed to explain her grading criteria more thoroughly.

Figure 1.

Unedited example of “STOP” results from online collaborative evaluation.

Using the example mentioned in the previous paragraph, the marketing research instructor took time in a class period following the evaluations to display the set of responses shown in Figure 1, and then to outline her grading policy for SPSS projects. Detailed explanation was given for how students lost points (e.g., not posting the test statistic value with the p value, not explaining exactly what the statistical “difference” or “association” actually was, etc.). As a result, the assignments (and the grades) improved. Being able to display graphically multiple students’ concerns with the instructor’s grading style provided the instructor with an opportunity to point out improvements that the students could make on their projects to lead to better grades.

START

A second example of “closing the loop” involved a section of a sales management class. Regarding what the instructor should “START” doing (with respect to exams), an extensive thread was generated by students revealing that many students felt unprepared for the analytics and Excel exam. This was both surprising and unexpected since the instructor had students complete a practice hands-on exam in the computer lab to prepare, and allowed students to use a “formula cheat sheet” during the actual exam. Additionally, following the practice hands-on exam activity, the instructor conducted a Q&A session in the classroom to provide final prep before the exam. During this face-to-face session, the general tone of students was one of readiness and confidence, particularly after the practice exam, which gave them a sense of what the exam would entail. However, reading the long thread on exam readiness was eye-opening due to the significant number of comments by students indicating a general lack of confidence surrounding the calculations and material covered, as shown in Figure 2.

Figure 2.

Unedited example of “START” results from online collaborative evaluation.

Because of this feedback, the instructor provided an exam debrief session during which the class worked through each calculation together, and then students were given the opportunity to earn a portion of their missed exam points by calculating the analytics using another data set to demonstrate their learning. In following semesters, the instructor further modified the course schedule to include an additional class session to cover analytics. During this session, students were assigned certain metrics that they had to teach to the entire class. This peer-teaching method, while a modest change, provided immediate and lasting effects since students seemed much more willing to ask questions of each other than they did of the instructor. This has resulted in improved average class scores on the analytics exam and improved satisfaction as noted in their midterm feedback.

CONTINUE

The final example, illustrating answers from the “Continue” component of the evaluation highlights both the strengths and weaknesses of a collaborative evaluation. The responses, shown in Figure 3, come from two groups from one section of a marketing research class: Group A, composed of eight students, and Group B, composed of seven students. All eight of the responses from Group A mentioned in some form the regular assignments/exercises that the instructor had been assigning the students to reinforce the statistical techniques that are taught in that class. Having an entire group of students agree and endorse the regular exercises was affirming to the instructor; correcting the exercises on a continual basis takes a great deal of time, but having an entire group of students agree that the exercises are valuable makes the effort spent grading them seem worthwhile. In Group B, while a couple of students also mentioned the value of the exercises, three of the students’ responses related to the announcements about upcoming university events (athletic, arts-related, and academic) that the instructor makes once a week. These examples from one class illustrate clearly the “groupthink” that can develop with any collaborative activity, since it seems unlikely that on an individual evaluation that all eight students would mention the course exercises in response to the “Continue” prompt, or even that three of seven students would endorse the announcements of university events. Still, what is lost in results that stem from independent thinking is gained in obtaining greater understanding of the dynamics of the social context in which students form judgements about their courses and instructors in a language that is closer to one students use when interacting with one another.

Figure 3.

Unedited example of “CONTINUE” results from online collaborative evaluation (Group A and Group B from one marketing research class).

Discussion

The purpose of this research was to present an innovative form of teaching evaluation that harnesses the collaborative strength of today’s marketing students to improve course feedback. In comparison to an individual paper evaluation, the online collaborative evaluation was rated significantly higher overall by students in three areas: enjoyment by the students in completing the evaluation, ease in completing the evaluation, and ability to provide high-quality feedback. In our own assessment of the ability of the evaluation to provide actionable results, we believe both individual and collaborative forms of midterm evaluations can be useful, with each of the types possessing distinct advantages. The traditional individual evaluation is superior at collecting responses with less concern of group influence (i.e., groupthink), although individual evaluations are by no means completely independent of influence (Aleamoni & Hexner, 1980). In addition, the individual evaluation can be organized and completed quickly, with a loss of only about 10 minutes of class time. The collaborative evaluation captures the advantages of group synergies, which may lead to the generation of better ideas and improved formative feedback, as we found during our study. It also increases student satisfaction according to our evaluations, which may lead to more thoughtful feedback. The collaborative evaluation brings the transparency of RateMyProfessors.com (and its sister online higher education instructor-rating sites) into the classroom, but since the entire class participates in providing feedback, the self-selection bias inherent in RateMyProfessor is mitigated (see Davison & Price, 2009). In addition, the transparent exposure of feedback may prevent complaining outside of the classroom via online vehicles such as blogs and social media sites, if students believe their evaluations are being taken seriously (Lala & Priluck, 2011). In other words, individual and collaborative evaluations provide different and insightful feedback, and the type of evaluation selected can be paired to the course.

While more time-efficient than Abbott et al.’s (1990) SGID method, the collaborative evaluation, like any team activity, requires detailed directions to implement—a lesson we learned in the pilot study. Several steps are necessary, including (1) students need to be organized into groups of four to eight, a size that is large enough to retain anonymity but small enough to allow for ample opportunity for interactivity; (2) students should be given an opportunity to practice working with interactive documents, such as Google Docs, prior to the activity; (3) students should be reminded of the importance of being respectful to their peers’ feedback; and (4) students should be informed that the purpose of the exercise is to provide useful feedback to improve the class (i.e., to increase their learning) and that their opinions are genuinely valued—a step that should always precede student evaluations. Because of this time requirement, as teaching methods and/or courses mature over subsequent terms, an instructor may find that administering a collaborative evaluation produces diminishing returns.

Still, in a class that emphasizes collaboration skills as tools for future success, the extra time that the collaborative evaluation takes can be time well spent. In addition to providing a new form of feedback for instructors, the evaluation process itself can also serve as an important learning opportunity. It reinforces to students that the instructor believes in the power of teamwork enough to harness it to improve his or her courses. Furthermore, the collaborative evaluation inherently promotes an ideology of students as partners in the learning process. To extend the notion of shared responsibility, a collaborative evaluation might also be a useful tool to evaluate the students’ contribution to learning, as well as the instructor’s (Bovill, 2011; Sierra, 2009). One scenario might involve the administration of a two-part Stop/Start/Continue evaluation, with the students first responding to the questions in reference to the instructor and then answering the same questions in reference to their peers, to provide feedback as to what both parties might do to increase the learning experience (Bovill, 2011).

While research has shown that midterm evaluations have distinct advantages in providing feedback for faculty (Abbott et al., 1990; Cohen, 1980; Overall & Marsh, 1979; Spencer & Schmelkin, 2002), some of the biggest criticisms of student evaluations of teaching evaluations are not assuaged by midterm evaluations, in general, or specifically by either of the methods of midterm evaluation we assessed. In particular, what student evaluations are measuring is student satisfaction and, at best, students’ perceptions of learning, rather than students’ actual learning (Clayson, 2009). In other words, conducting these evaluations may lead to higher student satisfaction, but there is no evidence that it will lead to higher student performance. Another major criticism of student evaluations—substantiated in research—that students conflate qualities of the instructors’ personalities with effective teaching in their evaluations (Ahmadi et al., 2001; Clayson & Sheffet, 2006)—is not addressed through the use of collaborative evaluations and might even be acerbated. It should also be noted that the timing of midterm evaluations means that students do not have the benefit of hindsight when providing feedback. A longer perspective by students can be important, since Clayson (2009) found that the relationship between rigor of course and evaluation of instructor flipped from negative to positive when evaluations were conducted after the students had completed the course. On the other hand, one persistent criticism of student evaluations—that evaluations are tainted by “halo effects” and extreme perceptions at both ends (Madden et al., 2010; Orsini, 1988)—may be mitigated by collaborative evaluations. Students who are exposed to the opinions of others with more positive or more negative viewpoints may moderate their assessments.

It should be noted that bias could have been introduced in this evaluation process. While the researchers took care to control for treatment bias, including providing similar degrees of instruction to both groups, bias may not have been completely eliminated. For example, the fact that the collaborative group was taken to the computer lab could have influenced their ratings. In addition, since students are not usually asked to work in teams to complete teaching evaluations, the collaborative group may have been more motivated to provide better feedback on the basis of novelty alone. Future research might compare an online individual evaluation to the online collaborative evaluation. Additional research might assess other types of collaborative evaluations, such as an adoption of the “think-pair-share” method (King, 1993) for midterm evaluations.

The public nature of collaborative evaluation is not without its risks. Students could potentially post comments that create public discomfiture for the instructor or for their fellow students, or are simply not constructive. For this reason, online collaborative evaluations might be best suited for smaller, upper level major classes, in which the instructor and students have established a rapport.

In future applications of the online collaborative evaluation, we plan to involve all the students in the class in participating in the activity, with more time invested in introducing best practices of collaboration. This lesson can be generalized to other classroom activities: When group activities are assigned in marketing classes, it may be as important to introduce tools for creating successful collaborations as it is to teach the content that will be practiced via the activities. In conclusion, in classrooms that emphasize teamwork as preparation for the workplace, the collaborative evaluation can provide actionable results while reinforcing the value of collaboration.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

AACSB International. (2013). Eligibility procedures and accreditation standards for business accreditation. Retrieved from http://www.aacsb.edu/accreditation/standards/2013-business/

Abbott

R. D.

Wulff

D. H.

Nyquist

J. D.

Ropp

V. A.

Hess

C. W.

(1990). Satisfaction with processes of collecting student opinions about instruction: The student perspective. Journal of Educational Psychology, 82, 201-206.

Ahmadi

Helms

M. M.

Raiszadeh

(2001). Business students’ perceptions of faculty evaluations. International Journal of Educational Management, 15(1), 12-22.

Aleamoni

L. M.

Hexner

P. Z.

(1980) A review of the research on student evaluation and a report on the effect of different sets of instruction on student course and instructor evaluation. Instructional Science, 9, 67-84.

Appleton-Knapp

S. L.

Krentler

K. A.

(2006). Measuring student expectations and their effects on satisfaction: The importance of managing student expectations. Journal of marketing education, 28, 254-264.

Arnould

Ducate

Kost

(2012). Collaboration or cooperation? Analyzing group dynamics and revision process in Wikis. CALICO Journal, 29, 431-448.

Aylesworth

(2008). Improving case discussion with an improv mind-set. Journal of Marketing Education, 30(2), 106-115.

Barkley

E. F.

Cross

K. P.

Major

C. H.

(2014). Collaborative learning techniques: A handbook for college faculty. San Francisco, CA: John Wiley.

Blaschke

L. M.

(2014). Using social media to engage and develop the online learner in self-determined learning. Research in Learning Technology, 22. Retrieved from http://www.researchinlearningtechnology.net/index.php/rlt/article/view/21635

10.

Bovill

(2011). Sharing responsibility for learning through formative evaluation: Moving to evaluation as learning. Practice and Evidence of Scholarship of Teaching and Learning in Higher Education, 6(2), 96-109.

11.

Braskamp

L. A.

Ory

J. C.

Pieper

D. M.

(1981). Student written comments: Dimensions of instructional quality. Journal of Educational Psychology, 73, 65-70.

12.

Bray

J. H.

Maxwell

S. E.

(1985). Multivariate analysis of variance (Vol. 54). Newbury Park, CA: Sage.

13.

Brennan

Williams

(2004). Collecting and using student feedback. New York, NY: Learning and Teaching Support Network.

14.

Chad

(2012). The use of team-based learning as an approach to increased engagement and learning for marketing students: A case study. Journal of Marketing Education, 34(2), 128-139.

15.

Chapman

K. J.

Meuter

M. L.

Toy

Wright

L. K.

(2010). Are student groups dysfunctional? Perspectives from both sides of the classroom. Journal of Marketing Education, 32(1), 39-49.

16.

Chen

Hoshower

L. B.

(2003). Student evaluation of teaching effectiveness: An assessment of student perception and motivation. Assessment & Evaluation in Higher Education, 28(1), 71-88.

17.

Chu

K. W. W.

Kennedy

D. M.

(2011). Using online collaborative tools for groups to co-construct knowledge. Online Information Review, 35, 581-597.

18.

Clayson

D. E.

(1999). Students’ evaluation of teaching effectiveness: Some implications of stability. Journal of Marketing Education, 21(1), 68-75.

19.

Clayson

D. E.

(2009). Student evaluations of teaching: Are they related to what students learn? A meta-analysis and review of the literature. Journal of Marketing Education, 31(1), 16-30.

20.

Clayson

D. E.

Sheffet

M. J.

(2006). Personality and the student evaluation of teaching. Journal of Marketing Education, 28(2), 149-160.

21.

Cohen

P. A.

(1980). Effectiveness of student-rating feedback for improving college instruction: A meta-analysis of findings. Research in Higher Education, 13, 321-341.

22.

Cronin

J. J.

(2009). Upgrading to Web 2.0: An experiential project to build a marketing wiki. Journal of Marketing Education, 31(1), 66-75.

23.

Davison

Price

(2009). How do we rate? An evaluation of online student evaluations. Assessment & Evaluation in Higher Education, 34(1), 51-65.

24.

Delong

T. J.

(2011). Three questions for effective feedback. Harvard Business Review. Retrieved from https://hbr.org/2011/08/three-questions-for-effective-feedback

25.

Dolnicar

Grün

(2009). Response style contamination of student evaluation data. Journal of Marketing Education, 31(2), 160-172.

26.

Dunegan

K. J.

Hrivnak

M. W.

(2003). Characteristics of mindless teaching evaluations and the moderating effects of image compatibility. Journal of Management Education, 27, 280-303.

27.

Duus

Cooray

(2014). Together we innovate cross-cultural teamwork through virtual platforms. Journal of Marketing Education, 36, 244-257.

28.

Elam

Stratton

Gibson

D. D.

(2007). Welcoming a new generation to college: The millennial students. Journal of College Admission, 195, 20-25.

29.

Ferrell

O. C.

Keig

D. L.

(2013). The marketing ethics course: Current state and future directions. Journal of Marketing Education, 35(2), 119-128.

30.

Finch

Nadeau

O’Reilly

(2012). The future of marketing education: A practitioner’s perspective. Journal of Marketing Education, 35(1), 54-67.

31.

Finelli

C. J.

Wright

M. C.

Pinder-Grover

(2010). Consulting the Delphi: A new idea for collecting student feedback through the two survey method (TSM). Journal of Faculty Development, 24(2), 25-33.

32.

Freeman

Greenacre

(2011). An examination of socially destructive behaviors in group work. Journal of Marketing Education, 33(1), 5-17.

33.

Gaillard

F. D.

Mitchell

S. P.

Kavota

(2011). Students, faculty, and administrators’ perception of students’ evaluations of faculty in higher education business schools. Journal of College Teaching & Learning, 3(8), 77-90.

34.

George

Cowan

(2004). A handbook of techniques for formative evaluation: Mapping the students’ learning experience. London, England: Routledge Falmer.

35.

Gerbing

D. W.

Anderson

J. C.

(1988). An updated paradigm for scale development incorporating unidimensionality and its assessment. Journal of Marketing Research, 25, 186-192.

36.

Granitz

Koernig

S. K.

(2011). Web 2.0 and marketing education: Explanations and experiential applications. Journal of Marketing Education, 33(1), 57-72.

37.

Grebennikov

Shah

(2013). Student voice: Using qualitative feedback from students to enhance their university experience. Teaching in Higher Education, 18, 606-618.

38.

Gruber

Lowrie

Brodowsky

G. H.

Reppel

A. E.

Voss

Chowdhury

I. N.

(2012). Investigating the influence of professor characteristics on student satisfaction and dissatisfaction: A comparative study. Journal of Marketing Education, 34(2), 165-178.

39.

Honeycutt

E. D.

Thelen

S. T.

Ford

J. B.

(2010). Evaluating and motivating faculty performance: Challenges for marketing chairs. Marketing Education Review, 20, 203-214.

40.

Hoon

Oliver

Szpakowska

Newton

(2014). Use of the “Stop, Start, Continue” method is associated with the production of constructive qualitative feedback by students in higher education. Assessment & Evaluation in Higher Education, 40, 755-767. doi:10.1080/02602938.2014.956282

41.

L. T.

Bentler

P. B.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1-55.

42.

Hunt

(2003). Does mid-semester feedback make a difference? Journal of Scholarship of Teaching and Learning, 3(2), 13-20.

43.

Huxham

Laybourn

Cairncross

Gray

Brown

Goldfinch

Earl

(2008). Collecting student feedback: A comparison of questionnaire and other methods. Assessment & Evaluation in Higher Education, 33, 675-686.

44.

Kabanoff

Richardson

Brown

(2003). Business graduates’ perceptions of the quality of their course: A view from their workplace. Journal of Institutional Research, 12(2), 1.

45.

Katzenbach

J. R.

Smith

D. K.

(1993). The discipline of teams. Boston, MA: Harvard Business Press.

46.

Kember

Leung

D. Y.

Kwan

(2002). Does the use of student feedback questionnaires improve the overall quality of teaching? Assessment & Evaluation in Higher Education, 27, 411-425.

47.

King

(1993). From sage on the stage to guide on the side. College Teaching, 41(1), 30-35.

48.

Kitzinger

(1995). Qualitative research. Introducing focus groups. British Medical Journal, 311, 299-302.

49.

Krishen

A. S.

(2013). Catch it if you can: How contagious motivation improves group projects and course satisfaction. Journal of Marketing Education, 35, 220-230.

50.

Kuh

G. D.

Cruce

T. M.

Shoup

Kinzie

Gonyea

R. M.

(2008). Unmasking the effects of student engagement on first-year college grades and persistence. Journal of Higher Education, 79, 540-563.

51.

Lala

Priluck

(2011). When students complain: An antecedent model of students’ intention to complain. Journal of Marketing Education, 33, 236-252.

52.

Laverie

D. A.

(2002). Improving teaching through improving evaluation: A guide to course portfolios. Journal of Marketing Education, 24(2), 104-113.

53.

Lenhart

Purcell

Smith

Zickuhr

(2010). Social media & mobile Internet use among teens and young adults. Retrieved from http://www.pewinternet.org/files/old-media/Files/Reports/2010/PIP_Social_Media_and_Young_Adults_Report_Final_with_toplines.pdf

54.

Loughry

M. L.

Ohland

M. W.

Woehr

D. J.

(2014). Assessing teamwork skills for assurance of learning using CATME team tools. Journal of Marketing Education, 36(1), 5-19.

55.

Madden

T. J.

Dillon

W. R.

Leak

R. L.

(2010). Students’ evaluation of teaching: Concerns of item diagnosticity. Journal of Marketing Education, 32, 264-274.

56.

Marsh

H. W.

(1987). Students’ evaluations of university teaching: Research findings, methodological issues, and direction for future research. International Journal of Education Research, 11, 253-388.

57.

Marsh

H. W.

Roche

L. A.

(1997). Making students’ evaluations of teaching effectiveness effective. The critical issues of validity, bias, and utility. American Psychologist, 52, 1187-1197.

58.

Meister

J. C.

Willyerd

(2010). Mentoring millennials. Harvard Business Review, 88(5), 68-72.

59.

Neu

W. A.

(2012). Unintended cognitive, affective, and behavioral consequences of group assignments. Journal of Marketing Education, 34(1), 67-81.

60.

Nunamaker

J. F.

Briggs

R. O.

Mittleman

D. D.

Vogel

D. R.

Balthazard

P. A.

(1997). Lessons from a dozen years of group support systems research: A discussion of lab and field findings. Journal of Management Information Systems, 13, 163-207.

61.

Orsini

J. L.

(1988). Halo effects in student evaluations of faculty: A case application. Journal of Marketing Education, 10(2), 38-45.

62.

Ory

J. C.

Braskamp

L. A.

Pieper

D. M.

(1980). Congruency of student evaluative information collected by three methods. Journal of Educational Psychology, 72, 181-185.

63.

Overall

J. U.

Marsh

H. W.

(1979). Midterm feedback from students: Its relationship to instructional improvement and students’ cognitive and affective outcomes. Journal of Educational Psychology, 71, 856-865.

64.

Penny

A. R.

(2003). Changing the agenda for research into students’ views about university teaching: Four shortcomings of SRT research. Teaching in Higher Education, 8, 399-411.

65.

Raines

(2003). Connecting generations: The sourcebook for a new generation. Seattle, WA: Crisp.

66.

Schlee

R. P.

Harich

K. R.

(2010). Knowledge and skill requirements for marketing jobs in the 21st century. Journal of Marketing Education, 32, 341-352.

67.

Sierra

J. J.

(2009). Shared responsibility and student learning: Ensuring a favorable educational experience. Journal of Marketing Education, 32(1), 104-111.

68.

Simpson

P. M.

Siguaw

J. A.

(2000). Student evaluations of teaching: An exploratory study of the faculty response. Journal of Marketing Education, 22, 199-213.

69.

Smimou

Dahl

D. W.

(2012). On the relationship between students’ perceptions of teaching quality, methods of assessment, and satisfaction. Journal of Education for Business, 87(1), 22-35.

70.

Snooks

M. K.

Neeley

S. E.

Revere

(2007). Midterm student feedback: Results of a pilot study. Journal of Excellence in College Teaching, 18(3), 55-73.

71.

Snooks

M. K.

Neeley

S. E.

Williamson

K. M.

(2004). From SGID and GIFT to BBQ: streamlining midterm student evaluations to improve teaching and learning. To Improve the Academy, 22, 110-124.

72.

Spencer

K. J.

Schmelkin

L. P.

(2002). Student perspectives on teaching and its evaluation. Assessment & Evaluation in Higher Education, 27, 397-409.

73.

Spooren

Brockx

Mortelmans

(2013). On the validity of student evaluation of teaching the state of the art. Review of Educational Research, 83, 598-642.

74.

Thompson

(2013). Creative conspiracy: The new rules of breakthrough collaboration. Boston, MA: Harvard Business Review Press.

75.

Wickramasinghe

S. R.

Timpson

W. M.

(2006). Mid-semester student feedback enhances student learning. Education for Chemical Engineers, 1, 126-133.

76.

Wulff

D. H.

Nyquist

J. D.

(1986). Using qualitative methods to generate data for instructional development. Retrieved from http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1118&context=podimproveacad.

77.

Yazici

H. J.

(2004). Student perceptions of collaborative learning on operations management classes. Journal of Education for Business, 80, 110-118.