Using cross-course peer grading with content expertise,anonymity,and perceived justice

Peer grading

Peer grading as a learning and social interaction has a foundation in social development theory (Vygotsky, 1978). Prior research has used the terms peer review, peer feedback, peer grading, and peer assessment interchangeably (Wilson et al., 2015). Panadero and Alqassab (2019) define peer assessment as “an arrangement in which individuals consider the amount, level, value, worth, quality, or success of the products or outcomes of learning of peers of similar status” (p. 1254). Peer grading falls under the umbrella of peer assessment with a focus on considering the performance of others, based on an established rubric (Nicol et al., 2014), who may or may not have a similar level of academic experience (Adachi et al., 2018).

Peer grading helps the instructor by creating efficiencies and saving time (Haddadi et al., 2018), promotes student ownership in the course (Ritchie, 2016), and helps better prepare students for work product critique in a post-graduation business environment (Trede & Jackson, 2021). One of the benefits of peer grading is having students experience critiquing the work of other students before having to make an evaluative presentation in a professional setting where the stakes are much higher (Jayathilake & Huxham, 2021; Onyia, 2014). Students will be placed in virtual teams early in their careers. Thus opportunities to engage in virtual experiences, including assessment, will enhance their preparedness for the workplace (Loucks & Ozogul, 2020). The results of the meta-analysis conducted by Zheng et al. (2020) showed that students who participated in technology-based peer grading outperformed those who did not experience peer assessment on their graded activities. Furthermore, the difference in performance was greater for written assignments than presentations.

Peer grading in groups increases the visibility of each student’s grading actions, thus encouraging students to be more accountable for their contribution and further develop interpersonal communication skills (Mayfield & Tombaugh, 2019). Peer group grading is more efficient because individual assessments are more cumbersome for instructors to orchestrate and require students to grade more assignments (Carless, 2020). However, a concern is that only one team member does most of the work while the credit is equally shared with non-contributing team members. Lack of effort may be misconstrued as social loafing when genuine difficulty in completing the grading task may be the source of apparent apathy (Freeman & Greenacre, 2011). These situations can be mitigated by keeping the grader team size small, regardless of the size of the teams submitting assignments, and requiring self-reported involvement (Broadbent, 2018).

Prior research on CCPG is limited. Kwan and Leung (1996) found disagreement in grades assigned by tutors and peer groups. Third-year students evaluated the research proposals of second-year students in the study by Sloman and Thompson (2010). Student feedback demonstrated that both groups were comfortable with the in-person cross-year assessment and found it beneficial. Ernst et al. (2015) used cross-institutional individual peer review citing the benefits of increased objectivity and critical evaluation skills by the graders. However, online CCPG using teams of senior students in a capstone course and novice students in an introductory course has yet to be examined.

Content expertise

Reciprocal peer reviews (i.e., students grading each other’s work) offer the benefit of learning from giving feedback to others (Cho & Cho, 2011). However, this advantage is most appropriate for developmental assignments. When it comes to grades that count in the course, students are leery of expertise possessed by their in-course classmates (Klein, 2018; Mulder et al., 2014). Variation in academic experience and achievement can be compensated by utilizing mature students, or fourth-year students who major in the content area, as graders to provide high-quality feedback (Gielen & De Wever, 2015). CCPG incorporates both elements in that seniors in a capstone course have higher proficiency and academic experience in the discipline than their novice student counterparts.

Students develop assessment judgments through practice and instruction in peer grading (Panadero, 2016). Furthermore, skills learned from peer grading can be readily transferred to a professional setting (Klucevsek, 2016). Team assessment can further develop critical review with the perspectives contributed by other knowledgeable colleagues. Part of improving one’s assessment skills is developing confidence and believing that one is capable and competent to do so (Vanderhoven et al., 2015). Therefore, it is posited that:

H₁: Grades assigned by senior student teams will not differ from instructor-assigned grades.

H₂: Senior students will experience assessment confidence from the CCPG.

Anonymity

Social identity theory (Tajfel, 1978) proposes that group members develop self-conception in group membership (Hogg, 2020). The idea is that group members sense an affiliation with each other that may create a potentially competitive situation. However, anonymous peer group grading encourages candid scoring for the graders by reducing negative social effects such as fear of disapproval, compromising friendships (Panadero et al., 2013), and fear of retaliation for giving a low grade (Li et al., 2020). Moreover, students report being more objective and critical in an anonymous grading situation while instructors find anonymous peer grading has higher validity (De Grez et al., 2012). The assurance of anonymity can be heightened when the peer graders are in a different section of a course, or in a different course altogether. Thus, anonymity adds to the overall perception of impartial assessment as a student satisfier (Broadbent, 2018).

A criticism of anonymous peer grading is that it does not reflect actual professional practice where one has to offer self-identifying critique (Panadero, 2016). Although anonymity in peer grading limits students by not having interactive feedback (Rotsaert et al., 2017), it does foster a safe and more objective setting. A study by Wagar and Carroll (2012) found students prefer anonymous evaluations and find them the most fair. Furthermore, a study by Wetsch (2009) found students were motivated to do their best when they knew their work would be evaluated by peers, even when the assessment was conducted anonymously. Thus, having anonymity in peer group grading may encourage greater effort and reduce concerns regarding peer group grading that may hinder student performance. The combination of such can foster perceptions of effectiveness in the process as a component of the course. Therefore, it is posited that:

H₃: Novice students will perceive the CCPG as an effective course component.

H₄: Senior students will perceive the CCPG as an effective course component.

Perceived justice

According to social exchange theory (Homans, 1961), there are three dimensions of justice including distributive (Adams, 1965), procedural (Thibaut & Walker, 1975), and interactional (Bies & Moag, 1986). Distributive justice refers to the fairness of outcome distributions or whether the expectation of a gain is proportional to the investment (Orsingher et al., 2010). Procedural justice has been described as the fairness of the process used to determine outcome distributions (Colquitt et al., 2001). Interactional justice reflects the degree to which people are treated with dignity and respect (Greenberg, 1993).

Perceived justice is compromised when students feel taken advantage of or do not sense they have anything to gain. Novice students may feel they put in more work in their written assignments than the instructor when grading is delegated to other students. However, the benefits of peer grading include learning from the peer grading process and increasing future academic performance on similar assignments (Li et al., 2020). Prior research shows that students are also concerned about being fairly assessed by their peers (Sridharan et al., 2019). However, a study by Luo et al. (2014) found peer grading by student groups provided increased assurance of perceived procedural justice because more than one student was involved in the evaluation. Interactional justice is less of a consideration when peer graders primarily utilize an established rubric and provide limited written feedback, as is the case in this study. Therefore, it is posited that:

H₅: Novice students will experience distributive justice using CCPG.

H₆: Novice students will experience procedural justice using CCPG.

Sample

CCPG was administered in two sections of an introductory course at a public regional university. One section of the introductory class was offered in an asynchronous online format which did not feature scheduled class meetings. This format relied heavily on recorded lectures, discussion boards, email, and chat rooms for interactions. The other section was offered in a synchronous online format and featured scheduled class meetings that were hosted on Zoom (videoconferencing). Both sections were taught by the same instructor. Student graders were enrolled in one of two sections of an online capstone course in the same discipline. The capstone course sections were taught by different instructors, however both sections employed the synchronous online format and were hosted on Zoom.

Measures

Novice students were given a survey near the end of the semester. The three-item scale from Syrdal et al. (2021) was utilized to capture perceived effectiveness of CCPG as a course component. A five-item scale adapted from Colquitt (2001) measured distributive justice. Two items from the scale developed by Davidow (2014) were applicable to CCPG and thus were utilized to measure procedural justice. Senior students were surveyed at the beginning and at the end of the semester. The pre-CCPG survey completed by the senior students inquired about assessment confidence using a four-item scale adapted from Klucevsek (2016). The same scale was replicated in the post-CCPG survey along with the three-item scale from Syrdal et al. (2021) to measure perceived effectiveness of the CCPG as a course component. Responses to all measures were collected using a scale from 1 (strongly disagree) to 7 (strongly agree). See Table 2 for scale items. The survey also collected demographic data.

OPEN IN VIEWER

Table 2.

Cross-course peer grading measures.

Source	Construct	Items
(Colquitt, 2001)	Distributive justice	Our peer grade reflects the effort we put into our work. Our peer grade is appropriate for the work that we contributed.
		Our peer grade reflects what we contributed to the assignment.
		Our peer grade is justified, given our performance.
		I believe the peer grading outcome was fair.
(Davidow, 2014)	Procedural	I believe the peer grading process was fair.
	justice	I am satisfied with the peer grading procedure.
(Klucevsek, 2016)	Assessment confidence	I feel confident in my ability to accurately assess writing quality.
		I feel confident in my ability to accurately assess marketing content quality.
		I feel confident in my ability to accurately assess business analysis quality.
		I feel prepared to critique the work of colleagues in a professional setting.
(Syrdal et al., 2021)	Perceived effectiveness	I recommend using peer grading in the course in the future.
		I rate using peer grading in the course as effective.
		I highly rate using peer grading in the course.

Exploratory factor analysis was utilized to assess the convergent validity of items for the dimensions on the survey administered to novice students. The analysis utilized varimax rotation based on eigenvalues greater than 1.0. Three dimensions emerged, explaining 90.5% of the variance. Cronbach alpha scores for all dimensions exceeded the .70 benchmark recommended by Nunnally (1978), thus establishing internal validity. The Cronbach alphas for the distributive justice scale (α = .96), procedural justice scale (α = .92), and perceived effectiveness scale (α = .95) indicate high internal consistency reliability.

Exploratory factor analysis was also utilized to assess the convergent validity of items for the dimensions on the post-CCPG survey administered to senior students. The analysis utilized varimax rotation based on eigenvalues greater than 1.0. Two dimensions emerged (i.e., perceived effectiveness and assessment confidence), explaining 76.8% of the variance. Cronbach alpha scores for all dimensions exceeded the .70 benchmark recommended by Nunnally (1978), thus establishing internal validity. The Cronbach alphas for the perceived effectiveness and assessment confidence scales were 0.96 and 0.79, respectively, revealing high internal consistency reliability.

Finally, the grades that senior teams awarded to the novice student teams on the marketing mix analysis assignment were recorded, along with the grades awarded by the instructor, and two teaching assistants. A third instructor who did not teach either of the focal classes collected the data via Qualtrics. Further, at the start of the survey students were instructed that they would be directed to a separate survey to input their name to receive credit for survey completion. This was conducted to delink the survey results from the reports that were provided to the instructors to award credit. Survey items were randomized within categories and responses to the survey instrument were anonymous, as recommended by Larson (2019).