Toward peer-review software and a rubric application in visual analytics classes: A case study

Abstract

Peer-review software is often used to allow authors to evaluate their code and its technical text content. In education, peer review is a common practice used not only to evaluate the quality of academic research but also to measure students’ engagement in the classroom. In visualization education, numerous researchers have addressed techniques of evaluation; however, such techniques rarely involve students in the evaluation and motivation process, and none have been discussed in the context of visual analytics courses. This study asks how peer-review software designed for student writing may be used in introductory courses to visual analytics at a large state university. To answer this question, this study conducted a questionnaire survey at the School of Information at the University of South Florida in 2016. Reporting on students’ experience during the course, the study reveals that the students supported the implementation of visual peer review as an engagement platform from which to understand visual analytics. Future studies need to address whether a new model and software based on visual analytics will better meet the needs of the students and instructors of visual analytics courses.

Keywords

Visual analytics visual peer review MyReviewers students experience

1. Introduction

Visualization has frequently been used to enhance the evidence of scientific discoveries, and this has led to it becoming an essential component for explaining knowledge in learning analytics (Siemens, 2013). In the last five years, visual analytics has become a mainstream research topic. Much of the literature on visual analytics has focused on building demonstration systems that show the potential of coupling visual interfaces with analytical technologies in different domains (see May et al., 2010; Novak & Cañas, 2006; Shamin et al., 2014). An excellent indicator of the field’s growth is the number of software applications that produce analytics together with visual representations (Bethel et al., 2016). Even the definition of the field consists of multiple disciplinary fields and the methodologies used to discover this knowledge (Keim et al., 2008). One of the challenges in the field, according to Keim et al. (2008, p. 156), is how we educate students. We have not found any in-depth discussion of how to engage students in an introductory visual analytics class. Peer review is a widespread and well-established means of evaluating educational progress, but such evaluation does not generally extend to visual analytics (e.g., 2D/3D, multidimensions, etc.) as an emerging and interdisciplinary field, which presents both a challenge and an opportunity to better understand the role of visualization in education (Friedman & Rosen, 2017). Our research aim is to identify the standard evaluation and values shaping the emerging field of visual analytics through a visualization peer-review process lens.

In the software industry, as in academia, peer review is an essential quality assurance mechanism. In higher education, student peer review was adopted early on in the field of writing composition. In many writing courses, students produce a substantial amount of writing that requires the instructor to spend significant time reviewing and commenting on student work. To reduce instructors’ workloads and increase students’ motivation, educators and developers have created student peer-review software that gives students opportunities to read and respond to one another’s writing (see Knight & Littleton, 2015; Moore, 2013; Friedman & Rosen, 2017). Such opportunities, when well planned, can help students improve their reading and writing skills and learn how to collaborate effectively (Moxley, 2013). One software application for this purpose is MyReviewers, a suite of cloud-based resources, tools, and workflows developed at the University of South Florida (USF) in the hope of facilitating peer-review practices and improving students’ writing skills. Since 2010, the University of South Florida has required all its incoming undergraduate students to take a basic writing composition course that uses MyReviewers.

Based on the established use of peer review software in composition classrooms, it seems likely that such software can be a useful tool for evaluating visualizations. Our research found no peer-review software currently in use that is specifically designed for visual analytics in the classroom. As a step toward assessing the need for such software, this study asks: Can peer review software designed for writing classrooms be used to evaluate student-produced visual analytics? More specifically:

Can a peer-review platform and rubric designed for writing instruction be useful in a visual analytics course at a large state university?

What does this rubric tell us about the need for a specific visual analytics rubric and software?

2. Background on learning analytics

The advancement of technology in higher education has drawn the attention of academics, researchers, and administrators to learning analytics (Elias, 2011). This interest is motivated by the need to better understand teaching, learning, “intelligent content”, and personalization and adaptation. While this domain is still in the early stages of research and implementation, many researchers and organizations (such as the Society for Learning Analytics Research and the International Educational Data Mining Society) have raised the issue of visualization in relation to the role of data analytics in education. Siemens (2013, pp. 1386–1389) outlines the challenges of learning analytics by calling for an increase in the scope of visualization capture so that the complexity of the learning process can be more accurately reflected in analysis. In 2009, with the emergence of social computing and learning analytics, Joe Moxley, a leading faculty member who promotes the student writing program at USF, in consultation with colleagues developed MyReviewers as an online platform for student writing. Using MyReviewers, students can review their peers’ work and provide comments on it. Numerous researchers in the field of students’ writing report that core premise of peer review is pedagogical in nature and that this premise is consistent with peer review’s roots in learning analytics. Knight and Littleton (2015) write that “our analytic techniques should not be limited by the ways in which educational dialogue has previously been operationalized; new analytic techniques afford new opportunities to theorize and reconsider what constitutes productive dialogue” (p. 193). To measure student writing analytically, MyReviewers was designed to provide writing document mark-up tools that give instructors and students access to feedback (Moxley, 2013).

MyReviewers also provides an assessment ecology machine with five key roles: students, course instructors, the institution assessment team, accreditors and stakeholders, and institution leadership. Each role is an important part of MyReviewers’ ecosystem and plays a role in its ability to adapt and implement standard writing analytic rubrics. This system has never before been tested for visualization or visual analytics.

In the visualization education community, the literature often has one of two foci: designing visualizations or teaching the analysis of visualizations. In the more mature area of the design of visualizations, the teacher trains students in using visualization methodologies (1D, 2D, 3D, open source vs. existing software applications, etc.) and programming languages to create visualizations. Many researchers have also discussed the topic of visualization evaluation (Elmqvist & Ebert, 2012; May et al., 2010; Novas et al., 2013). While many of the techniques and algorithms involved can be objectively measured, none of those discussed offers a practical rubric for examining students’ peer review comments in the visualization classroom (Friedman & Rosen, 2017). To examine students’ peer review comments in visual analytics classes at USF, this study used Moxley’s rubric for writing to measure students’ comments at one of the largest public universities in Florida.

3. This study’s visual peer-review practices

In 2013, the School of Information at the University of South Florida hired several new faculties to build a visual and data science curriculum. This effort took place in the context of a larger university-led effort to improve students’ knowledge of statistics, data mining, and visualization. It was expected that the new faculty would focus on teaching visualization through techniques for creating images, diagrams, and animations to communicate a message using different platforms and programming languages. This effort involved creating visual courses using online platforms. The course “Introduction to Visual Analytics” focused on structuring data and analytics to build visualizations that require and capture the data analysis. The course covered a wide range of areas, including data management, statistics and analytics, space and time, visual analytics tools, human perception, and visual evaluation. The assignments gave students hands-on experience in applying statistical theoretical concepts to design and develop projects and taught them to convert the results of statistical analysis into visualizations. The first project required students to collect their own data to produce an infographic using open source R. With this project, students learned how to combine three variables into a visual graph. An example of one such student project is shown in Fig. 1 student assignment. The student who created this example used data on public school expenditures collected found in Arel-Bundock’s datasets (2018). The student applied ggplot2 to illustrate the changes in four variables: education, income, age, and urban population.

Figure 1.

Student assignment in the class.

This progress in the visual curriculum raises the additional questions of how to evaluate student work and how to engage students in the classroom. These questions have never been addressed in the literature. To address them, and in response to a lack of visual analytics education applications, this study used MyReviewers and Moxley’s rubric in a visual analytics class.

The majority of our students had become familiar with MyReviewers during introductory writing courses at USF. For our course, we revised MyReviewers’ interface and modified the system to accept the visual analytics work that the students performed in the class. The students had to write their own programming code and statistical analysis to present their visual work. Figure 2 captures the interface we used.

Figure 2.

Visual peer review via MyReviewers.

Figure 3.

Moxley’s rubric, cited in Friedman and Rosen (2017).

To provide numeric evaluations for the student peer-review comments, we used Moxley’s rubric. When using MyReviewers, instructors choose between two versions of Moxley’s rubric: (1) the numeric rubric, which requires students to score the rubric criteria (focus, evidence, organization, style, and format) on a five-point scale; or (2) the discuss rubric, which requires students to write textual comments regarding these criteria rather than give numeric scores. Three of the rubric criteria – focus, organization, and style – contain two subcategories: style/basics and critical thinking. The style/basics subcategory focuses on local concerns – i.e., specific language conventions (grammar, mechanics, punctuation). In contrast, the critical thinking subcategory identifies global rhetorical concerns. Figure 3 is a reproduction of the Moxley Rubric that this study used.

4. Results

We evaluated the students’ reactions with a survey administered at the end of the course. This survey was primarily summative, but it had a formative component in that it elicited student feedback that we planned to use to help us refine the course in subsequent iterations. Participation was voluntary, and 12 of the students enrolled in the course filled out the form. Of these students, 25% were women and 75% were men. The average age of the students was 25. No other demographic information was collected. The survey was conducted independently of the USF standardized student evaluation. It used open-ended questions and a Likert-scale questionnaire. We used frequency analysis to calculate student responses. The questionnaire assessed student attitudes towards visual peer review, towards the statistics education used to produce the visualizations, and towards the visualizations of their own output. In the section on the visual peer review application, the students were asked to numerically evaluate MyReviewers based on five criteria: appreciation, access, productivity, abilities, and flexibility.

Each scale addresses the students’ reaction to the MyReviewers platform. Our results indicated that visual peer review had strong support from students. Eighty-five percent of the students reported positive attitudes towards MyReviewers as an evaluation platform. In terms of productivity, 72% rated it positively, and 63% rated it positively for flexibility. Only 5% rated it positively in terms of access. Figure 4 shows the score for each category based on the students’ evaluations.

Another important thing to note is students’ answers to open-ended questions. Many of the students referred to technical difficulties they experienced while attempting to display their visual work on MyReviewers. One of the students expressed disappointment in the process, stating:

“I was disappointed using MyReviewers, where I had to build my code on GitHub and then share a link inside MyReviewers. This made it hard for me to ask for feedback after I had to correct my code based on the comments I received and report the link again, sometime multiple times.”

We also examined the most frequent terms students used in their answers on the questionnaire. We found that “applications” (79%), “design patterns” (73%), and “multivariate” (68%) were the top three terms used. Table 1 lists the ten words most often used by the students.

Table 1
Most frequently used terms in student answers to the questionnaire

Most user term
1. Applications 79%
2. Design patterns 73%
3. Multivariate 68%
4. Time series design 32%
5. Scatterplot analysis 22%
6. Table design 12%
7. Colour scheme 9%
8. Data visualization patterns 6.5%
9. Graph label title 5.1%
10. Graphic organizer 4.4%

Figure 4.

Summary of the study questionnaire about MyReviewers.

Overall, this study has presented a writing framework for evaluating student visual analytics based on visual peer review. The majority of our students found that peer review practices contributed to their understanding and appreciation of visual peer review. We also found that the students found it difficult to use MyReviewers as a platform for conducting visual peer review.

5. The study challenges

During this research, we encountered some unexpected challenges. The first and most important challenge was a lack of resources for teaching visual analytics at the university level. One reason for this is the complexity of visual analytics as a subject: the instructor needs to cover not only basic statistics and data mining principles, together with visualization principles, but also support the student experience. Second, we found a lack of visualization evaluation in the classroom. While many of the techniques and algorithms involved in visual analytics can be objectively measured, to date we have not found a practical rubric for examining students’ peer-review comments in the visualization classroom. Third, engaging with technology is challenging in any classroom, but it is becoming even more complex as institutions offer a greater number of courses in an online environment. This is an issue that visual analytics teachers will need to confront. Overall, visual analytics via visual peer review is not established, and future research is needed to create a rubric specifically for visual analytics.

6. Reflecting on the study

Assessment is at the heart of any higher education environment. Student assessment takes many forms in higher education, and one of these forms is peer review. Peer review is established as an essential component of many professional practices, such as the scholarly publication process. The fundamental principle of peer review is that experts in a given domain appraise the professional performance, creativity, or quality of work produced by others in their area of competence.

Visual analytics education is still a work in progress, and as we enter a new age of online environments, instructors face more pressure than ever before. To investigate peer review in a visual education class, this study used the visual peer-review rubric proposed by Moxley (2013) to teach writing. Our results, based on a questionnaire, indicate that students became more engaged in the sense that they wrote longer comments as the semester developed. The results of this study suggest that using an established model as the core foundation of visual peer review offered an effective and flexible way to identify student engagement in visualization education. It also offered a measure that is transparent to both the student and the instructor. Future studies should address a new model based on visual grammar that will be more useful for visual peer review.

References

Arel-Bundock,

(2018). R datasets. Retrieved from https://vincentarelbundock.github.io/Rdatasets/datasets.html.

Bacchelli,

, & Bird,

(2013, May). Expectations, outcomes, and challenges of modern code review. In Proceedings of the 2013 International Conference on Software Engineering, pp. 712–721. IEEE Press.

Bethel,

E.W.

Bauer,

Abbasi,

Ahrens,

Childs,

Geveci,

, & Whitlock,

(2016, June). In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms, a State-of-the-art (STAR) Report. In Computer Graphics Forum 35(3). Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States).

Elias,

(2011). Learning analytics. Learning, 1–22.

Elmqvist,

, & Ebert,

D.S.

(2012). Leveraging multidisciplinarity in a visual analytics graduate course. IEEE Computer Graphics and Applications, 32(3), 84–87.

Eppler,

M.J.

(2006). A comparison between concept maps, mind maps, conceptual diagrams, and visual metaphors as complementary tools for knowledge construction and sharing. Information Visualization, 5(3), 202–210.

Friedman,

, & Rosen,

(2017). Leveraging peer review in visualization education: A proposal for a new mode. IEEE Pedagogy of Data Visualization Workshop, Phoenix, AZ.

Keim,

Andrienko,

Fekete,

J.D.

Görg,

Kohlhammer,

, & Melançon,

(2008). Visual analytics: Definition, process, and challenges. In Information Visualization, pp. 154–175. Springer, Berlin, Heidelberg.

Knight,

, & Littleton,

(2015). Discourse-centric learning analytics: mapping the terrain. Journal of Learning Analytics, 2(1), 185–209.

10.

May,

Hanrahan,

Keim,

D.A.

Shneiderman,

, & Card,

(2010, October). The state of visual analytics: views on what visual analytics is and where it is going. IEEE: Symposium on Visual Analytics Science and Technology, pp. 257–259. IEEE.

11.

Moore,

(2013) The theory of transactional distance. M.G. Moore (Ed.), Handbook of distance education (3rd ed.) Routledge, 65–85.

12.

Moxley,

(2013). Everything is illuminated: What big data can tell us about teacher commentary. Assessing Writing, 18(4), 241–256.

13.

Novais,

R.L.

Torres,

Mendes,

T.S.

Mendonça,

, & Zazworka,

(2013). Software evolution visualization: A systematic mapping study. Information and Software Technology, 55(11), 1860–1883.

14.

Novak,

J.D.

, & Cañas,

A.J.

(2006). The origins of the concept mapping tool and the continuing evolution of the tool. Information Visualization, 5(3), 175–184.

15.

Siemens,

(2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–1400.

16.

Shamim,

Balakrishnan,

, & Tahir,

(2014). Evaluation of opinion visualization techniques. Information Visualization, 14(4), 339–358.