Studying the Usability of an Intervention to Promote Teachers’ Use of Robotics in STEM Education

Abstract

We developed RoboSTEM, a portal for open educational resources for the use of robotics in teaching and learning, to help teachers learn how to design and implement lessons using robotics. In this article, the design and functionality of RoboSTEM and the theoretical foundations behind its design are described. Two usability testing studies are also presented. Participants were 13 pre- and in-service teachers. Results from both quantitative and qualitative data analyses show that (a) usability survey ratings were positive overall and improved after revisions from the first usability study to the second; (b) data from participants’ screen activities and interviews were aligned with the survey results; (c) high levels of behavioral, cognitive, and emotional engagement in RoboSTEM were observed; and (d) the positive impact of RoboSTEM on example-based learning was acknowledged among teachers. Applications of RoboSTEM as well as future research and development directions are discussed.

Keywords

robotics STEM education teacher professional learning usability testing web-based community

Introduction

Science, technology, engineering, and mathematics (STEM) education has been acknowledged by researchers, educators, and policy makers as crucial to preparing students to be productive citizens and workers in the U.S. economy (Tseng, Chang, Lou, & Chen, 2013). However, many Americans lack the required knowledge and skills for STEM positions (National Research Council, 2011). In recent years, there has been an effort to improve STEM education by applying various strategies such as inquiry-based, problem-based, and project-based learning (e.g., Duran, Höft, Lawson, Medjahed, & Orady, 2014; Lou, Shih, Diez, & Tseng, 2011; Tseng et al., 2013). However, individual STEM disciplines are often covered separately in school, if at all, although interdisciplinary, authentic learning is rightly emphasized in STEM education (Becker & Park, 2011; Hansen & Gonzalez, 2014). In this article, a Web portal called RoboSTEM, for Open Educational Resources (OERs) for the use of robotics in teaching and learning is introduced and its usability studies are reported. RoboSTEM aims to enhance STEM teaching through robotics activities that promote interdisciplinary, authentic learning (Bers, 2008; Mataric, Koening, & Feil-Seifer, 2007; Nugent, Barker, Grandgenett, & Adamchuk, 2010; Osborne, Thomas, & Forbes, 2010).

Educational Robotics

Manipulatives have been used to help children understand abstract concepts (Bers & Portsmore, 2005). Robotic kits are newer manipulatives that students can use to design, develop, and program interactive artifacts (Bers & Portsmore, 2005; Ortiz, Bos, & Smith, 2015). Educational robotics can be defined as “the use of robotics as a learning tool” (Eguchi, 2012, p. 3). The most popular robotics platform is LEGO Mindstorms (Eguchi, 2012). There has been an effort to use robotics technology for STEM education (Benitti, 2012; Danahy et al., 2014; Toh, Causo, Tzuo, Chen, & Yeo, 2016). Robotics competitions are designed to draw students to fun and practical ways to learn and implement STEM concepts and principles, and have been deployed widely in high school and university-level education (Altin & Pedaste, 2013; Sklar, Parsons, & Stone, 2004). Robotics-based curricula are often integrated into summer camps and afterschool programs (Barker, Nugent, & Grandgenett, 2014). Even kindergarteners have benefitted from learning with robotics (Kazakoff, Sullivan, & Bers, 2013). In a variety of K-12 contexts where robotics was used, students exhibited improvements in learning of mathematics (e.g., Lindh & Holgersson, 2007), science (e.g., Karahoca, Karahoca, & Uzunboylub, 2011; Williams, Ma, Prejean, Ford, & Lai, 2007), and engineering (e.g., Larkins, Moore, Rubbo, & Covington, 2013).

In addition, using robotics in education has good potential for helping students learn 21st-century skills such as creative thinking, problem-solving, collaboration, and communications skills that are practiced through robotics (Albo-Canals et al., 2013; Alimisis, 2013; Beer, Chiel, & Drushel, 1999; Bers, 2008; Mitnik, Recabarren, Nussbaum, & Soto, 2009). Student engagement is also a critical issue in K-12 education; a study showed that students’ dropout rates are explained by their disengagement in school (Henry, Knight, & Thornberry, 2012). Even students who tend not to be attracted to academic tasks tend to be engaged when robotics activities are used in class (Baker, 2011). This can be attributed to motivation, interest, self-confidence, and positive emotions that are often experienced among students through learning with robotics (Goldman, Eguchi, & Sklar, 2004; McGill, 2012).

Need for RoboSTEM

The benefits of robotics cannot be achieved without effective teacher preparation (Pittí, Curto, Moreno, & Rodríguez, 2013). However, traditional teacher education programs have not adequately prepared teachers to implement educational robotics (Nugent et al., 2010). For example, no science course is required in 70% of elementary teacher preparation programs (Greenberg, McKee, & Walsh, 2013). Robotics technology provides a great tool for teachers to facilitate active learning while at the same time addressing STEM content (Mataric et al., 2007; Perritt, 2010). Using robotics can enhance teacher knowledge and practice, which in turn promotes interdisciplinary work habits, learning by design, collaborative learning, and student engagement (Bers, 2008).

Impacting teachers’ STEM teaching has the potential to prepare students for STEM careers. By empowering teachers to create and utilize age-appropriate and standards-aligned robotics activities, students’ STEM engagement and achievement can be enhanced (Alimisis, 2013). Efforts have been made to improve teacher knowledge and practice on the use of robotics in teaching (Kim, Kim, Yuan, Hill, Doshi and Thai, 2015; Bers & Portsmore, 2005; Jaipal-Jamani & Angeli, 2016; Kay, Moss, Engelman, & McKlin, 2014; Ortiz et al., 2015; Perritt, 2010; Sullivan & Moriarty, 2009; Tocháček & Lapeš, 2012). However, active networking among teachers is not present although such networking can lead to “open educational and technological products and practices (curriculum and resources)” (Alimisis, 2013, p. 69).

A web portal supporting teachers’ sharing resources and having dialogues on teaching will be beneficial for teachers in several ways. First, from a theoretical perspective, social constructivism emphasizes the important influence of social interactions on knowledge construction (Vygotsky, 1978). Teachers will learn from interacting with other teachers. Second, teachers can share their expertise, access teaching resources, have a discussion with others, and engage in reflective practice (Loucks-Horsley & Matsumoto, 1999). Of special note, online discussions foster a higher level of reflective thinking among teachers than does face-to-face communication (Hawkes, 2001). Third, teachers can form a learning community consisting of a group of individuals dedicated to information sharing, support, and knowledge construction (Yuan and Kim, 2014) and benefit from teleapprenticeship available in a learning community without having to meet in a particular place at the same time (Levin & Waugh, 1998).

Given the lack of a networking and the importance of web-based support for sharing and collaboration among teachers, we developed RoboSTEM for sharing OERs to help teachers learn how to design and implement lessons using robotics. We designed RoboSTEM to (a) have greater functionalities than TeacherTube (e.g., no ads), (b) allow for a more focused environment for educational robotics than YouTube, and (c) be less likely to be blocked at schools. In RoboSTEM, not only teacher ownership and autonomy are allowed but also teacher efficacy can be incrementally built from vicarious experiences (e.g., from seeing an art teacher using robotics in his or her classroom; Bandura, 1997; Tschannen-Moran & Hoy, 2001).

Research Question

In this article, the design and functionality of RoboSTEM and theoretical foundations behind its design are described. Two usability testing studies are also presented. The primary goal of the usability testing was to improve the usability of RoboSTEM. The central research question was as follows: How usable and useful do teachers consider RoboSTEM? To address the research question, (a) specific characteristics of RoboSTEM such as navigation, interactivity, content, and learnability and (b) teachers engagement in RoboSTEM were investigated. Detailed descriptions of the usability testing are provided in the Method section.

Rationales for RoboSTEM Design: Theoretical Foundations

Example-Based Learning

One of the theoretical foundations for designing RoboSTEM is a theory of example-based learning (Renkl, 2014). On RoboSTEM, teachers view lesson plans, videos, robot programming files, and reflections to learn what other teachers do with robots and how that works for students. Three approaches of example-based learning, “worked examples, observational learning, and analogical reasoning”, are effective in acquiring basic cognitive skills and problem-solving. (Renkl, 2014, p. 1). Teaching with robots requires not only basic cognitive skills (e.g., robot assembly) but also problem-solving to incorporate robotics activities in teaching for student STEM engagement and learning.

Although we intend to apply all three example-based learning approaches to the RoboSTEM development, currently, the explicit use of the example-based learning theory involves only the observational learning approach. Observational learning is defined as “learning by observing other persons’ behavior (i.e., models)” (Renkl, 2014, p. 8), which fits the aim of RoboSTEM to help teachers learn from each other by seeing others’ STEM lessons using robots. There is extensive research reporting the efficacy of observational learning (e.g., Bjerrum, Hilberg, Gog, Charles, & Eika, 2013) including observations via electronic media. For example, in a study that utilized example-based learning theory, models for an effective collaboration presented in a video format led to positive effects on knowledge acquisition as well as on perceived competence and usefulness of content, interest, and enjoyment (Rummel, Spada, & Hauser, 2009). On RoboSTEM, teachers observe models through videos and other electronic formats. As Bandura (1986) noted in his sociocognitive learning theory, observing multiple models increases the possibility of learning from at least one model chosen by the observer. RoboSTEM provides users with multiple models (e.g., multiple teachers presenting lesson plans and videos). The fact that the models are peer teachers (i.e., peer models) is a critical motivating factor for teacher learning (e.g., Schunk & Hanson, 1985).

Engagement

Engagement theories also provide foundations for the RoboSTEM design. Engagement refers to a person’s involvement in a task such as learning and teaching (Reschly & Christenson, 2012). Learning does not occur without engagement (Skinner, Kindermann, & Furrer, 2008). Engagement is a meta-construct that includes behavioral, cognitive, and emotional engagement (Fredricks, Blumenfeld, & Paris, 2004). Behavioral engagement refers to participation in tasks. Time on task and taking initiatives are indicators of behavioral engagement. Cognitive engagement indicates psychological invest in tasks, including valuing tasks and being strategic while working on tasks. Emotional engagement usually means affective reactions to academic tasks, including enjoyment, boredom, happiness, and the like (Fredricks et al., 2004).

We focus on two forms of engagement in this research. One is teacher engagement in RoboSTEM to learn robotics for teaching STEM. The other is student engagement that teachers need to learn about for teaching STEM via robotics. Based on engagement theories, RoboSTEM introduces techniques and examples to engage students in STEM education through robotics. RoboSTEM is designed to be a tool for teachers to connect theory to practice of STEM teaching, and vice versa.

Teacher engagement is promoted in RoboSTEM mainly through the promotion of perceptions of belonging and valuing—crucial variables in engagement models (e.g., Finn & Zimmer, 2012). Belonging refers to a person’s sense of being included and an important member in the community of learning (and community of practice as teacher learning in the present research aims at giving them the tools to enhance their participation in the community of practice); valuing refers to a person’s recognition of the value of the community as a social and learning tool (Finn & Zimmer, 2012; Voelkl, 1997). To enhance belonging, RoboSTEM is designed in ways that teachers’ (a) shared goals are pursued, (b) social goals are satisfied, and (c) shared standards are built together (Belland, Kim, & Hannafin, 2013). For example, first, the goal of RoboSTEM, explicitly described on the website, is aligned with that of the teachers who intend to learn to design and implement STEM lessons using robotics and the goal is shared among teachers. Second, the features for creating, joining, and participating in a group (e.g., robotics newbies) on RoboSTEM is to aid teachers to achieve their social goals, such as social networking with each other by sharing their learning experience, getting reactions to it, and reflecting on it together. Last, endorsing other teachers’ lessons as likes and bookmarking them as favorites lead to creating shared standards for the practices that are valuable to the teachers on RoboSTEM. For example, the lessons in which Common Core Standards are specifically listed and linked to robotics activities are viewed as helpful by the shared standards co-constructed by the teachers.

Valuing, which contributes to teacher engagement in RoboSTEM, means in the present research that teachers perceive RoboSTEM as a worthwhile and useful tool for their social activities and learning. Teachers’ perceived value increases when the website content interests them and gives something tangible that is relevant and useful to them (Belland et al., 2013). RoboSTEM prompts teachers to select aspects of STEM education, robotics, and student engagement that are relevant to them. For example, when teachers browse others’ lessons, they choose a grade level, subjects, a robot platform, and more to see the lessons that connect to their current needs and interests. Also when RoboSTEM introduces student STEM engagement theories, practical examples are given to intrigue teachers’ interests in reading more about the theory and viewing related lessons. Thus, the lessons and videos that were chosen to view “provide explanatory rationales for relevance to current and future life” (Belland et al., 2013, p. 252). Also, through seeing others’ comments on posts and crafting their own comments, teachers reflect on and articulate what is worthwhile and useful to them, which is about “attainment value” of the posts (Belland et al., 2013, p. 253).

As briefly described earlier, RoboSTEM illustrates how lessons using robots can promote the factors that are critical in students’ STEM engagement such as student interest, success expectancy, autonomy, and enjoyment. Teachers search, view, and save lessons that promote each factor, which is designed to help teachers learn about how to engage students in STEM learning with robotics. More features of RoboSTEM are described in the Method section.

Method

RoboSTEM Architecture

Although user-created content is intended to sustain RoboSTEM in later stages, the RoboSTEM team initiated its content development. Fundamentals for beginners were created on RoboSTEM that include tutorials on robot construction and programming. Lesson examples integrating robotics into K-12 classrooms were posted as well. Lesson examples were from a teacher education course that covered a robotics unit for STEM teaching.

The interface of RoboSTEM consists of the main menus for Home, Toolbox, Community, Support, My Account, and Upload (Figure 1). Based on findings from the Phase 1 usability testing study (see Phase 1 Study in the Method section for a detailed description), the Upload menu was deleted and the submenus of the Toolbox were revised before the Phase 2 usability testing study. As the revisions are reported in the Results section, the initial design (i.e., Phase 1) is described here. The Home page presents the purpose of the RoboSTEM website and lists featured media, recent videos, lessons, and groups. The Toolbox page contains (a) lessons for STEM learning and (b) theories for STEM engagement. Both offer lessons that teachers can search, collect, and use. Lessons are organized per activity style (e.g., such as challenge-based, inquiry-based, project-based, example-based activities), educational standards such as the common core, grade levels, and subject areas. Theories illustrate how lessons using robots can promote student interest, mastery goals, success expectancy, autonomy, volition, and enjoyment, all of which are critical in STEM teaching and learning (Belland et al., 2013; Kim & Bennekin, 2013). The Community page allows teachers to create and participate in groups and forums, search and view user channels, and search and keep the calendar for events on educational robot and STEM education. On the Support page, teachers can contact the RoboSTEM team for a technical help or have a question or a suggestion. On the My Account page, teachers can (a) edit their profile; (b) see the media that they bookmarked and pinned as their favorites; (c) see and edit the lessons, videos, and other documents that they created; and (d) see the groups that they participate in and created. The Upload page allows teachers to upload various types of materials for a lesson plan to RoboSTEM, including word documents, PDFs, pictures, videos, and programming files.

Figure 1.

A screenshot of RoboSTEM.

Phase 1 Study

Participants

Participants were eight pre-service teachers and were recruited from a large university in the southeastern United States. The average age of participants was 24.00 (SD = 3.024) and 87.5% were women (n = 7). Of them, 75% were White (n = 5), 12.5% were Black (n = 1), 12.5% were other (n = 1), and 12.5% did not wish to answer the question about race (n = 1). One was an undergraduate student and seven were graduate students. Half of the participants were majoring in Early Childhood Education. Three were majoring in Science Education, Social Studies, and Communication Sciences and Disorders, respectively. One had not decided on her area of study and teaching yet. Four participants planned to become elementary school teachers; two planned to become middle school or high school teachers; one planned to become a high school teacher; and one did not specify her plan for a grade level to teach.

Procedures

Participant recruitment strategies varied according to their instruction medium. For those enrolled in online courses, participants were recruited via e-mail; for those in face-to-face courses, one of the researchers visited classrooms to summarize the study procedures and ask for volunteers. In both cases, only courses offered in the College of Education were included to recruit pre-service and in-service teachers. Students who agreed to participate in the study were e-mailed individually to schedule a convenient time for them to meet for a usability testing session. The session was held in a research laboratory room in the College of Education building and participants chose to use either a Macintosh or a Windows laptop. During the session, each participant was asked first to explore RoboSTEM and complete a set of tasks for 15 to 20 minutes. Their face, voice, and computer screen during the usability session were recorded using Camtasia, a screen capture tool (Silva, 2012). Then, they responded to a survey containing demographic questions and questions assessing the perceived usability of RoboSTEM for 5 to 10 minutes. Last, they were interviewed for 20 to 30 minutes about their experiences with and perceptions about RoboSTEM (Fry & Rich, 2011; Preece, Rogers, & Sharp, 2002) and to elicit areas for improvement (Granić & Ćukušić, 2011). The list of tasks, survey questions, and interview protocol are described in the Data Collection section. Participants were paid $20 for their participation.

Data collection

Screen, face, and voice recording

Participants’ screens were recorded along with their faces and think-aloud audio during the usability testing sessions. Participants were given a list of tasks to complete on RoboSTEM. These included uploading a lesson plan including a video file and selecting identifiers for the uploaded lesson (e.g., grade, subject, standards). The concurrent think-aloud protocol (Ericsson, 2003) was not used for the usability testing to be as least intrusive as possible (Karahasanović, Hinkel, SjØberg, & Thomas, 2009). That is, participants were not asked to think out loud during their RoboSTEM exploration and task completion but the voice of those who naturally talked to themselves was recorded.

Usability survey

Earlier studies testing usability focused on a wide range of website characteristics; however, we limited our study to the aspects that are directly relevant to the purpose and functionality of RoboSTEM. The aspects are called scales in the present study. The scales include navigation (Hinchliffe & Mummery, 2008; World Wide Web Consortium, 2008), visual clarity (Zaharias & Poylymenakou, 2009), content (Chhetri, 2011; Zaharias & Poylymenakou, 2009), interactivity (Zaharias & Poylymenakou, 2009), attractiveness (Hinchliffe & Mummery, 2008), user support (Hinchliffe & Mummery, 2008; Zaharias & Poylymenakou, 2009), helpfulness (Mentes & Turan, 2012), learnability (Fry & Rich, 2011; Mentes & Turan, 2012), technical functionality (Zaharias & Poylymenakou, 2009), exchangeability (Poll, 2007; Wade & Lyng, 1999), nonredundancy (Wade & Lyng, 1999), and general (Chhetri, 2011). Items under each scale were either selected from earlier usability test studies or modified. Table 1 lists the scales, the meaning and rationale for the inclusion of each scale, and survey items for each scale. Most survey questions asked participants to select on a 5-point scale the extent to which they agreed with the statements (1 = strongly disagree, 5 = strongly agree). Questions in the Content and General categories (see Table 1 for the categories) were open-ended questions. Specifically, the open-ended questions included: “What do you think is the purpose of the site?” “What do you think is the intended audience?” “Was there something missing you were expecting to see? If there was, what is it?” “Was anything too well hidden? If there was, what is it?” “Name your three favorite things about the site and your three least favorite things.”

Table 1.

An Overview of Survey Scales and Items.

Scale	Description	Items	References
Navigation	Navigation is included to examine how effectively a user could navigate around a website for task completion.	My current location was always clear. There is always a clear link to the Home page. All major parts of the site are accessible from the Home page. The search function is easy to use. The site structure is simple. I can easily find what I’m looking for. The buttons have clear labels. All links in the site are working. Standard colors are used for links. I was able to search for what I wanted to find. The site allowed me to easily move whenever I wanted to go. I could expect what’s in menus at first glance. Icons and images fully represented what they were supposed to do.	Chhetri (2011); Hinchliffe and Mummery (2008); Matera, Rizzo, and Carughi (2006); World Wide Web Consortium (2008)
Visual clarity	Visual clarity indicates how clearly a website displays itself and determines the effectiveness of the website.	The layout is clear. Texts are easy to read (both font style and size). There is adequate text-to-background contrast.	Ardito et al. (2006); Chhetri (2011); Hinchliffe and Mummery (2008); World Wide Web Consortium (2008); Zaharias and Poylymenakou (2009)
Content	Content refers to the extent to which a website contains what it is supposed to have. The items in this scale range from the appropriateness of terminology, and vocabulary to the perceived purpose of the website to determine whether the website is well designed to consider target audiences.	The major headings are clear and descriptive. What do you think the purpose of the site is? Who do you think the intended audience is? The use of terminology and vocabulary were appropriate for intended users.	Chhetri (2011); Zaharias and Poylymenakou (2009)
Interactivity	Interactivity refers to the extent to which users are allowed to interact with the website.	The site contained factors to attract my attention. The site well showed what’s new or important on visible places.	Zaharias and Poylymenakou (2009)
Attractiveness	Attractiveness of website not only affects users’ perception of information and satisfaction, but also determines the success of a website.	The site reflected recent trends in terms of its design. The site’s design well represented what it supposed to provide. The site has unique attributes compared to other sites for a similar purpose.	Mentes and Turan, 2012
User support	User support can play an important role in guiding users when problems occur.	I was able to get adequate supports when having trouble with using the site.	Hinchliffe and Mummery (2008); Wade and Lyng (1999); World Wide Web Consortium and Others (2008); Zaharias and Poylymenakou (2009)
Helpfulness	Helpfulness measures the extent to which users obtain useful information and knowledge.	The site seemed to fully function as intended. I got enough information I needed.	Mentes and Turan (2012)
Learnability	Learnability refers to how easy it is for users to become familiar with the website.	I could easily get familiar with functions and structures of the site.	Hinchliffe and Mummery (2008); Mentes and Turan (2012)
Technical functionality	Technical functionality refers to how easy it is for users to complete tasks without technical problems.	There was no technical problem while surfing on the site. There was no delay when trying to load pages.	Zaharias and Poylymenakou (2009)
Exchangeability	Exchangeability measures how easy it is for users to post comments and download and upload files.	I could easily upload and download postings and files. I could easily leave comments on postings.	Poll (2007); Wade and Lyng (1999)
Nonredundancy	Nonredundancy is a factor that determines whether a website contains redundant information that distracts users.	Menus, categories, information, or pages didn’t overlap with others.	Wade and Lyng (1999)
General	General impression about a website refers to whether users like or dislike the website.	Was there something missing you were expecting to see? If there was, what is it? Was anything too well hidden? If so, what is it? Name your three favorite things about the site, and your three least favorite.	Chhetri (2011)

Interview

A semi-structured interview was conducted to acquire in-depth understanding of how usable and useful teachers considered RoboSTEM. Three researchers participated in the interviews. One of them led all the interviews and one of the other two joined the interviews, which limited the number of interviewers to two. The interview protocol was constructed to investigate participants’ perceived usability of RoboSTEM and their behavioral, cognitive, and emotional engagement in RoboSTEM. Table 2 presents an overview of the interview protocol listing the investigation aims of each question and its prior research basis.

Table 2.

An Overview of the Semistructured Interview Protocol.

Interview question	Investigating	Constructed based on
▪ What do you think about each page of the site? ▪ How would you summarize your overall experience with the site? ▪ What do you think could have been done differently to make the site better (more fun, more engaging)? ▪ Is there anything that we have not asked that you want to share about your experience with the site?	Perceived usability	Chhetri (2011)
▪ Did you try something else other than the list of tasks we gave to you?	Behavioral engagement	Fredrick, Blumenfeld, and Paris (2004); Reeve, Jang, Carrell, Jeon, and Barch (2004)
▪ What benefits do you think there would be for the intended audience? ▪ Do you think the site could be useful to you? Or other (pre- and in-service) teachers? ▪ What kinds of distractions did you experience? ▪ How would you use what you learned from the site for your own teaching? ▪ What more would you like to see from the site to learn about teaching with robotics and STEM engagement? ▪ When did you feel uncertain or unsure about something while exploring the site? ▪ When were things difficult?	Cognitive engagement	Finn and Zimmer (2012); Fredricks et al. (2004); Ge and Land (2004); Reeve et al. (2004); Zimmerman, (1990)
▪ When you first learned about the purpose of the site, what were your initial reactions? What did you expect? ▪ How did you feel, typically, while you were exploring the site? ▪ What did you like and dislike about the site?	Emotional engagement	Fredricks et al. (2004)

Data analysis

Descriptive statistics were used to analyze the survey data. Missing data were replaced with values computed with the Expectation–maximization method (Schafer, 1997). The screen, face, and voice recordings were transcribed verbatim in terms of what participants did and said during their RoboSTEM exploration. Interviews were also transcribed verbatim. We analyzed interviews using constant comparative analysis (Glaser, 1965). Codes were developed by three researchers initially based on the theoretical foundations of the RoboSTEM design as well as the literature on usability testing. Based on comparisons and discussions among the three researchers, some codes were deleted or merged and additional codes were created as part of refining the coding scheme. Two researchers analyzed two transcripts using the final codes, compared their analyses, and discussed discrepancies. They then analyzed another two interviews independently. Cohen’s κ was .797, which is considered satisfactory (Landis & Koch, 1977).

Phase 2 Study

Participants

Among the in-service teachers, one was an elementary teacher; one taught family and consumer science to sixth through eighth graders; and one taught agriculture to sixth through eighth graders. The average age of the participants was 30.8 (SD = 14.08). All of the participants were women. Of them, 80% were White (n = 4) and 20% did not wish to reveal their ethnicity (n = 1); 80% were graduate students (n = 4) and 20% were undergraduate students (n = 1). Among the in-service teachers, one was an elementary teacher, one taught family and consumer science to sixth through eighth graders, and one taught agriculture to sixth through eighth graders. One pre-service teacher planned to teach health and physical education to secondary students and the other planned to teach pre-K and kindergarten students.

Procedures

The Phase 2 study procedures, data collection, and data analysis were the same as that of the Phase 1 study except that (a) the revised RoboSTEM was tested and (b) the list of tasks given to participants included also the tasks of modifying their uploaded lesson plan and searching lessons and videos that met specific criteria such as subject, grade level, and educational standards. The revisions made in RoboSTEM for Phase 2 are as follows:

The menu option, Upload, was removed. Users uploaded their lesson plans using the Create Lesson Plan submenu under Toolbox. We changed “Upload” to “Create Lesson Plan” because participants’ responses to open-ended questions show that they did not know where to create a lesson.

Recent videos were listed on the Home page along with the number of views and likes, and the length as the indication of length gives users more information about the videos and the numbers of views and likes suggest the trustworthiness of the content (Kietzmann, Hermkens, McCarthy, & Silvestre, 2011).

Users could search for lesson plans based on STEM engagement theories, educational levels, subjects, activities, lesson standards, unit tags, and robotic platforms. The research team had several discussions and we wanted users to be able to easily search for lesson plans using multiple keywords.

The website logo and login fields were moved to make room for repositioning the search fields. The research team had a discussion regarding how to improve RoboSTEM. We thought it would be easier for users to navigate the website if the search box is in the top right corner where people expect to find it.

Minor changes were made in various components such as thumbnail views, file display, calendar function, lesson category selection styles, and others. These changes were made based on participants’ responses to the open-ended questions. They indicated that they encountered difficulties using the calendar and labeling lesson plans with multiple categories.

Results

Phase 1 Study

Screen, face, and voice recording

Participants exhibited a high level of behavioral engagement by taking initiatives while performing the usability test tasks. They performed additional tasks on RoboSTEM that were not listed on the usability testing task list. For example, they checked what other users had done on the website and read the lesson plans or posters created by other users. Participants’ emotional engagement was revealed through expressions of confusion first (e.g., it took a while for a few participants to find where to create a lesson) and relief, joy, and excitement later.

Usability survey

Descriptive statistics

The descriptive statistics including means and standard deviations of all scales in the usability survey are presented in Table 3. In all scales—ranging from 1 (strongly disagree) to 5 (strongly agree)—students rated the target website aspect as positive. With the exception of two scales, content (3.94) and attractiveness (3.50), all scales were rated between agree and strongly agree. Exchangeability, rated the highest (4.75), indicates that participants acknowledged the ease of sharing information and having discussions. User support was rated the second highest (4.63), which means that participants thought adequate support was available when needed.

Table 3.

Means and Standard Deviations for All Scales in Usability Survey.

	Phase 1		Phase 2
	Mean	SD	Mean	SD
Navigation	4.27	.46	4.17	.63
Visual clarity	4.36	.63	4.80	.18
Content	3.94	.68	4.60	.42
Interactivity	4.06	.86	4.00	.94
Attractiveness	3.50	.71	4.20	.77
User support	4.63	.74	3.80	.84
Helpfulness	4.00	1.00	4.50	.50
Learnability	4.25	.89	4.40	.55
Technical functionality	4.31	.96	4.60	.42
Exchangeability	4.75	.38	4.90	.22
Nonredundancy	4.25	1.04	4.60	.55
Total	4.21	.76	4.40	.55

Note. Possible range of each category is 1 (strongly disagree) to 5 (strongly agree).

Open-ended questions

Participants’ responses to the five open-ended questions are presented in Table 4. First, all participants answered that the purpose of the site was for teachers to share lesson plans. Second, five participants answered that the intended audience was teachers but three participants answered that the intended audience was both teachers and students. Third, three participants stated that nothing was missing; to three participants, it was unclear what should be uploaded under the main menu item Upload and two participants wanted to see more colors. Fourth, one participant answered that all designs were intuitive but four participants stated that it did not take too long for them to find where to create lesson plans but the location was not in an obvious place. Last, the favorite things reported included: (a) easy navigation and use (n = 6); (b) sharing lessons (n = 5); (c) the community activities such as the forum and joining groups (n = 4); (d) the home page such as featured media and links to other pages and features (n = 4); (e) design attractiveness such as layout, appearance, and logo (n = 3); (f) fast functioning of uploads (n = 2); (g) toolbox (n = 2); and (h) user profile page (n = 2). The least favorite things reported included: (a) unclear navigation for creating lesson plans (n = 3); (b) difficulties encountered such as event time change, profile picture upload, multiple selections, and captcha error messages (n = 3); (c) design of colors (n = 3) and font size (n = 2); (d) content such as insufficient lesson plans posted and a lack of descriptions about the groups created (n = 2); and (e) organization such as overall structure and menu (n = 2).

Table 4.

A Summary of Participants’ Responses to Open-Ended Questions.

Open-ended question	Phase 1 response (N = 8)^a	Phase 2 response (N = 5)^a
What do you think the purpose of the site is?	• For teachers to share lesson plans (n = 8)^b	• For teachers to share resources (n = 5)
Who do you think the intended audience is?	• Teachers (n = 5) • Teachers and students (n = 3)	• Teachers (n = 5)
Was there something missing you were expecting to see? If there was, what is it?	• Nothing was missing (n = 3) • It was unclear what should be uploaded under the main menu item upload (n = 3) • More colors on the site would be good (n = 2)	• Nothing was missing (n = 3) • Could not figure out how to upload a lesson plan (n = 1) • Wish there were easier ways to search for things (n = 1)
Was anything too well hidden? If there was, what is it?	• All designs were intuitive (n = 1) • The location of where to create lesson plans was not in an obvious place (n = 4) • It was difficult finding the lesson one uploaded (n = 1) • It was a little difficult finding the suggestion box forum (n = 1) • Did not know what suggestion box was (n = 1) • It was hard to find where to change the date/time for an event in the calendar (n = 1)	• Nothing was hidden (n = 2) • Could not figure out how to upload a lesson plan or group lesson plans (n = 1) • Could not figure out how to edit a lesson (n = 1) • Could not find a lesson supported by a particular theory (n = 1)
Name our three favorite things about the site, and your three least favorite things about the site.	Three favorite things: • Easy navigation and use (n = 6) • Sharing lessons (n = 5) • The community activities such as the forum and joining groups (n = 4) • The home page such as featured media and links to other pages and features (n = 4) • Design attractiveness such as layout, appearance, and logo (n = 3) • Fast functioning of uploads (n = 2) • Toolbox (n = 2) • User profile page (n = 2) Three least favorite things: • Unclear navigation for creating lesson plans (n = 3) • Difficulties encountered such as event time change, profile picture upload, multiple selections, and captcha error messages (n = 3) • Design of color (n = 3) • Design of font size (n = 2) • Insufficient lesson plans posted and a lack of descriptions about the groups created (n = 2) • Organization such as overall structure and menu (n = 2)	Three favorite things: • Searching for lesson plans based on various criteria (n = 3) • Sharing lesson plans (n = 1) • Discussion forums (n = 1) • Fast uploading of videos (n = 1) • Lesson plans with videos (n = 1) • Featured media (n = 1) • The easy accessibility of the lessons (n = 1) • Simplicity such as layout (n = 1), simple process to request support (n = 1), and simplicity of everything (n = 1) • Group features (n = 1) • Various options available on the site (n = 1) • Emphasis on robotics and its connection to other subject areas (n = 1) Three least favorite things: • Only one participant reported three, two listed one, one participant listed two, and one participant reported none. • Search tools (n = 2) • The catch phrase (n = 1) • Calendar (n = 1) • Forum appearance (n = 1) • Design of color (n = 1) • Editing lesson plan was difficult to find (n = 1)

N refers to the total number of participants.

n refers to the number of participants providing a particular response.

Interviews

Usability

Participants commented on learnability positively; they indicated that it was easy to learn how to use the site because the site was intuitive and user friendly. For example, Alyssa stated, “after playing with it, you could figure it out.” However, there were a few negative comments on the main menu—difficulty knowing what was on each page just by looking at the menu. Still, all participants reported their overall experience was positive. With regard to navigation, Jane stated, “It’s pretty easy to find things.” and “I like again how simple it is.” In terms of content, three participants considered STEM engagement theories especially useful components on the site, for example, Nancy said:

I think all of this [theory] is definitely ways to increase their [students’] engagement. From the little bit of what I’ve learned, students like to constantly have something to do, makes them think they don’t want to do the worksheet, or listen to teacher talk for 40 minutes. They don’t capture as much as if they do something on their own. They get involved in the actual tasks. I would definitely use most of these theories [as a teacher]. Even if you didn’t want to use somebody else’s lesson, you could still come on here for the theories, and everything else, and get more ideas, to make yours better.

A comment from Holly on content emphasizes that videos can be for teachers’ view but also for students’ view:

I was looking at the videos. I think that’s cool. That [videos] could be a tool in class for showing relevant things, or clips, just like function, I think could be directly, not just for me [as a teacher] to create lessons, but actually show to students, if it is relevant to what we were planning on doing.

Behavioral engagement

Participants stated that they did additional tasks with RoboSTEM that were not asked by the researchers. Joan stated, “I looked at the different tabs. I went to the theories for STEM. I looked at that.” Jane mentioned, “I went to the other people’s profiles, like the different things you could do, like interact with those people.” Taking initiatives evidences behavioral engagement (Fredricks et al., 2004).

Cognitive engagement

Participants reported a high level of cognitive engagement in RoboSTEM, evidenced by “valuing learning” from the site (Fredricks et al., 2004, p. 64). They recognized that the site was of value. In fact, value was most frequently coded among all the nodes. All comments on value were about sharing lesson plans and relevant resources and using them for their teaching. For example, Jane said, “I think it’s always helpful to see what other people are doing in their classrooms, their students, even just to get feedback from what people think you are doing right, what you could change, from what you share.” Harper highlighted the usefulness of RoboSTEM for teachers with little knowledge about educational robotics by stating, “I think it will be helpful for teachers especially who don’t know much about robotics because they have access to lesson plans and examples of the ways teaching robotics can be done and incorporate it into classrooms.” In addition, Harper specified the instrumental value of theories for STEM engagement as follows, “I think that it could be useful to find lessons based on theories.”

Emotional engagement

Three kinds of emotional engagement were reported: (a) liking, (b) interest, and (c) confusion. All said they liked RoboSTEM. Some pointed out specific aspects of the site that they liked. For example, Nancy said, “I like that from the home page, there is the featured media, there’s post, recent videos, and recent lesson plans.” They also showed their interests in what other teachers do. However, two participants got confused at first with specific functions or terminologies used for the site, due to unfamiliarity with them yet.

Phase 2 Study

Screen, face, and voice recording

In general, a high level of behavioral engagement was observed. Participants took initiative by performing a few more tasks beyond the usability testing tasks such as further exploration of RoboSTEM. With regard to cognitive engagement (rehearsal), participants tried out multiple files to explore how to create a lesson. With regard to emotional engagement, two participants were confused about searching for a lesson to which a STEM engagement theory was applied but exhibited joy from “an aha” moment or relief when confusion was resolved.

Usability survey

Descriptive statistics

Table 3 lists the means and standard deviations of all categories. Exchangeability was rated the highest (4.90), which means that participants recognized the high potential of the website as a dynamic space. Visual clarity was rated the second highest (4.80), which shows that participants thought the website displayed its information extremely clearly. User support (3.80) and interactivity (4.00) were rated the lowest (see Tables 1 and 3 for details).

Open-ended questions

Table 4 shows participants’ responses to open-ended questions. All participants answered the purpose of the site was for teachers to share resources. They also stated that the intended audience was teachers. Most participants said there was nothing missing or hidden on the site but two participants said they were lost for a little bit. Participants’ favorite things included: (a) searching for lesson plans based on various criteria (n = 3); (b) sharing lesson plans (n = 1); (c) discussion forums (n = 1); (d) fast uploading of videos (n = 1) and lesson plans with videos (n = 1); (e) featured media (n = 1); (g) the easy accessibility of the lessons (n = 1); (h) simplicity such as layout (n = 1), simple process to request support (n = 1), and simplicity of everything (n = 1); (i) group features (n = 1); (j) various options available on the site (n = 1); and (k) emphasis on robotics and its connection to other subject areas (n = 1). As to the things they disliked, only one participant reported three, two listed one, one participant listed two, and one participant reported none. Their least favorites included: (a) search tools (n = 2), (b) the catch phrase (n = 1), (c) calendar (n = 1), (d) forum appearance (n = 1), (e) design of color (n = 1), and (f) the editing lesson plan, which was difficult to find (n = 1).

Interview

Usability

All participants reported positive experience overall. Of special note, for example, Leah, who described herself as a non tech-savvy said, “For me to do it [using a website] for the first time and have a good experience is actually something.” They also mentioned the learnability of the site positively. For example, Alice stated, “There were a few times when I may have wanted a little more guidance where to find things, but I was usually able to figure out where they were.” Most participants commented on navigation positively, as illustrated in Leah’s following statement “I enjoyed it. It was easy to go from one place to next.” However, one participant, Bella, reported a negative experience with finding the page to modify her lesson. As in Phase 1, theories and videos were regarded as useful content; for example, Alice highlighted, “I like how it tells you what they are, in case you don’t know, kind of what you are looking for, and now the examples, too, what kind of theories.” Only one participant, Leah, suggested improvement, as follows, “I like it [the site], but maybe more color. A little bit, but not too much.”

Behavioral engagement

All participants took initiatives while exploring RoboSTEM, which is evidence of behavioral engagement. For example, Alice carefully examined classification information for a lesson, as illustrated in the following comment, “I just wanted to think about … there is information thing for each category like the State Performance Standards. I just wanted exact CC [Common Cores] points. I wanted to put that in there.”

Cognitive engagement

All participants indicated a high level of cognitive engagement in using RoboSTEM. As in Phase 1, value was most frequently coded among all the nodes. Communities of like-minded teachers created via RoboSTEM were valued. For example, Vivian mentioned,

Just because it’s rare for me to find websites in which me and other teachers are sharing instead of just being provided with some materials or some information. Many are the commercial way, the lesson plans, something like that, or the materials you are using. It was interesting because it is a community of teachers, not something you are offering to me.

Nancy emphasized that not only teaching with robots but also teaching without robots could be improved through RoboSTEM because the site is helpful in understanding how student engage and learn, by stating, “I think there is a lot of stuff here that I can use as a teacher. Even if not with robots, there is still a good bit on here just for student learning.” Bella mentioned that the site would be useful for all teachers including non-STEM teachers as follows: “I think I would be able to use it as a non-STEM teacher, I will still be uploading, pull it in.”

Emotional engagement

As in Phase 1, (a) liking, (b) interest, and (c) confusion were reported. All liked RoboSTEM. Those who referred to specific features of the site favored theories for STEM engagement, groups, sharing lessons, and simple design, as shown in Alice’s comment: “I like that you could search lessons based on theory or [other] category. I like that you could modify your lesson plans and change them” and Leah’s comment: “I would definitely use the lesson plans, and especially the groups, I like that, where you can just create a topic, and just have people reply to it.” Vivian who explained what interested her said, “It was interesting because it is a community of teachers, not something you are offering to me.”

Discussion

Summary of Findings and Interpretations

This section summarizes and discusses major study findings. It should be noted first that findings are discussed with caution due to nonexperimental nature of the study, with the small number of participants. Most teachers reported positive experience with RoboSTEM on both usability ratings as well as open-ended questions. Although the ratings between Phases 1 and 2 could not be statistically compared because of the small sample size, trends were in the direction of improvement from Phase 1 to Phase 2; that is, the mean of the Phase 2 ratings was 4.40 out of 5.00 (SD = .55) compared to 4.21 (SD = 0.76) in Phase 1. Scores on the scales on which the lowest ratings were recorded from Phase 1—content and attractiveness—increased from 3.94 to 4.60 and from 3.50 to 4.20, respectively. The highest ratings from Phase 2—visual clarity (4.80), exchangeability (4.90), content (4.60), attractiveness (4.20), helpfulness (4.50), learnability (4.40), and technical functionality (4.60) suggest that teachers considered RoboSTEM usable and useful (Table 3). These findings were aligned with teachers’ responses to open-ended questions. Most teachers reported not only that RoboSTEM was easy to use and its design was simple and attractive but also that they liked the lesson search and sharing features and group, community, and discussion features. A few things that participants disliked during Phase 1 (e.g., unclear placeholders for creating lessons and lesson search processes) were fixed before Phase 2. However, a couple of teachers in Phase 2 were still confused with lesson search and editing processes.

In addition to the aforementioned findings from the self-report survey, the analysis of the screen, face, and voice recordings from usability test sessions (hereafter, the video analysis) indicates that the level of teachers’ behavioral engagement in RoboSTEM was high in general during both phases. Teachers did more than they were asked to do during the usability testing. Taking initiative is an indicator of a high level of behavioral engagement (Fredricks et al., 2004). Behavioral engagement also improved from Phase 1 to Phase 2 in that in Phase 2, all tasks given by the researchers were completed. Although not much inference could be made with regard to cognitive engagement from the video analysis because specific tasks and the order of completing the tasks were assigned to teachers, there were a few instances in which a high level of cognitive engagement, specifically, monitoring and rehearsal, was observed. For example, participants sought to confirm whether their lessons, videos, or events that they created showed up properly (monitoring), and tested multiple files when creating a lesson (rehearsal). With regard to emotional engagement, a few participants expressed confusion while searching for and creating a lesson, for example. However, in all these cases, they succeeded in completing tasks quickly after confusion and showed the emotions of relief and joy.

Findings from interview data analyses were aligned with findings discussed earlier in general and provide explanations for reasons behind teachers’ survey responses and actions during usability testing sessions. All teachers reported positive experiences with usability overall. Even when a few issues with navigation were reported, teachers acknowledged learnability of the site. Teachers’ positive perception of learnability explains their emotional engagement from confusion to relief and joy observed from the video analysis. Especially, in Phase 2, after fixing menu and search features, no teacher thought of suggestions for improvement other than adding more colors to the site.

The high ratings on content in Phase 2 were corroborated by teacher interviews. Teachers’ comments on theories for STEM engagement and lesson videos, in both phases, show that teachers were interested in the example materials integrating robotics into classroom practice. Teachers valued the sharing aspect of RoboSTEM as well, which promoted their cognitive engagement (value) in the site. Most extra activities that teachers did, which evidences a high level of behavioral engagement, were also related to searching, viewing, and commenting on what other teachers had done and connecting themselves with other teachers. These initiatives by teachers suggest that the strategies embedded in the design of RoboSTEM to enhance belonging and valuing may have worked as aimed. The extra activities were to satisfy social goals. Conducting further examinations of other teachers’ work per their interests seems to be a result of RoboSTEM design to foster their interests in the use of RoboSTEM and instrumental value. However, activities showing teachers’ pursuing shared goals and building standards together were not observed, probably due to the one-time use of RoboSTEM for the purpose of the usability test. Also, cognitive engagement related to “motivation to learn” (Fredricks et al., 2004, p. 63) such as valuing RoboSTEM and example-based learning was more apparent than cognitive engagement related to “being strategic” such as monitoring and rehearsal (Fredricks et al., 2004, p. 64). More of the latter such as elaboration and evaluation, in addition to monitoring and rehearsal, may have been observed while performing the usability tasks or commented during the interviews, if RoboSTEM was tried out more than once. In addition, findings may have been also different if RoboSTEM included not only mastery models but also coping models (Renkl, 2014). Mastery models refer to the models that demonstrate a performance process toward a successful outcome without failure. Coping models refer to the models that demonstrate incorrect performance but also a process of fixing it. Presentation of both examples and nonexamples are recommended as an instructional strategy (Merrill, 2002). Especially, teachers’ learning about what could go wrong and how to prepare for it in their lesson planning is critical toward just-in-time modifying their teaching in class.

Teachers pointed out the potentially positive impact of RoboSTEM on teachers with little knowledge of STEM or robotics. The aims of RoboSTEM include motivating more teachers to engage in use of robotics for STEM education. The observation of non-STEM teachers’ potentially becoming self-efficacious about STEM education via robotics is a significant finding. Teacher efficacy influences teacher practice (Tschannen-Moran & Hoy, 2001; Yoon, Evans, & Strobel, 2014). In addition, a few teachers even modeled example-based learning for their current or future students as illustrated in the following comment: “not just for me [as a teacher] to create lessons, but actually show to students.” Teachers’ modeling of the example-based learning approach is also meaningful, given only one time use of RoboSTEM.

Indicators of teachers’ behavioral, cognitive, and emotional engagement, discussed earlier, illustrate how usable and useful they perceived RoboSTEM was. Especially, substantial comments on STEM engagement theories and teaching examples on the site suggest the possibility of using RoboSTEM as a connector between theory and practice as designed. Nonetheless, these findings do not guarantee teachers’ actual practice using RoboSTEM and teaching STEM via robotics. There is a need to embed strategies in RoboSTEM to promote teacher practice (i.e., reproduction of the modeled behavior) between observations on RoboSTEM, which is hinted in the following comment:

It would be nice if there is a reflection box underneath [posts], because I can ask in the comment how well this [lesson] works. Maybe each teacher already put something underneath, like, I used this lesson, here is what worked well, here is what didn’t work well. If you are thinking about using this lesson, these are good.

Although the commenting feature was already on RoboSTEM and this teacher did not have a chance to see and use it, the feature could be modified to promote its use for creating and sharing lessons and videos in response to the original post. Reproduction is not just a copy of the model (Renkl, 2014). Thus, the response posts could illustrate how the original lesson can be modified for specific student needs. Sharing reflections on reproduction experiences would also add richness of OERs on RoboSTEM.

Limitations of the Present Studies

First, the usability testing was done in a laboratory setting. Teachers may have engaged in RoboSTEM differently from a real setting such as a classroom. Second, the small sample size (n = 13) is another limitation of the study. The teachers who volunteered to participate in the research may have been more interested in educational robotics, STEM education, or website designs. Third, their remarks may have been influenced by their prior knowledge about STEM and robotics although the majority of participants, nine, indicated that they had not heard of the use of robotics for STEM education or educational robotics at all prior to their participation in the usability testing. Three indicated that they had only heard of it, and one did not indicate whether he or she had heard of it. Fourth, other possible users of RoboSTEM such as school administrators, parents, and students were not included in the study because of the scope of this research—investigating the perceived usability for teachers. However, more diverse viewpoints could have led to more design improvements. Fifth, if the participant pool included more in-service teachers teaching one or more of the STEM subjects (e.g., elementary teachers), more content-specific data would have been produced. Sixth, although it was practically impossible to have teachers to create and share their own lesson plan during the usability testing session that already took 40 to 60 minutes, their perceived usability, engagement, and example-based learning may have been different if they actually utilized RoboSTEM while planning their own lessons. Last, value was considered in the current study as evidence of cognitive engagement although it could be included in emotional engagement as well (see Fredricks et al., 2004, p. 84 for details).

RoboSTEM Applications

RoboSTEM is applicable to teacher education contexts guiding teacher learning of robotics technology for teaching as well as designing and implementing lessons using robotics. It can help pre-service and in-service teachers (a) recognize the benefits of robotics technology in standards-based teaching and learning, (b) integrate robots into teaching STEM subjects, and (c) create lesson plans using robotics technology that aim to engage students in learning 21st-century skills. Students and parents can also use RoboSTEM. Because of the multidisciplinary nature of robotics, RoboSTEM also has the potential to promote active collaborations of researchers and instructors among different disciplines. Possible scenarios include expertise sharing between an engineering course (advanced knowledge of robotics) and a teacher education course (expansion of robotics application) through RoboSTEM. Collaborations in multiple academic areas can be sustained even outside of formal courses because of the community of learning and practice fostered by RoboSTEM.

Future Research and Development Directions

Further research is needed in real-world contexts to see how usable and useful RoboSTEM is perceived by teachers. For example, RoboSTEM can be used in a course for undergraduate students preparing to be elementary teachers. Those pre-service teachers need to be equipped to teach STEM (Greenberg et al., 2013). First, their learning from RoboSTEM about lessons using robotics that tailor specific educational standards, second, creating their own lessons, and, third, sharing them with other teachers would provide valuable perspectives to designers, researchers, and instructors. Such a usability testing would inform not only how to improve the RoboSTEM design but also how to implement RoboSTEM in teacher education contexts. Future development could also consider a new feature that allows teachers to add prompts to their lessons and videos to help others effectively and efficiently understand the content. Such prompts are for “learning … from observing model videos” and called “learning points” (Rummel et al., 2009, p. 74).

Although initial efforts have been made by the project team to get resources available on RoboSTEM, the goal of RoboSTEM is to be sustained through user-created content. Thus, there should be ways to initiate teachers’ involvement. The team plans to call for showcases in which teachers post their lesson plans using robots for STEM education. Based on the number of views and likes and user ratings as well as the positive comments that their showcases received, teachers could be awarded and recognized within and among school districts. At the same time, to maintain the quality of OERs on RoboSTEM, guidelines to suitable theoretical foundations for their design of resources could be added to RoboSTEM.

Conclusion

The design of RoboSTEM and its theoretical foundations guide future research and development that aim to create online environments not only for teaching and training but also for educational robotics and STEM education. In addition, usability testing designed based on a synthesis of research-based usability studies provides an instrumental checklist for researchers and practitioners to improve their development projects. Moreover, the purpose of RoboSTEM itself has the potential to contribute to the teacher education that leads to growing STEM interests and competencies among students.

Footnotes

Acknowledgements

This article is supported by the Learning Technologies Grant of the Center for Teaching and Learning at the University of Georgia. The views, opinions, and findings expressed are those of the authors, and do not necessarily represent those of the University of Georgia.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This article is supported by the Learning Technologies Grant of the Center for Teaching and Learning at the University of Georgia. The views, opinions, and findings expressed are those of the authors, and do not necessarily represent those of the University of Georgia. Declaration of Conflicting Interests is correct.

Author Biographies

ChanMin Kim is an associate professor of Learning, Design, and Technology at the University of Georgia (UGA). Her current research involves programming, debugging, robotics, and teacher education.

Jiangmei Yuan is an assistant professor of Instructional Design and Technology at West Virginia University. She has received her PhD degree from Learning, Design, and Technology program at UGA. Her research focus is in STEM education through technologies (e.g., robotics) and peer assessment.

Dongho Kim has recently graduated from the Learning, Design, and Technology program at UGA. He worked as a research assistant when this reported study was conducted. He currently works at Northern Illinois University.

Prashant Doshi is a professor of Computer Science and the director of the THINC Lab at the University of Georgia. He is also a faculty member of the Institute for AI at the University of Georgia and directs the Faculty of Robotics at UGA. His research interests lie in artificial intelligence, robotics, and the semantic Web.

Chi N. Thai is an associate professor of the College of Engineering at the University of Georgia. He researches in the areas of multispectral imaging and theatre robotics. He has taught robotics to numerous Duke TIP students.

Roger B. Hill is a professor of Workforce Education at the University of Georgia. His research focuses on affective characteristics necessary for success in current and future occupations. He has integrated this line of research into teaching related to engineering and technology education and computer information systems.

Ernst Melias is a computer scientist with expertise in programming and business development. He earned his Bachelor of Science degree in Computer Science with an emphasis in Artificial Intelligence from the University of Georgia.

References

Albo-Canals, J., Heerink, M., Diaz, M., Padillo, V., Maristany, M., Barco, A., … Rogers, C. (2013). Comparing two LEGO robotics-based interventions for social skills training with children with ASD. Paper presented at the 22nd IEEE International Symposium on Robot and Human Interactive Communication, Gyeongju, Korea.

Alimisis

(2013) Educational robotics: Open questions and new challenges. Themes in Science & Technology Education 6(1): 63–71.

Altin

Pedaste

(2013) Learning approaches to applying robotics in science education. Journal of Baltic Science Education 12(3): 365–377.

Ardito, C., Costabile, M. F., de Marsico, M., Lanzilotti, R., Levialdi, S., Roselli, T., & Rossano, V. (2006). An approach to usability evaluation of e-learning applications. Universal Access in the Information Society, 4(3), 270–283.

Yuan, J., & Kim, C. (2014). Guidelines for facilitating the development of learning communities in online courses. Journal of Computer Assisted Learning, 30(3), 220–232.

Kim, C., Kim, D., Yuan, J., Hill, R. B., Doshi, P., & Thai, C. N. (2015). Robotics to promote elementary education pre-service teachers' STEM engagement, learning, and teaching. Computers & Education, 91, 14–31.

Baker, J. Y. (2011). Mathematics activity in an after-school robotics program: An investigation of mathematics use in a semiformal learning environment (PhD thesis). University of Pennsylvania, Philadelphia.

Bandura

(1986) Social foundations of thought and action: A social cognitive theory, Englewood Cliffs, NJ: Prentice Hall.

Bandura

(1997) Self-efficacy: The exercise of self-control, Gordonsville, VA: WH Freeman.

10.

Barker

B. S.

Nugent

Grandgenett

(2014) Examining fidelity of program implementation in a STEM-oriented out-of-school setting. International Journal of Technology & Design Education 24(1): 39–52. doi:10.1007/s10798-013-9245-9.

11.

Becker

K. H.

Park

(2011) Integrative approaches among science, technology, engineering, and mathematics (STEM) subjects on students’ learning: A meta-analysis. Journal of STEM Education: Innovations and Research 12(5–6): 23–37.

12.

Beer

R. D.

Chiel

H. J.

Drushel

R. F.

(1999) Using autonomous robotics to teach science and engineering. ACM 42(6): 85–92. doi:10.1145/303849.303866.

13.

Belland

Kim

Hannafin

M. J.

(2013) A framework for designing scaffolds that improve motivation and cognition. Educational Psychologist 48(4): 243–270.

14.

Benitti

F. B. V.

(2012) Exploring the educational potential of robotics in schools: A systematic review. Computers & Education 58(3): 978–988. doi:10.1016/j.compedu.2011.10.006.

15.

Bers

M. U.

(2008) Blocks to robots: Learning with technology in the early childhood classroom, New York, NY: Teachers College Press.

16.

Bers

M. U.

Portsmore

(2005) Teaching partnerships: Early childhood and engineering students teaching math and science through robotics. Journal of Science Education & Technology 14(1): 59–73.

17.

Bjerrum

A. S.

Hilberg

van Gog

Charles

Eika

(2013) Effects of modelling examples in complex procedural skills training: A randomised study. Medical Education 47(9): 888–898. doi:10.1111/medu.12199.

18.

Chhetri, R. (2011). Usability test of HSPLS website. Retrieved from http://scholarspace.manoa.hawaii.edu/handle/10125/20019.

19.

Danahy

Wang

Brockman

Carberry

Shapiro

Rogers

C. B.

(2014) LEGO-based robotics in higher education: 15 Years of student creativity. International Journal of Advanced Robotic Systems 11(2): 27. doi:10.5772/58249.

20.

Duran

Höft

Lawson

D. B.

Medjahed

Orady

E. A.

(2014) Urban high school students’ IT/STEM learning: Findings from a collaborative inquiry- and design-based afterschool program. Journal of Science Education and Technology 23(1): 116–137. doi:10.1007/s10956-013-9457-5.

21.

Eguchi

(2012) Educational robotics theories and practice: Tips for how to do it right. In: Barker

B. S.

Nugent

Grandgenett

Adamchuk

V. I. I.

(eds) Robots in K-12 education: A new technology for learning, Hershey, PA: Information Science Reference, pp. 1–30.

22.

Ericsson

(2003) Valid and non-reactive verbalization of thoughts during performance of tasks—Towards a solution to the central problems of introspection as a source of scientific data. Journal of Consciousness Studies 10(9–10): 1–18.

23.

Finn

Zimmer

(2012) Student engagement: What is it? Why does it matter? In: Christenson

S. L.

Reschly

A. L.

Wylie

(eds) Handbook of research on student engagement, New York, NY: Springer, pp. 97–131. doi:10.1007/978-1-4614-2018-7_5.

24.

Fredricks

J. A.

Blumenfeld

P. C.

Paris

A. H.

(2004) School engagement: Potential of the concept, state of the evidence. Review of Educational Research 74(1): 59–109.

25.

Fry

Rich

(2011) Usability testing for e-resource discovery: How students find and choose e-resources using library web sites. The Journal of Academic Librarianship 37(5): 386–401.

26.

Ge, X., & Land, S. M. (2004). A conceptual framework for scaffolding ill-structured problem-solving processes using question prompts and peer interactions. Educational Technology Research and Development, 52(2), 5 –22.

27.

Glaser

B. G.

(1965) The constant comparative method of qualitative analysis. Social Problems 12(4): 436–445. doi:10.2307/798843.

28.

Goldman

Eguchi

Sklar

(2004) Using educational robotics to engage inner-city students with technology. In: Kafai

Y. B.

(ed.) Proceedings of the sixth international conference of the learning sciences, Mahwah, NJ: Lawrence Erlbaum, pp. 214–221.

29.

Granić

Ćukušić

(2011) Usability testing and expert inspections complemented by educational evaluation: A case study of an e-learning platform. Journal of Educational Technology & Society 14(2): 107–123.

30.

Greenberg, J., McKee, A., & Walsh, K. (2013). Teacher prep review 2013 report. National Council on Teacher Quality. Retrieved from http://www.nctq.org/dmsView/Teacher_Prep_Review_2013_Report.

31.

Hansen

Gonzalez

(2014) Investigating the relationship between STEM learning principles and student achievement in math and science. American Journal of Education 120(2): 139–171.

32.

Hawkes

(2001) Variables of interest in exploring the reflective outcomes of network-based communication. Journal of Research on Computing in Education 33(3): 299.

33.

Henry

Knight

Thornberry

(2012) School disengagement as a predictor of dropout, delinquency, and problem substance use during adolescence and early adulthood. Journal of Youth & Adolescence 41(2): 156–166. doi:10.1007/s10964-011-9665-3.

34.

Hinchliffe

Mummery

W. K.

(2008) Applying usability testing techniques to improve a health promotion website. Health Promotion Journal of Australia 19(1): 29–35.

35.

Jaipal-Jamani

Angeli

(2017) Effect of robotics on elementary preservice teachers' self-efficacy, science learning, and computational thinking. Journal of Science Education and Technology 26(2): 175–192.

36.

Karahasanović

Hinkel

U. N.

SjØberg

D., J., K.

Thomas

(2009) Comparing of feedback-collection and think-aloud methods in program comprehension studies. Behaviour & Information Technology 28(2): 139–164. doi:10.1080/01449290701682761.

37.

Karahoca

Uzunboylub

(2011) Robotics teaching in primary school education by project based learning for supporting science and technology courses. World Conference on Information Technology 3: 1425–1431. doi:10.1016/j.procs.2011.01.025.

38.

Kay, J. S., Moss, J. G., Engelmann, S., & McKlin, T. (2014). Sneaking in through the back door: Introducing K-12 teachers to robot programming. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (pp. 499–504). Atlanta, GA.

39.

Kazakoff

Sullivan

Bers

(2013) The effect of a classroom-based intensive robotics and programming workshop on sequencing ability in early childhood. Early Childhood Education Journal 41(4): 245–255. doi:10.1007/s10643-012-0554-5.

40.

Kietzmann

J. H.

Hermkens

McCarthy

I. P.

Silvestre

B. S.

(2011) Social media? Get serious! Understanding the functional building blocks of social media. Business Horizons 54(3): 241–251.

41.

Kim

Bennekin

K. N.

(2013) Design and implementation of volitional control support in mathematics courses. Educational Technology Research & Development 61(5): 793–817. doi:10.1007/s11423-013-9309-2.

42.

Landis

J. R.

Koch

G. G.

(1977) The measurement of observer agreement for categorical data. Biometrics 33: 159–174.

43.

Larkins, D. B., Moore, J. C., Rubbo, L. J., & Covington, L. R. (2013). Application of the cognitive apprenticeship framework to a middle school robotics camp. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education (pp. 89–94). Denver, Colorado.

44.

Levin

Waugh

(1998) Teaching teleapprenticeships: Electronic network-based educational frameworks for improving teacher education. Interactive Learning Environments 6(1–2): 39–59.

45.

Lindh

Holgersson

(2007) Does Lego training stimulate pupils’ ability to solve logical problems? Computers & Education 49(4): 1097–1111.

46.

Lou

S.-J.

Shih

R.-C.

Diez

C. R.

Tseng

K.-H.

(2011) The impact of problem-based learning strategies on STEM knowledge integration and attitudes: An exploratory study among female Taiwanese senior high school students. International Journal of Technology and Design Education 21(2): 195–215.

47.

Loucks-Horsley

Matsumoto

(1999) Research on professional development for teachers of mathematics and science: The state of the scene. School Science and Mathematics 99(5): 258–71.

48.

Mataric, M. J., Koening, N., & Feil-Seifer, D. (2007). Materials for enabling hands-on robotics and STEM education. Paper presented at the AAAI Spring Symposium on Robots and Robot Venues Renues: Resources for AI Education.

49.

Matera

Rizzo

Carughi

G. T.

(2006) Web usability: Principles and evaluation methods. In: Mendez

Mosley

(eds) Web engineering, Berlin, Germany: Springer, pp. 143–180. Retrieved from https://link-springer-com.web.bisu.edu.cn/chapter/10.1007/3-540-28218-1_5.

50.

McGill

M. M.

(2012) Learning to program with personal robots: Influences on student motivation. ACM Transactions on Computing Education 12(1): 4.

51.

Mentes

S. A.

Turan

A. H.

(2012) Assessing the usability of university websites: An empirical study on Namik Kemal University. Turkish Online Journal of Educational Technology-TOJET 11(3): 61–69.

52.

Merrill

M. D.

(2002) First principles of instruction. Educational Technology Research and Development 50(3): 43–59.

53.

Mitnik

Recabarren

Nussbaum

Soto

(2009) Collaborative robotic instruction: A graph teaching experience. Computers & Education 53(2): 330–342. doi:10.1016/j.compedu.2009.02.010.

54.

National Research Council. (2011). Successful K-12 STEM education: Identifying effective approaches in science, technology, engineering, and mathematics. Washington, DC: National Academies Press.

55.

Nugent

Barker

Grandgenett

Adamchuk

V. I.

(2010) Impact of robotics and geospatial technology interventions on youth STEM learning and attitudes. Journal of Research on Technology in Education 42(4): 391–408.

56.

Ortiz

A. M.

Bos

Smith

(2015) The power of educational robotics as an integrated STEM learning experience in teacher preparation programs. Journal of College Science Teaching 44(5): 42.

57.

Osborne, R. B., Thomas, A. J., & Forbes, J. (2010). Teaching with robots: A service-learning approach to mentor training. In Proceedings of the 41st ACM technical symposium on computer science education (pp. 172–176). New York, NY: ACM. Retrieved from http://dl.acm.org/citation.cfm?id=1734321.

58.

Perritt

D. C.

(2010) Including professional practice in professional development while improving middle school teaching in math. National Teacher Education Journal 3(3): 73–76.

59.

Poll, R. (2007). Evaluating the library website: Statistics and quality measures. Presented at the World Library and Information Congress: 73rd IFLA General Conference and Council, Durban, South Africa.

60.

Pittí, K., Curto, B., Moreno, V., & Rodríguez, M. J. (2013). Resources and features of robotics learning environments (RLEs) in Spain and Latin America. In Proceedings of the 1st international conference on technological ecosystem for enhancing multiculturality (pp. 315–322). New York, NY: ACM. doi:10.1145/2536536.2536584.

61.

Preece

Rogers

Sharp

(2002) Interaction design: Beyond human-computer interaction, Chichester, England: John Wiley & Sons.

62.

Reeve, J., Jang, H., Carrell, D., Jeon, S., & Barch, J. (2004). Enhancing students' engagement by increasing teachers' autonomy support. Motivation and Emotion Motivation and Emotion, 28(2), 147 –169.

63.

Renkl

(2014) Toward an instructionally oriented theory of example-based learning. Cognitive Science 38(1): 1–37. doi:10.1111/cogs.12086.

64.

Reschly

Christenson

(2012) Jingle, jangle, and conceptual haziness: Evolution and future directions of the engagement construct. In: Christenson

S. L.

Reschly

A. L.

Wylie

(eds) Handbook of research on student engagement, New York, NY: Springer, pp. 3–19. Retrieved from doi:10.1007/978-1-4614-2018-7_1.

65.

Rummel

Spada

Hauser

(2009) Learning to collaborate while being scripted or by observing a model. International Journal of Computer-Supported Collaborative Learning 4(1): 69–92. doi:10.1007/s11412-008-9054-4.

66.

Schafer

J. L.

(1997) Analysis of incomplete multivariate data, London, England: Chapman & Hall.

67.

Schunk

D. H.

Hanson

A. R.

(1985) Peer models: Influence on children’s self-efficacy and achievement. Journal of Educational Psychology 77(3): 313–322. doi:10.1037/0022-0663.77.3.313.

68.

Silva

M. L.

(2012) Camtasia in the classroom: Student attitudes and preferences for video commentary or Microsoft Word comments during the revision process. Computers and Composition 29(1): 1–22. doi:j.compcom.2011.12.001.

69.

Skinner

E. A.

Kindermann

T. A.

Furrer

C. J.

(2008) A motivational perspective on engagement and disaffection: Conceptualization and assessment of children’s behavioral and emotional participation in academic activities in the classroom. Educational and Psychological Measurement 69(3): 493–525. doi:10.1177/0013164408323233.

70.

Sklar

Parsons

Stone

(2004) Using RoboCup in university-level computer science education. Journal on Educational Resources in Computing 4(2): 1–21.

71.

Sullivan

F. R.

Moriarty

M. A.

(2009) Robotics and discovery learning: Pedagogical beliefs, teacher practice, and technology integration. Journal of Technology and Teacher Education 17(1): 109–142.

72.

Tocháček

Lapeš

(2012) The project of integration the educational robotics into the training programme of future ICT teachers. International Conference on Education & Educational Psychology (ICEEPSY 2012) 69: 595–599. doi:10.1016/j.sbspro.2012.11.451.

73.

Toh

L. P. E.

Causo

Tzuo

P.-W.

Chen

I.-M.

Yeo

S. H.

(2016) A review on the use of robots in education and young children. Journal of Educational Technology & Society 19(2): 148–163.

74.

Tschannen-Moran

Hoy

A. W.

(2001) Teacher efficacy: Capturing an elusive construct. Teaching and Teacher Education 17(7): 783–805.

75.

Tseng

K.-H.

Chang

C.-C.

Lou

S.-J.

Chen

W.-P.

(2013) Attitudes towards science, technology, engineering and mathematics (STEM) in a project-based learning (PjBL) environment. International Journal of Technology & Design Education 23(1): 87–102. doi:10.1007/s10798-011-9160-x.

76.

Voelkl

K. E.

(1997) Identification with school. American Journal of Education 105(3): 294.

77.

Vygotsky

L. S.

(1978) Mind in society, Cambridge, MA: Harvard University Press.

78.

Wade, V. P., & Lyng, M. (1999). Experience improving WWW based courseware through usability studies (Vol. 1999, pp. 951–956). Waynesville, NC: AACE. Retrieved from.

79.

Williams

D. C.

Prejean

Ford

M. J.

Lai

(2007) Acquisition of physics content knowledge and scientific inquiry skills in a robotics summer camp. Journal of Research on Technology in Education 40(2): 201–216.

80.

World Wide Web Consortium. (2008). Web content accessibility guidelines (WCAG) 2.0. Retrieved from http://travesia.mcu.es/portalnb/jspui/handle/10421/2546.

81.

Yoon

S. Y.

Evans

M. G.

Strobel

(2014) Validation of the teaching engineering self-efficacy scale for K-12 teachers: A structural equation modeling approach. Journal of Engineering Education 103(3): 463–485. doi:10.1002/jee.20049.

82.

Zaharias

Poylymenakou

(2009) Developing a usability evaluation method for e-learning applications: Beyond functional usability. International Journal of Human-Computer Interaction 25(1): 75–98. doi:10.1080/10447310802546716.