Impact of Augmented Reality on Programming Language Learning: Efficiency and Perception

Abstract

Although the learning of programming language is critical in science and technology education, it might be difficult for some students, especially novices. One possible reason might be the fact that programming language, especially for three-dimensional (3D) applications, is too complex and abstract for these students to understand. Programming for 3D applications requires understanding the spatial relationship of 3D objects and hence needs a visualization technique more. In view of this, this article presents an augmented reality (AR)-enhanced learning system that offers visual representation and interactivity to help students learn programming for 3D applications. To examine the influences of such an AR-enhanced system on student learning, a within-group experiment with 34 college students was conducted. All students used both of an AR-enhanced version and an ordinary version. The findings revealed that the AR-enhanced version made students have better learning efficiency than the ordinary system. In addition, the AR-enhanced system also made students have enhanced perceptions in terms of system usability, flow experience, and usage perception. Based on the results, further development of AR-enhanced learning systems is also suggested and discussed.

Keywords

augmented reality programming language usability perception augmented reality-based learning augmented reality learning environment

Introduction

Recently, programming has come to be regarded not only as a foundational skill in Science, Technology, Engineering, and Mathematics education but also a critical element for the development of computational thinking, which involves the decomposition of decision making and all possible solutions to make sure that the right decision could be made (Wing, 2006). However, programming languages are specifically constructed to communicate with the computer through a set of instructions, tailored to the structure of the computer and its operations (e.g., input, output, control, and memory units). For novices who are not familiar with computer structures and operations, learning programming might be a complex and abstract process (Lahtinen, Ala-Mutka, & Järvinen, 2005).

Recently, because of the increasing prevalence of three-dimensional (3D) applications, developing 3D applications via writing program has become an important skill for students, especially for those who major in multimedia-related disciplines. Developing 3D applications is typically achieved by writing programs using some 3D graphics libraries, such as OpenGL or Direct3D. This process usually involves the drawing of 3D graphics or models in a virtual space using a set of abstract instructions. However, students (especially novices) need to become familiar with basic concepts about 3D graphics rendering and 3D object operations, such as movement, rotation, and scaling. In other words, the learning of programming for 3D applications, that is, developing 3D applications using the mentioned 3D graphics libraries, requires more prior knowledge and support for 3D operations and visualization.

With advances in information technology, various technologies have been developed and applied to enhance students’ learning. For instance, game-based learning highlights the enhancement of both interest and understanding (Tundjungsari, 2016). In addition, some studies further investigate how to benefit students’ programming learning by using an 3D environment (Cooper, Wanda, & Randy, 2000) or visual tools (Sáez-López, Román-González, & Vázquez-Cano, 2016), and the findings of them demonstrate some positive outcomes and potentials. Recently, augmented reality (AR)-based learning has been attracting increasing research attention (Chang, Hou, Pan, Sung, & Chang, 2015; Hsu, 2017; Yoon, Anderson, Lin, & Elinich, 2017) because it could offer a natural way to present information based on the reality (Carmigniani & Furht, 2011). In particular, AR technology might bring new opportunities to enhance student learning in terms of contextual visualization and learning interactivity (Ibáñez, Di Serio, Villarán, & Delgado Kloos, 2014).

Regarding contextual visualization, in AR visualization, the real-world environment is combined with computer-generated sensory data derived from graphic computing and object recognition technologies (Azuma et al., 2001; Bacca, Baldiris, Fabregat, Graf, & Kinshuk, 2014). This visualization process might help students understand the use of programming codes, such as for the positioning of objects and their relevant relationships. Regarding learning interactivity, AR allows augmented information for the surrounding real world to become interactive and manipulable (Santos, Yamamoto, Taketomi, Miyazaki, & Kato, 2013), which could encourage students to have positive attitudes toward exploration, inquiry, logical thinking, and reasoning. In short, the two features of AR, that is, visualization and interactivity (Azuma, 1997), might have the potential to overcome the two difficulties that novices encounter in learning programming for 3D applications (i.e., abstraction and complexity).

Although the potential of AR technology is valuable, few studies have investigated how AR technology could be used to empower students’ programming language learning, and what the influences of such a learning system on students’ learning might be. To fill this gap, this study aims to develop an AR-based learning system and investigate its influences on student learning. More specifically, the research questions addressed in this study are as follows: (a) Could AR technology be incorporated to develop a learning system for programming language? (b) Does such an AR-based learning system benefit student learning?

Literature Review

With the advance of information technology, technology-enhanced learning has attracted increasing attention, such as mobile learning, ubiquitous learning, and game-based learning (Johnson, Adams Becker, Estrada, & Freeman, 2014). One of the recently emerging technologies is AR, which combines or supplements visualization of the real-world environment with virtual objects through graphic computing and object recognition technologies (Bacca et al., 2014), allowing virtual objects to coexist in the same space as real-world objects (Azuma et al., 2001).

AR’s two characteristics (visualization and interactivity) could be used to enhance student learning in terms of contextual visualization and learning interactivity. Contextual visualization refers to the presentation of virtual information in the rich context of a real environment, where information and learning cues can help students construct their knowledge (Santos et al., 2014). Learning interactivity refers to embodied interactions with virtual content that allow students to observe a 3D model on a marker, which can be further manipulated by hand (Santos et al., 2013). Previous studies have also demonstrated that these two characteristics of AR produce a positive impact on student motivation (Bujak et al., 2013; Chang et al., 2014; Di Serio, Ibáñez, & Kloos, 2013) and further drive different educational applications.

Reviewing previous studies, it can be found that there are two learning theories underpinning various AR-based applications. One is experiential learning (Kolb, 1984), which regard entire experiences as the source of learning and advocates that students should learn by creating meaningful experiences from scenarios. For example, Huang, Chen, and Chou (2016) developed an AR-based ecological system to support experiential learning. Since AR can provide on-site information in an authentic outdoor environment, the students can be immersed in environmental exploration and interaction with AR-enhanced ecological information. Another AR-based system was developed to support students’ English learning (Liu, Tan, & Chu, 2007, 2010). With the support of handheld devices and AR technology, students are guided to visit different learning zones, where context-aware learning materials are offered.

The other learning theory supporting AR-based applications is contiguity principle of multimedia learning (Clark & Mayer, 2011; Mayer, 2009), which advocates aligning texts to corresponding graphics or objects. In other words, the text should not be physically separated from the graphics or objects. For instance, the AR technology could be used to develop a visualization approach for electromagnetic ﬁelds, which are normally invisible to the students (Buchau, Rucker, Wossner, & Becker, 2009; Ibáñez et al., 2014). In this system, AR is used to display the electromagnetic field so that students can understand the distribution of electromagnetic fields in space. This visual approach allows students to easily understand the concept of electromagnetic ﬁeld theory. Kaufmann and Meyer (2008) used AR technology to deliver physics education. In this AR-based system, students can conduct experiments in a 3D virtual world. For instance, they can manipulate objects to analyze forces, mass, and paths during the experiments. During the process, the AR-based visualization and information can benefit student learning and discussion. The AR technology could also be used to improve students’ spatial abilities (Maier, Klinker, & Tönnis, 2009; Martín-Gutiérrez et al., 2010). An AR-based system was developed to display 3D virtual models based on the markers to promote the development of their spatial ability. Students can turn or move the markers to observe virtual models and complementary information from different perspectives, which further contributes to their comprehensive understanding.

Recently, programming has come to be regarded as a required capability in Science, Technology, Engineering, and Mathematics education. However, because of its abstract nature and complexity, some students have difficulty learning programming (Lahtinen et al., 2005). To alleviate the difficulties, previous studies have applied various technologies to help students learn. For instance, a 3D interactive environment is developed to engage students in improving their programming skills (Cooper et al., 2000). By writing simple scripts, students can control the appearance and behavior of 3D objects which helps to understand the concepts. In addition, visual programming language tools (Sáez-López et al., 2016) and game-based environments (Tundjungsari, 2016) are also used in formal or informal learning settings to enhance students’ programming ability. However, although AR is an emerging technology, few studies have investigated how it can be used to help students learn a programming language. To fill the gap, this study develops an AR-based learning system.

Two Learning Systems

Ordinary Version

The learning content used in this study is OpenGL, which allows students to develop rich interactive 3D applications via a set of commands. As illustrated in Figure 1, an ordinary version of the learning system contains two mechanisms to help students learn: the coding window and the output window.

Figure 1.

Ordinary version of the learning system.

Coding window

The coding window allows students to type OpenGL commands (or functions) to execute them and also examines whether the spelling of the typed commands is correct. When students type the correct commands, the result is shown in the output window. Students are required to remember the correct spelling of each command, the order of the commands, and how they are used.

Output window

The output window shows a two-dimensional (2D) view of the result of executing the typed commands. The result is illustrated from the perspective of a virtual camera within the 3D virtual world. In other words, this window offers a 2D view as it would appear on the virtual camera. When students type a new command (e.g., moving a box) in the coding window, they can immediately observe the result (e.g., the new position of the box from the perspective of the virtual camera) in this window. In this way, students can examine the correctness of the result and further adjust the commands.

AR-Enhanced Version

In addition to the aforementioned output window, the AR-enhanced version further offers students two mechanisms to help them learn: the holistic window and command cards, as shown in Figure 2. The coding window in the AR-enhanced version is somewhat different from that of the ordinary version. It is used to display the equivalent OpenGL codes for reference.

Figure 2.

AR-enhanced version of the learning system.

Holistic window

The holistic window offers students a global 3D perspective view to enhance their comprehensive understanding. Unlike the local 2D view provided by the output window, the holistic window visualizes the whole context of the virtual world which benefits the students’ understanding. More specifically, with the support of AR technology, a specific AR marker is used for camera tracking so that the real camera can see the whole 3D virtual world in the holistic window. In this way, students can inspect the objects from different viewpoints by moving the real camera or rotating the AR marker. The holistic window supplements the local and 2D perspective provided by the output window, leading to a more comprehensive understanding of the operations of the OpenGL commands and the changes applied by those commands.

Command cards

Command cards allow students to write the programs by sequencing the cards labeled with OpenGL commands, instead of just typing the commands. In this way, students can avoid any misspelling of commands which could lead to compilation of errors in the ordinary version and instead focus on the logical thinking and reasoning behind using these commands. With the support of quick response codes, each command card with a quick response code can be recognized when it is placed in front of the camera. The executed result is further shown in both the output window and the holistic window. Students can understand the relations to the commands by observing the visual response.

There are four categories of command cards in our system as illustrated in Figure 3. The Object cards are used to generate 3D objects in the AR environment. There are two types of Object cards. One is the user-defined object which is generated by any modeling software. The other is Glut Object which is provided by the GLUT library. The Instruction cards are used for manipulating the 3D objects. Users can move the 3D objects by showing these cards to the camera. The Light cards are used to generate a light source in the space. With the Light cards, users can control the properties of a light source. The final type of cards is Camera cards, which are used to set up a camera in the space. With these cards, a user can define the position and orientation of the virtual camera.

Figure 3.

Four categories of command cards.

Method

To address the second research question (i.e., does such an AR-based learning system benefit student learning?), an experimental design was conducted to evaluate the impacts of the AR-based learning system, in terms of four aspects: system usability, learning efficiency, flow experience, and usage perception.

Participants

The participants were 34 first-year college students in Taiwan. Although all of them had taken the course of “introduction to programming language” (18 weeks) in the first semester, they never learn OpenGL programing language. In other words, they did not have background knowledge about the OpenGL, especially the concept of 3D coordinates, and its commands to move, rotate, and modify 3D objects.

Procedure

To evaluate the influences of the AR-based learning system, this study conducted a within-subject experiment, where all of participants experienced both of an ordinary version (described in Ordinary Version section) and an AR-enhanced version (described in AR-Enhanced Version section), and then their reactions to the two versions were further compared. In addition, to reduce the biases resulting from treatment order, the participants were divided into two groups (i.e., Group A and Group B), where the Group A students first used the AR-enhanced version and then used the ordinary version, whereas the Group B students first used the ordinary version and then used the AR-enhanced version. By doing so, the treatment order could be balanced. Specifically, the following procedures were employed, as illustrated in Figure 4.

Before using the learning systems, the participants were given a brief period of instruction for 10 min as to how to use the systems. In addition, to force them to pay attention to the learning tasks, they were asked to do 17 practical exercises, where the participants were required to learn some of OpenGL’s basic functions, including creating, translating, rotating, and scaling primitive objects.

During the session of system usage, participants in the two groups used different versions of the learning systems. The Group A students first used the AR-enhanced version for 25 min and then used the ordinary version for 25 min, while the Group B students first used the ordinary version first and then switched to the AR-enhanced system for the same length of time.

For data collection, all of the participants were requested to fill out three questionnaires about flow experience, usage perception, and system usability. In this way, the participants’ reactions for the two versions were collected after they had used both systems.

Figure 4.

Procedures employed in this study

Measurements

System usability

To collect information about the usability of the systems, an existing system usability scale aimed at giving a global view of usability assessment (Brooke, 1996) was used. The scale had a total of 10 items, including five positively worded items and five negatively worded items (see Appendix). Each item was rated on the 5-point Likert scale, with respondents ranging from strongly disagree to strongly agree. The scale finally yielded a number to represent an overall measure of the system usability, ranging from 0 to 100. The scale had an adequate reliability, with a Cronbach’s alpha of 0.92 (Lewis & Sauro, 2009).

Learning efficiency

Learning efficiency could be defined as the speed by which novices’ knowledge or skills are acquired and become fluency like experts (Kolfschoten, Lukosch, Verbraeck, Valentin, & de Vreede, 2010). In other words, learning efficiency emphasizes the levels of proficiency, with which students could successfully perform their tasks during a period of time. Thus, the following formula was used in this study to measure learning efficiency: the numbers of successfully completed tasks during the system usage session (i.e., 25 min). During the session, the numbers of completed tasks were recorded in the system logs for further analysis.

Flow experience

Flow experience was measured using an existing scale for measuring flow experience in digital games (Fu, Sua, & Yub, 2009). More specifically, four facets of the original scale were appropriate for the purposes of this study, including concentration (six items, e.g., no distraction from the task is highlighted; I can remain concentrated in the system.), goal clarity (four items, e.g., overall system goals were presented in the beginning of the game; overall system goals were presented clearly), autonomy (three items, e.g., I know next step in the game; I feel a sense of control over the game), and immersion (seven items, e.g., I forget about time passing while using the system; I feel emotionally involved in the system). Each item of the four facets was scaled on the 5-point Likert scale, ranging from strongly disagree (Point = 1) to strongly agree (Point = 5). The reliabilities (Cronbach’s alpha) of the four facets of the questionnaire were 0.54, 0.65, 0.46, and 0.50, respectively.

Usage perception

To measure student perception on system usage, a perception questionnaire developed by the authors of this article was used. The questionnaire contained five items: (a) Helpfulness: which system helps you understand the OpenGL instructions better? (b) Ease of use: which system is easier to learn and use? (c) Enjoyment: which system do you enjoy learning with more? (d) Efficiency: which system allows you to more effectively complete the exercises? and (e) Preference: which system do you prefer to use in learning OpenGL? For each item, the participants were asked to choose one of the two versions.

Data Analysis

The independent variable of the experiment was the system instrument with two versions: the ordinary version and the AR-enhanced version. The dependent variables were the participants’ reactions in terms of the four dimensions: system usability, learning efficiency, flow experience, and usage perception. Paired-sample t tests were conducted to examine the different reactions between the two systems for each dimension. All of the analyses were conducted using the Statistical Package in Social Science (SPSS v20) software.

Results and Discussion

System Usability

Table 1 shows the means and SD of the scores for the system usability scale. The results reveal that the mean of the AR-enhanced version was higher than that of the ordinary version. A further paired-sample t test demonstrated that a statistically significant difference (t = 8.57, p < .01) existed. The results imply that, in general, the AR-enhanced version enhanced system usability.

Table 1.

The Results of System Usability.

	AR-enhanced version		Ordinary version		t
	Mean	SD	Mean	SD	t
Usability	71.02	12.53	58.16	11.74	8.57**

p < .01.

A possible explanation for this finding is that the AR-enhanced version could display virtual information with AR-annotated objects in the context of a real environment, which further benefited perceived usability. In addition, the command cards could also give students more opportunities to control the codes and manipulate these objects, which might be helpful to participation and perception of system usability.

Learning Efficiency

Table 2 illustrates the number of completed and uncompleted tasks in the two groups. For Group A, students using the AR-enhanced version completed more exercises (i.e., 15 units) than those using the ordinary version (i.e., 12 units). Similarly, for Group B, it was also found that students completed more exercises with the AR-enhanced version (i.e., 17 units) than with the ordinary version (i.e., 7 units). In total, students with the AR-enhanced version completed 32 (i.e., 15 + 17) units, whereas those with the ordinary version completed 19 (i.e., 12 + 7) units. These results suggest that the AR-enhanced version was more proficient at successfully completing exercises than the ordinary version within the given time.

Table 2.

The Number of Tasks Completed With the Two Versions.

	Group A		Group B
	AR-enhanced version	Ordinary version	AR-enhanced version	Ordinary version
Completed tasks	15	12	17	7
Uncompleted tasks	2	5	0	10

A possible explanation for this finding is that the additional support of the holistic window and command cards of the AR-enhanced version offered students a global view to know what they did and the consequent results. In other words, they might have a better awareness of what happened in the virtual world, and meanwhile they could quickly adjust their actions based on the feedback, so they spent less time in completing the exercises, leading to an improvement in the amount of completed tasks.

Flow Experience

Table 3 illustrates the means and SD of the scores on the flow scale in terms of four facets. The results of further paired-sample t tests showed that there was no significant difference in the facet of goal, indicating that both of the two versions offered clear goals to make students have similar perceptions in this facet. In contrast, the results demonstrated a statistically significant difference in the facets of concentration (t = 2.76, p < .01), autonomy (t = 3.60, p < .01), and immersion (t = 3.19, p < .01). The results suggested that the students who used the AR-enhanced version had a better flow experience than when they used the ordinary version. In other words, the AR-enhanced version made them become more engaged in the learning tasks.

Table 3.

The Results of the Flow Scale.

	AR-enhanced version		Ordinary version		t
	Mean	SD	Mean	SD	t
Concentration	4.34	0.47	4.13	0.43	2.76**
Goal clarity	4.38	0.51	4.22	0.52	1.58
Autonomy	4.28	0.67	3.83	0.64	3.60**
Immersion	4.09	0.50	3.86	0.53	3.19**

p < .01.

A possible explanation for this finding is that the command cards offered the students an easy and manipulable approach to write the programs, so they had better control which enhances their sense of autonomy, and they paid more attention and became more deeply immersed because of the quick and immediate feedback from the holistic window. This might explain why there was an increase in the three aspects of the flow experience.

Usage Perception

The results for usage perception also indicated that most students preferred the AR-enhanced version over the ordinary version, for all five facets, as illustrated in Table 4. This result seems to imply that the AR-enhanced version was more useful, easy to use, and efficient for them. Meanwhile, most students also felt that using the AR-enhanced version to learn was a more enjoyable experience. They liked to use the AR-enhanced version to learn the OpenGL materials.

Table 4.

The Number of Students Choosing the Systems.

	AR-enhanced version	Ordinary version
Helpfulness	31	3
Ease-of-use	32	2
Enjoyment	33	1
Efficiency	32	2
Preference	33	1

A possible explanation is that the AR-enhanced version offered students an interesting, useful, and easy-to-use way to learn programming for 3D applications. Thus, most of them showed a positive and preferential attitude for these aspects.

Implications

The aforementioned results for the AR-enhanced version are summarized in Figure 5. The findings are further discussed in terms of four aspects: usability, cognitive, motivational, and affective aspects.

Figure 5.

Effects of AR-enhanced learning.

First of all, usability is critical from the perspective of human–computer interaction. According to Nielsen’s (1993) principles, usability involves key issues of system function, namely, memorability, learnability, efficiency, and error prevention. The results of this study indicate that the use of AR technology increased the usability of the learning system. Most students felt that the AR-enhanced version was helpful and easy-to-use. Second, regarding the cognitive aspect, the AR-enhanced version made students perform higher learning efficiency, implying that a well-designed AR system could serve as an efficient cognitive tool to help students learn programming for 3D applications. In other words, the AR-enhanced version could offer students better efficiency in the cognitive dimension.

However, a wider view of usability should be advocated: “from usability to enjoyment” (Blythe, Overbeeke, Monk, & Wright, 2003). In addition to functionality, how enjoyable it is for students to interact with a learning system is also a critical consideration. In other words, emerging technologies not only play an important role in students’ learning efficiency but also in their engagement and affection. Some pioneering researchers have conducted related work on enjoyment. For instance, Malone (1984) proposed some design heuristics for enjoyable interfaces. Norman (2002) emphasized emotional design that could offer users a more enjoyable experience. Thus, system usability and user enjoyment are two key features that should be considered in the human–computer interaction and educational settings (Thomas & Macredie, 2002).

The findings of this study reveal that the AR-enhanced version contributed to the students’ perception and flow experience. It is possible that the AR technology enables a learning-by-doing model, which combines manipulating AR-markers and observing the outcome of visual representation, which offers students a more enjoyable and engaging way to learn programming for 3D applications. Specifically, when students used the AR system, what they needed to do was to organize these command cards in a highly ordered and reasonable way, not always thinking about the detailed syntax. Such quick and visual feedback based on their participation contributed to continuous challenge. According to flow theory (Csikszentmihalyi, 1990), students thus found it easier to enter into the zone of deeper engagement.

Future development of AR-enhanced learning systems requires more studies to design systems that incorporate usability and enjoyment. The findings suggested using a framework consisting of four aspects (i.e., cognitive, motivational, usability, and affective aspects) as a starting point to find the balance between usability and enjoyment, enabling students to learn in a way that is both effective and enjoyable.

Conclusions

Responses to Research Questions

This article addressed two research questions. In response to the first (i.e., could AR technology be incorporated to develop a learning system for programming language?), this study proposed an AR-enhanced system, where students can directly manipulate an AR-based marker as input to learn OpenGL functions and immediately observe the outcomes via visual representations.

In response to the second research question (i.e., does such an AR-based learning system benefit student learning?), the findings showed that (a) the AR-enhanced system enhanced students’ perceived system usability; (b) the students who used the AR-enhanced system completed more learning tasks within the given time; (c) the AR-enhanced system made students become more engaged in the learning tasks; (d) the students perceived the AR-enhanced system as more useful, easy to use, and efficient, thus leading to a more enjoyable experience.

Contributions and Limitations

The contributions of this study cover two aspects: development and evaluation. For development, this study offers an experience in how to incorporate AR technology with a learning system to help students learn programing language. Such experience might enrich the practical applications of technology-enhanced language learning. For evaluation, such an AR-enhanced learning system is further evaluated and offers a framework to discuss its implication in terms of cognitive, motivational, usability, and affective aspects. The framework might be helpful for the future design of AR-enhanced learning systems.

However, further investigation is required because of the limitations of this study. First, this was a pilot study with a small sample size. The effects with larger sample sizes should be further investigated. Second, this was merely a short-term study. More long-term evaluations are required to investigate the influences on student learning and examine their consistency with this study.

Footnotes

Appendix

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Ministry of Science and Technology, Taiwan, under Grant Nos. MOST 101-2221-E-155-064, MOST 105-2218-E-155-010, and MOST-104-2511-S-155-002. This research is supported also by the “Aim for the Top University Project” of National Taiwan Normal University (NTNU), sponsored by the Ministry of Education, Taiwan, R.O.C.

Author Biographies

Chin-Hung Teng received his PhD degree in Electrical Engineering from National Tsing Hua University, Hsinchu, Taiwan, in 2005. He is currently an assistant professor at the Department of Information Communication, Yuan Ze University. His research interests include color image processing, computer vision, computer graphics, image-based modeling, and augmented reality.

Jr-Yi Chen received his MS degree in Information Communication from Yuan Ze University, Taiwan, in 2013. His research interests include QR code application and augmented reality.

Zhi-Hong Chen is an associate professor in the Graduate Institute of Information and Computer Education at National Taiwan Normal University, Taiwan. His research interests include technology-enhanced language learning and game-based learning.

References

Azuma

(1997) A survey of augmented reality. Presence: Teleoperators and Virtual Environments 6(4): 355–385.

Azuma

Baillot

Behringer

Feiner

Julier

MacIntyre

(2001) Recent advances in augmented reality. IEEE Computer Graphics and Applications 21(6): 34–47.

Bacca

Baldiris

Fabregat

Graf

Kinshuk (2014) Augmented reality trends in education: A systematic review of research and applications. Educational Technology & Society 17(4): 133–149.

Blythe, M. A., Overbeeke, K., Monk, A. F., & Wright, P. C. (2003). From usability to enjoyment. Human computer interaction series (Vol. 3). Dordrecht, The Netherlands: Kluwer Academic Publishers.

Brooke, J. (1996). SUS: A quick and dirty usability scale. Usability Evaluation in Industry, 189, 4–7.

Buchau

Rucker

W. M.

Wossner

Becker

(2009) Augmented reality in teaching of electrodynamics. The International Journal for Computation and Mathematics in Electrical and Electronic Engineering 28(4): 948–963.

Bujak

K. R.

Radu

Catrambone

MacIntyre

Zheng

Golubski

(2013) A psychological perspective on augmented reality in the mathematics classroom. Computers & Education 68: 536–544. doi:10.1016/j.compedu.2013.02.017.

Carmigniani, J., & Furht, B. (2011). Augmented reality: An overview. In B. Furht (Ed.), Handbook of augmented reality (Chapter 1, pp. 3–46). New York, NY: Springer-Verlag.

Chang

K. E.

Chang

C. T.

Hou

H. T.

Sung

Y. T.

Chao

H. L.

Lee

C. M.

(2014) Development and behavioral pattern analysis of a mobile guide system with augmented reality for painting appreciation instruction in an art museum. Computers & Education 71: 185–197.

10.

Chang

Y. L.

Hou

H. T.

Pan

C. Y.

Sung

Y. T.

Chang

K. E.

(2015) Apply an augmented reality in a mobile guidance to increase sense of place for heritage places. Educational Technology & Society 18(2): 166–178.

11.

Clark

R. C.

Mayer

R. E.

(2011) E-Learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning, 3rd ed. San Francisco, CA: John Wiley & Sons.

12.

Cooper, S., Wanda, D., & Randy, P. (2000). Alice: A 3-D tool for introductory programming concepts. Proceedings of the 5th Annual CCSC Northeastern Conference, Mahwah, NJ, 28–29.

13.

Csikszentmihalyi

(1990) Flow: The psychology of optimal performance, New York, NY: Cambridge University Press.

14.

Di Serio

Á.

Ibáñez

M. B.

Kloos

C. D.

(2013) Impact of an augmented reality system on students’ motivation for a visual art course. Computers & Education 68: 586–596. doi:10.1016/j.compedu.2012.03.002.

15.

F. L.

Sua

R. C.

Yub

S. C.

(2009) A scale to measure learner’s enjoyment of e-learning games. Computers & Education 52(1): 101–112.

16.

Hsu

T. C.

(2017) Learning English with augmented reality: Do learning styles matter? Computers & Education 106: 137–149.

17.

Huang

T. C.

Chen

C. C.

Chou

Y. W.

(2016) Animating eco-education: To see, feel, and discover in an augmented reality-based experiential learning environment. Computers & Education 96: 72–82.

18.

Ibáñez

M. B.

Di Serio

Á.

Villarán

Delgado Kloos

(2014) Experimenting with electromagnetism using augmented reality: Impact on flow student experience and educational effectiveness. Computers & Education 71: 1–13. doi:10.1016/j.compedu.2013.09.004.

19.

Johnson

Adams Becker

Estrada

Freeman

(2014) Horizon report 2014 – Higher education edition, Austin, TX: The New Media Consortium.

20.

Kaufmann, H., & Meyer, B. (2008). Simulating educational physical experiments in augmented reality. Proceedings of SIGGRAPH ASIA, New York, NY, 8.

21.

Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Upper Saddle River, NJ: Prentice Hall.

22.

Kolfschoten

Lukosch

Verbraeck

Valentin

de Vreede

G. J.

(2010) Cognitive learning efficiency through the use of design patterns in teaching. Computers & Education 54(3): 652–660.

23.

Lahtinen

Ala-Mutka

Järvinen

H. M.

(2005) A study of the difficulties of novice programmers. ACM SIGCSE Bulletin 37(3): 14–18.

24.

Lewis, J. R., & Sauro, J. (2009). The factor structure of the system usability scale. In M. Kurosu (Ed.), International conference on human centered design (pp. 94–103). Berlin/Heidelberg, Germany: Springer.

25.

Liu, T.-Y., Tan, T.-H., & Chu, Y.-L. (2007). 2D barcode and augmented reality supported English learning system. Sixth IEEE/ACIS International Conference on Computer and Information Science, Melbourne, Australia.

26.

Liu

T.-Y.

Tan

T.-H.

Chu

Y.-L.

(2010) QR code and augmented reality –supported mobile English learning system. Lecture Notes in Computer Science (LNCS) 5960: 37–52.

27.

Maier, P., Klinker, G., & Tönnis, M. (2009). Augmented reality for teaching spatial relations. Conference of the International Journal of Arts & Science, Toronto.

28.

Malone, T. W. (1984). Heuristics for designing enjoyable user interfaces: Lessons from computer games. In J. C. Thomas & M. L. Schneider (Eds.), Human factors in computer systems (pp. 1–12). Norwood, NJ: Ablex.

29.

Martín-Gutiérrez

Saorín

J. L.

Contero

Alcañiz

Pérez-López

Ortega

(2010) Design and validation of an augmented book for spatial abilities development in engineering students. Computers & Graphics 34: 77–91.

30.

Mayer

R. E.

(2009) Multimedia learning, Cambridge, England: Cambridge University Press.

31.

Nielsen

(1993) Usability engineering, New York, NY: Morgan.

32.

Norman

(2002) Emotion & design, attractive things work better. Interactions 4: 36–42.

33.

Sáez-López

J. M.

Román-González

Vázquez-Cano

(2016) Visual programming languages integrated across the curriculum in elementary school: A two year case study using “Scratch” in five schools. Computers & Education 97: 129–141.

34.

Santos

M. E. C.

Chen

Taketomi

Yamamoto

Miyazaki

Kato

(2014) Augmented reality learning experiences: Survey of prototype design and evaluation. IEEE Transactions on Learning Technologies 7(1): 38–56.

35.

Santos, M. E. C., Yamamoto, G., Taketomi, T., Miyazaki, J., & Kato, H. (2013). Augmented reality X-ray interaction in K-12 education. Proceedings of the 13th IEEE International Conference on Advanced Learning Technologies (ICALT ’13), Beijing, China, 141–145.

36.

Thomas

Macredie

R. D.

(2002) Introduction to the new usability. ACM Transactions on Computer-Human Interaction 9(2): 69–73.

37.

Tundjungsari, V. (2016). E-learning model for teaching programming language for secondary school students in Indonesia. 13th International Conference on Remote Engineering and Virtual Instrumentation (REV), Madrid, Spain, 262–266.

38.

Wing

J. M.

(2006) Computational thinking. Communications of the ACM 49(3): 33–35.

39.

Yoon

Anderson

Lin

Elinich

(2017) How augmented reality enables conceptual understanding of challenging science content. Educational Technology & Society 20(1): 156–168.