Abstract
Augmented reality (AR) has developed rapidly since its conception less than 30 years ago and is now a hot topic for both consumers and scientists. Although much attention has been paid to its application in industry, medicine, education, and entertainment, the use of AR in psychological research has been less noted. In this article, we survey recent progress in basic research that uses AR to explore the plasticity of the adult visual system. We focus on a particular application of AR called altered reality, which has been used to shed new light on mechanisms of long-term contrast adaptation and ocular-dominance plasticity. The results suggest that AR could also be a useful tool for the treatment of visual disorders.
The term augmented reality (AR) was coined by Caudell and Mizell (1992). In contrast to virtual reality (VR), which completely immerses a user inside a computer-generated virtual environment, AR is a technology in which virtual objects are superimposed on or composited with the real world so the user can see or interact with them. Over the past few decades, AR has rapidly developed and has been applied to many different domains, including personal information assistance, navigation, medicine, education, and entertainment. These uses of AR have received substantial attention in the media.
As a timely complement, we review the use of AR in basic research, focusing on studies of visual plasticity. The visual system can change its function in response to environmental demands, and two lines of research have demonstrated this: learning of specific tasks (perceptual learning) and effects of sustained exposure to a particular visual feature (adaptation; for reviews, see Basgoze, Mackey, & Cooper, 2018; Maniglia & Seitz, 2018; Shibata, Sagi, & Watanabe, 2014; Webster, 2015). In past adaptation work, stimuli were generally displayed on monitors; more recently, some researchers have used AR to investigate a series of questions regarding visual plasticity (Zhang, Bao, Kwon, He, & Engel, 2009).
Why Use AR in Research of Visual Plasticity?
A traditional way to improve vision is perceptual training, which can produce large benefits for a wide variety of tasks in adults (Karni & Sagi, 1991; Shibata et al., 2014). However, the effects of perceptual learning are often limited to the trained stimulus (Bao, Yang, Rios, He, & Engel, 2010; Karni & Sagi, 1991; Schwartz, Maquet, & Frith, 2002; but see Xiao et al., 2008). Especially large amounts of training (overtraining), potentially required to produce plasticity in earlier stages of visual processing, may show particularly high specificity (Sagi, 2011). Accordingly, some researchers have adopted a complementary approach—allowing observers to perform everyday tasks in an environment that places demands on mechanisms of plasticity. This long-term adaptation approach has been enhanced by the development and use of AR.
AR Displays
AR systems have been developed with three different presentation methods. In video see-through, video images of the world are captured digitally and combined with additional digital content; the result is presented on a head-mounted display (HMD) or handheld video display. In optical see-through, digital content is superimposed via transparent displays on a “direct” view of the world that is not digitally captured but potentially processed via optics. In projective displays, digital content is displayed on real-world objects using video projectors, and the scene is viewed naturally.
Optical see-through is often used in both commercial (e.g., Google Glass) and clinical applications (e.g., augmented-vision devices to aid patients with peripheral vision loss; Peli, Luo, Bowers, & Rensing, 2007). As early as 1992, video see-through was used to study visual adaptation to reverse-contrast (i.e., negative) images (as seen through the viewfinder of a video camera; Anstis, 1992). The HMD-based video-see-through method has been used in recent AR-based visual-plasticity studies. Unlike in studies using the typical method, in which digital content is overlaid on the camera video of the world, in visual-plasticity studies, global image processing is applied in real time to the entire video to alter some feature of the images (Bai, Dong, He, & Bao, 2017; Bao, Dong, Liu, Engel, & Jiang, 2018; Bao & Engel, 2012; Grush, Jaswal, Knoepfler, & Brovold, 2015; Haak, Fast, Bao, Lee, & Engel, 2014; Schweinhart, Shafto, & Essock, 2017; Zhang et al., 2009). This type of manipulation has been termed altered reality. The primary advantage of this approach is its flexibility; because the viewed display is entirely digital, custom computer programs can easily remove, add, or modify many different features of visual input. The disadvantages are the limits in spatial resolution, field of view, and updating speed that real-time image processing imposes.
Using Altered Reality to Study Long-Term Contrast Adaptation
The visual system adapts to changes of the visual environment over time scales spanning many orders of magnitude (Webster, 2015). Very-long-term adaptation has been studied for more than a century, beginning with George Stratton’s (1897) pioneering work investigating multiple days of adaptation to inverting prisms. Prisms and spectacles remain useful tools to this day for studying long-term visual adaptation (Kwon, Legge, Fang, Cheong, & He, 2009; Neitz, Carroll, Yamauchi, Neitz, & Williams, 2002; Tregillus, Werner, & Webster, 2016; Yehezkel, Sagi, Sterkin, Belkin, & Polat, 2010). However, optical methods are limited in the kinds of image transformations they can compute. Thus, it is difficult to use them to change the statistics of the environment in complex ways. For example, whereas it is easy to remove high-spatial-frequency information with lenses that blur the image, removing a midrange band of spatial frequencies is impossible with conventional optics.
Although some researchers have presented processed information on video screens (Falconbridge, Wozny, Shams, & Engel, 2009), the full altered-reality technique allows observers to interact with the environment freely. This makes it possible to investigate adaptation at relatively long time scales. The first work using an HMD was published in 2009 and measured effects of depriving the visual system of vertical information (Zhang et al., 2009). Video frames from a head-mounted camera were filtered in real time, using a simple multiplicative filter in the Fourier domain, and then displayed (see Fig. 1). Viewing this altered reality for 4 hr increased observers’ sensitivity to the deprived orientation (Zhang et al., 2009).
Orientation-specific deprivation using the altered-reality method. In several of the studies reviewed here (Bao & Engel, 2012; Haak, Fast, Bao, Lee, & Engel, 2014; Zhang, Bao, Kwon, He, & Engel, 2009), observers wore a head-mounted display (HMD) and a video camera (a). The camera streamed video to a laptop computer, where the images were filtered and displayed on the HMD. Observers could engage in many everyday activities, such as eating, walking, watching movies, and playing video games. In (b), the filter is shown in the Fourier domain, where orientation is indicated by polar angle and spatial frequency is indicated by distance from the center of the image. Black colors show where the filter passed zero energy and are centered on vertical orientations (along the x-axis by convention). Dots indicate the frequency and orientation of test stimuli used by Zhang et al., 2009. The bottom row shows a sample image in intact form (c) and after being filtered (d). Figure adapted from Zhang et al. (2009).
Subsequent studies measured the effects of adaptation for a range of durations (Bao & Engel, 2012; Haak et al., 2014). Bao and Engel (2012) found that the effects of adaptation gradually increased in strength and duration over a range of durations from 1 min to 8 hr. Haak et al. (2014) showed that effects continued to grow stronger and longer lasting for durations of up to 4 days. For the visual system to control adaptation over all of these durations, it seems likely that distinct mechanisms operate at different time scales.
One prediction of this multiple-mechanism theory is that residual effects of long-term adaptation will spontaneously reemerge following a short-term deadaptation that cancels the long-term effects. Such “spontaneous recovery” of long-term aftereffects has been observed in a number of studies covering many different types of visual adaptation (Bao & Engel, 2012; Bao, Fast, Mesik, & Engel, 2013; Haak et al., 2014; Mei, Dong, Dong, & Bao, 2015; Mei, Yuan, Liu, Pan, & Bao, 2018; Mesik, Bao, & Engel, 2013). It appears to be a general rule in the visual system that multiple distinct mechanisms control visual adaptation over different time scales that can range from milliseconds to days.
More subtle manipulations of the environment have also been explored recently (Schweinhart et al., 2017). Natural images contain the most content around horizontal orientations, the second most around vertical orientations, and the least near oblique orientations (±45°). Correspondingly, when matching the perceived salience of a reference pattern, observers perceive a horizontal test pattern as least salient and an oblique pattern as strongest. Schweinhart et al. (2017) used AR to place observers for 2 hr in a world in which images had equal amounts of content (formally, Fourier energy) at all orientations and at each spatial frequency. This exposure reduced the horizontal effect, suggesting that observers adjusted the salience of orientations depending on recent experience.
Using Altered Reality to Study Ocular-Dominance Plasticity
Monocular deprivation
Usually, one of our two eyes is stronger than the other. It has long been known that ocular dominance at the neural level is developed and formed during the critical period after birth and remains stable during adulthood (Wiesel & Hubel, 1963). Recent work, however, has revealed that perceptual ocular dominance in adult humans can be reshaped via several hours of monocular deprivation (Lunghi, Burr, & Morrone, 2011). Visual inputs from one eye were blocked by an eye patch, and perceptual dominance was measured with the binocular-rivalry task, in which dissimilar images were presented to the two eyes, resulting in rivalrous perception of the two images in alternation. The stronger an eye is, the more often the image presented to that eye is perceived. Lunghi et al. found that after 2.5 hr of wearing the eye patch, adult human observers perceived the image presented to the patched eye more frequently (Lunghi et al., 2011; Lunghi, Burr, & Morrone, 2013). This suggests that monocular deprivation increased the perceptual dominance of the deprived eye.
Perceptual ocular dominance is usually defined as one eye being dominant in certain behavioral tasks, such as binocular rivalry or interocular contrast integration. Another form of ocular dominance, sighting dominance, relates to which eye is used to foveate targets; its relation to perceptual dominance remains unclear, and the neural bases of both are not completely understood. Nevertheless, Lunghi et al.’s results make clear that adult binocular interactions are highly plastic at some level.
Lunghi et al.’s (2011) conclusion received support from other work adopting a different psychophysical test: binocular phase combination (Zhou, Reynaud, & Hess, 2014). Unlike in binocular rivalry, the stimuli in this task are two dichoptically presented horizontal gratings that differ only in spatial phase. The similarity between stimuli in the two eyes allows observers to experience a fused (or an integrated) grating. The perceived phase of the fused grating depends on the strength of the neural responses to each monocular grating. In general, its perceived phase is closer to that of the grating presented to the stronger eye.
Relying on this test, Zhou and colleagues (2014) explored the aftereffects following 2.5 hr of adaptation using an HMD-based setup. Observers viewed movies on an HMD, which allowed presentation of an intact movie to one eye and an image-processed movie to the “deprived” eye. They found a significant shift in perceptual ocular dominance when the deprived eye viewed low-contrast or low-pass-filtered movies. These results suggest that the interocular imbalance of contrast energy and especially the high spatial frequencies in the amplitude spectrum are critical for driving the aftereffects of monocular deprivation.
Zhou et al. (2014) did not observe significant changes of perceptual ocular dominance when the deprived eye viewed phase-scrambled movies. This manipulation leaves the Fourier amplitude spectrum of the input image alone but randomizes its Fourier phase information. These results suggest that phase information is not important in perceptual ocular-dominance plasticity. However, different results were found when perceptual dominance was measured with a binocular-rivalry task (see Fig. 2; Bai et al., 2017) in a study that also replicated the lack of an effect for binocular phase combination. These results indicate that the two measurements are likely supported by different neural mechanisms and that monocular deprivation of Fourier phase information alone can be sufficient to induce a shift of eye dominance. A number of additional articles from the groups of researchers cited above have examined effects of long-term adaptation on ocular-dominance plasticity (most are reviewed by Basgoze et al., 2018). We have limited our review to key results using AR.
The altered-reality system used in Bai, Dong, He, and Bao’s (2017) work. Observers wore the head-mounted display (HMD) during adaptation. The computer processed the images taken by the camera in real time and then presented the images to the HMD. The original image was presented to one eye while a phase-scrambled image was presented to the other eye. The small LCD monitor shows what the observer is viewing. Figure adapted from Bai, Dong, He, and Bao (2017).
Treating amblyopia with AR
The visual disorder called amblyopia (also known as lazy eye) is characterized by extreme ocular imbalance. Amblyopia is present in about 2% to 3% of the population, and people with amblyopia have very poor visual acuity and contrast sensitivity in the nondominant, amblyopic eye. Amblyopia also leads to impairment or absence of stereopsis. It is traditionally treated by placing a patch on the fellow eye (i.e., “patching”), pharmacologically penalizing the fellow eye, or a combination of these. Patching is generally most effective during childhood (Holmes, Repka, Kraker, & Clarke, 2006), leaving a shortage of treatments for adults.
Over the past 20 years, researchers have tested the effectiveness of perceptual learning to treat amblyopia (for a review, see Basgoze et al., 2018), employing both monocular training (Levi & Polat, 1996) and dichoptic training (Hess, Mansouri, & Thompson, 2010). Recent work suggests that dichoptic training induces a greater level of plasticity than monocular training (Li et al., 2013; Liu & Zhang, 2018). However, perceptual learning relies on training with specific tasks in the laboratory. Frequent training sessions can be difficult to integrate into patients’ life and work, which limits compliance and may have reduced its effectiveness in a large clinical trial (Gao et al., 2018). In addition, overtraining may limit transfer of perceptual gains to general vision outside the task (Sagi, 2011).
To overcome some of these challenges, Bao et al. (2018) used AR to study ocular-dominance plasticity and to treat patients with amblyopia. Observers adapted to complementary patchwork video images presented to each eye. Camera images were divided into squares on a grid. Each square was randomly assigned to be displayed intact to one eye, while in the other eye, the corresponding square was filled with its mean color. Viewing these images forced the visual system to weight the input from each eye equally in order to process the complete scene.
Five days of adapting to patchworks for a few hours each day increased interocular balance, as measured with binocular rivalry, in nonamblyopic observers who had relatively imbalanced eyes. Two other experiments showed that repeated patchwork adaptation produced gains in visual function in adults who were formally diagnosed with amblyopia, improving visual acuity in the amblyopic eye along with other measures of binocular function (Bao et al., 2018).
Conclusions
By allowing observers to interact with systematically altered worlds, AR has extended studies of visual adaptation to durations of hours and days. This enabled systematic investigations of long-term visual adaptation that are difficult to realize with traditional approaches. The methods described here can be extended in many directions. Future work could apply AR to study plasticity in the perception of other visual qualities, such as motion or color (Grush et al., 2015). It would also be useful to incorporate neuroimaging techniques into AR rigs to allow continuous measurement of the neural changes that produce long-term adaptation (Engel, Mesik, & Vergeer, 2018). AR also provides a natural way to study the long-term plasticity of interactions between the senses (Bai, Bao, Zhang, & Jiang, 2019).
AR technology continues to improve rapidly, with both cameras and HMDs becoming lighter and of higher resolution and image-processing hardware showing parallel gains. We expect the potential scientific value of AR to also rise in the future.
Recommended Reading
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6, 355–385. A clearly written and relatively comprehensive review for readers who wish to expand their knowledge on augmented reality.
Dey, A., Billinghurst, M., Lindeman, R., & Edward Swan, J. (2018). A systematic review of 10 years of augmented reality usability studies: 2005 to 2014. Frontiers in Robotics and AI, 5, 37. doi:10.3389/frobt.2018.00037. A comprehensive, highly accessible overview of what was known about augmented reality up to 2014.
Ehrlich, J. R., Ojeda, L. V., Wicker, D., Day, S., Howson, A., Lakshminarayanan, V., & Moroi, S. E. (2017). Head-mounted display technology for low-vision rehabilitation and vision enhancement. American Journal of Ophthalmology, 176, 26–32. doi:10.1016/j.ajo.2016.12.021. A more detailed discussion than presented in the current article of the use of head-mounted displays to enhance visual functions in people with low vision.
van Krevelen, D. W. F., & Poelman, R. (2010). A survey of augmented reality technologies, applications and limitations. The International Journal of Virtual Reality, 9(2), 1–20. A comprehensive, highly accessible overview of what is known about augmented reality.
Webster, M. A. (2009). Visual perception: Adapting to a loss. Current Biology, 19, R1030–R1032. doi:10.1016/j.cub.2009.10.015. A clearly written, short, and relatively comprehensive review for readers who wish to expand their knowledge on altered reality.
Footnotes
Action Editor
Randall W. Engle served as action editor for this article.
Declaration of Conflicting Interests
M. Bao holds a patent on an invention for using an altered-reality system to rebalance ocular dominance. M. Bao and S. A. Engel declare no other potential conflicts of interest with respect to the authorship or the publication of this article.
Funding
This research was supported by the National Natural Science Foundation of China (Grants 31571112, 31371030, 31871104, and 31830037) and the National Science Foundation (Grant BCS1558308).