Augmented Reality as a Tool for Studying Visual Plasticity: 2009 to 2018

Abstract

Augmented reality (AR) has developed rapidly since its conception less than 30 years ago and is now a hot topic for both consumers and scientists. Although much attention has been paid to its application in industry, medicine, education, and entertainment, the use of AR in psychological research has been less noted. In this article, we survey recent progress in basic research that uses AR to explore the plasticity of the adult visual system. We focus on a particular application of AR called altered reality, which has been used to shed new light on mechanisms of long-term contrast adaptation and ocular-dominance plasticity. The results suggest that AR could also be a useful tool for the treatment of visual disorders.

Keywords

augmented reality altered reality visual plasticity contrast adaptation ocular dominance

The term augmented reality (AR) was coined by Caudell and Mizell (1992). In contrast to virtual reality (VR), which completely immerses a user inside a computer-generated virtual environment, AR is a technology in which virtual objects are superimposed on or composited with the real world so the user can see or interact with them. Over the past few decades, AR has rapidly developed and has been applied to many different domains, including personal information assistance, navigation, medicine, education, and entertainment. These uses of AR have received substantial attention in the media.

As a timely complement, we review the use of AR in basic research, focusing on studies of visual plasticity. The visual system can change its function in response to environmental demands, and two lines of research have demonstrated this: learning of specific tasks (perceptual learning) and effects of sustained exposure to a particular visual feature (adaptation; for reviews, see Basgoze, Mackey, & Cooper, 2018; Maniglia & Seitz, 2018; Shibata, Sagi, & Watanabe, 2014; Webster, 2015). In past adaptation work, stimuli were generally displayed on monitors; more recently, some researchers have used AR to investigate a series of questions regarding visual plasticity (Zhang, Bao, Kwon, He, & Engel, 2009).

Why Use AR in Research of Visual Plasticity?

A traditional way to improve vision is perceptual training, which can produce large benefits for a wide variety of tasks in adults (Karni & Sagi, 1991; Shibata et al., 2014). However, the effects of perceptual learning are often limited to the trained stimulus (Bao, Yang, Rios, He, & Engel, 2010; Karni & Sagi, 1991; Schwartz, Maquet, & Frith, 2002; but see Xiao et al., 2008). Especially large amounts of training (overtraining), potentially required to produce plasticity in earlier stages of visual processing, may show particularly high specificity (Sagi, 2011). Accordingly, some researchers have adopted a complementary approach—allowing observers to perform everyday tasks in an environment that places demands on mechanisms of plasticity. This long-term adaptation approach has been enhanced by the development and use of AR.

AR Displays

AR systems have been developed with three different presentation methods. In video see-through, video images of the world are captured digitally and combined with additional digital content; the result is presented on a head-mounted display (HMD) or handheld video display. In optical see-through, digital content is superimposed via transparent displays on a “direct” view of the world that is not digitally captured but potentially processed via optics. In projective displays, digital content is displayed on real-world objects using video projectors, and the scene is viewed naturally.

Optical see-through is often used in both commercial (e.g., Google Glass) and clinical applications (e.g., augmented-vision devices to aid patients with peripheral vision loss; Peli, Luo, Bowers, & Rensing, 2007). As early as 1992, video see-through was used to study visual adaptation to reverse-contrast (i.e., negative) images (as seen through the viewfinder of a video camera; Anstis, 1992). The HMD-based video-see-through method has been used in recent AR-based visual-plasticity studies. Unlike in studies using the typical method, in which digital content is overlaid on the camera video of the world, in visual-plasticity studies, global image processing is applied in real time to the entire video to alter some feature of the images (Bai, Dong, He, & Bao, 2017; Bao, Dong, Liu, Engel, & Jiang, 2018; Bao & Engel, 2012; Grush, Jaswal, Knoepfler, & Brovold, 2015; Haak, Fast, Bao, Lee, & Engel, 2014; Schweinhart, Shafto, & Essock, 2017; Zhang et al., 2009). This type of manipulation has been termed altered reality. The primary advantage of this approach is its flexibility; because the viewed display is entirely digital, custom computer programs can easily remove, add, or modify many different features of visual input. The disadvantages are the limits in spatial resolution, field of view, and updating speed that real-time image processing imposes.

Using Altered Reality to Study Long-Term Contrast Adaptation

The visual system adapts to changes of the visual environment over time scales spanning many orders of magnitude (Webster, 2015). Very-long-term adaptation has been studied for more than a century, beginning with George Stratton’s (1897) pioneering work investigating multiple days of adaptation to inverting prisms. Prisms and spectacles remain useful tools to this day for studying long-term visual adaptation (Kwon, Legge, Fang, Cheong, & He, 2009; Neitz, Carroll, Yamauchi, Neitz, & Williams, 2002; Tregillus, Werner, & Webster, 2016; Yehezkel, Sagi, Sterkin, Belkin, & Polat, 2010). However, optical methods are limited in the kinds of image transformations they can compute. Thus, it is difficult to use them to change the statistics of the environment in complex ways. For example, whereas it is easy to remove high-spatial-frequency information with lenses that blur the image, removing a midrange band of spatial frequencies is impossible with conventional optics.

Although some researchers have presented processed information on video screens (Falconbridge, Wozny, Shams, & Engel, 2009), the full altered-reality technique allows observers to interact with the environment freely. This makes it possible to investigate adaptation at relatively long time scales. The first work using an HMD was published in 2009 and measured effects of depriving the visual system of vertical information (Zhang et al., 2009). Video frames from a head-mounted camera were filtered in real time, using a simple multiplicative filter in the Fourier domain, and then displayed (see Fig. 1). Viewing this altered reality for 4 hr increased observers’ sensitivity to the deprived orientation (Zhang et al., 2009).

Fig. 1.

Orientation-specific deprivation using the altered-reality method. In several of the studies reviewed here (Bao & Engel, 2012; Haak, Fast, Bao, Lee, & Engel, 2014; Zhang, Bao, Kwon, He, & Engel, 2009), observers wore a head-mounted display (HMD) and a video camera (a). The camera streamed video to a laptop computer, where the images were filtered and displayed on the HMD. Observers could engage in many everyday activities, such as eating, walking, watching movies, and playing video games. In (b), the filter is shown in the Fourier domain, where orientation is indicated by polar angle and spatial frequency is indicated by distance from the center of the image. Black colors show where the filter passed zero energy and are centered on vertical orientations (along the x-axis by convention). Dots indicate the frequency and orientation of test stimuli used by Zhang et al., 2009. The bottom row shows a sample image in intact form (c) and after being filtered (d). Figure adapted from Zhang et al. (2009).

Subsequent studies measured the effects of adaptation for a range of durations (Bao & Engel, 2012; Haak et al., 2014). Bao and Engel (2012) found that the effects of adaptation gradually increased in strength and duration over a range of durations from 1 min to 8 hr. Haak et al. (2014) showed that effects continued to grow stronger and longer lasting for durations of up to 4 days. For the visual system to control adaptation over all of these durations, it seems likely that distinct mechanisms operate at different time scales.

One prediction of this multiple-mechanism theory is that residual effects of long-term adaptation will spontaneously reemerge following a short-term deadaptation that cancels the long-term effects. Such “spontaneous recovery” of long-term aftereffects has been observed in a number of studies covering many different types of visual adaptation (Bao & Engel, 2012; Bao, Fast, Mesik, & Engel, 2013; Haak et al., 2014; Mei, Dong, Dong, & Bao, 2015; Mei, Yuan, Liu, Pan, & Bao, 2018; Mesik, Bao, & Engel, 2013). It appears to be a general rule in the visual system that multiple distinct mechanisms control visual adaptation over different time scales that can range from milliseconds to days.

More subtle manipulations of the environment have also been explored recently (Schweinhart et al., 2017). Natural images contain the most content around horizontal orientations, the second most around vertical orientations, and the least near oblique orientations (±45°). Correspondingly, when matching the perceived salience of a reference pattern, observers perceive a horizontal test pattern as least salient and an oblique pattern as strongest. Schweinhart et al. (2017) used AR to place observers for 2 hr in a world in which images had equal amounts of content (formally, Fourier energy) at all orientations and at each spatial frequency. This exposure reduced the horizontal effect, suggesting that observers adjusted the salience of orientations depending on recent experience.

Using Altered Reality to Study Ocular-Dominance Plasticity

Monocular deprivation

Usually, one of our two eyes is stronger than the other. It has long been known that ocular dominance at the neural level is developed and formed during the critical period after birth and remains stable during adulthood (Wiesel & Hubel, 1963). Recent work, however, has revealed that perceptual ocular dominance in adult humans can be reshaped via several hours of monocular deprivation (Lunghi, Burr, & Morrone, 2011). Visual inputs from one eye were blocked by an eye patch, and perceptual dominance was measured with the binocular-rivalry task, in which dissimilar images were presented to the two eyes, resulting in rivalrous perception of the two images in alternation. The stronger an eye is, the more often the image presented to that eye is perceived. Lunghi et al. found that after 2.5 hr of wearing the eye patch, adult human observers perceived the image presented to the patched eye more frequently (Lunghi et al., 2011; Lunghi, Burr, & Morrone, 2013). This suggests that monocular deprivation increased the perceptual dominance of the deprived eye.

Perceptual ocular dominance is usually defined as one eye being dominant in certain behavioral tasks, such as binocular rivalry or interocular contrast integration. Another form of ocular dominance, sighting dominance, relates to which eye is used to foveate targets; its relation to perceptual dominance remains unclear, and the neural bases of both are not completely understood. Nevertheless, Lunghi et al.’s results make clear that adult binocular interactions are highly plastic at some level.

Lunghi et al.’s (2011) conclusion received support from other work adopting a different psychophysical test: binocular phase combination (Zhou, Reynaud, & Hess, 2014). Unlike in binocular rivalry, the stimuli in this task are two dichoptically presented horizontal gratings that differ only in spatial phase. The similarity between stimuli in the two eyes allows observers to experience a fused (or an integrated) grating. The perceived phase of the fused grating depends on the strength of the neural responses to each monocular grating. In general, its perceived phase is closer to that of the grating presented to the stronger eye.

Relying on this test, Zhou and colleagues (2014) explored the aftereffects following 2.5 hr of adaptation using an HMD-based setup. Observers viewed movies on an HMD, which allowed presentation of an intact movie to one eye and an image-processed movie to the “deprived” eye. They found a significant shift in perceptual ocular dominance when the deprived eye viewed low-contrast or low-pass-filtered movies. These results suggest that the interocular imbalance of contrast energy and especially the high spatial frequencies in the amplitude spectrum are critical for driving the aftereffects of monocular deprivation.

Zhou et al. (2014) did not observe significant changes of perceptual ocular dominance when the deprived eye viewed phase-scrambled movies. This manipulation leaves the Fourier amplitude spectrum of the input image alone but randomizes its Fourier phase information. These results suggest that phase information is not important in perceptual ocular-dominance plasticity. However, different results were found when perceptual dominance was measured with a binocular-rivalry task (see Fig. 2; Bai et al., 2017) in a study that also replicated the lack of an effect for binocular phase combination. These results indicate that the two measurements are likely supported by different neural mechanisms and that monocular deprivation of Fourier phase information alone can be sufficient to induce a shift of eye dominance. A number of additional articles from the groups of researchers cited above have examined effects of long-term adaptation on ocular-dominance plasticity (most are reviewed by Basgoze et al., 2018). We have limited our review to key results using AR.

Fig. 2.

The altered-reality system used in Bai, Dong, He, and Bao’s (2017) work. Observers wore the head-mounted display (HMD) during adaptation. The computer processed the images taken by the camera in real time and then presented the images to the HMD. The original image was presented to one eye while a phase-scrambled image was presented to the other eye. The small LCD monitor shows what the observer is viewing. Figure adapted from Bai, Dong, He, and Bao (2017).

Treating amblyopia with AR

The visual disorder called amblyopia (also known as lazy eye) is characterized by extreme ocular imbalance. Amblyopia is present in about 2% to 3% of the population, and people with amblyopia have very poor visual acuity and contrast sensitivity in the nondominant, amblyopic eye. Amblyopia also leads to impairment or absence of stereopsis. It is traditionally treated by placing a patch on the fellow eye (i.e., “patching”), pharmacologically penalizing the fellow eye, or a combination of these. Patching is generally most effective during childhood (Holmes, Repka, Kraker, & Clarke, 2006), leaving a shortage of treatments for adults.

Over the past 20 years, researchers have tested the effectiveness of perceptual learning to treat amblyopia (for a review, see Basgoze et al., 2018), employing both monocular training (Levi & Polat, 1996) and dichoptic training (Hess, Mansouri, & Thompson, 2010). Recent work suggests that dichoptic training induces a greater level of plasticity than monocular training (Li et al., 2013; Liu & Zhang, 2018). However, perceptual learning relies on training with specific tasks in the laboratory. Frequent training sessions can be difficult to integrate into patients’ life and work, which limits compliance and may have reduced its effectiveness in a large clinical trial (Gao et al., 2018). In addition, overtraining may limit transfer of perceptual gains to general vision outside the task (Sagi, 2011).

To overcome some of these challenges, Bao et al. (2018) used AR to study ocular-dominance plasticity and to treat patients with amblyopia. Observers adapted to complementary patchwork video images presented to each eye. Camera images were divided into squares on a grid. Each square was randomly assigned to be displayed intact to one eye, while in the other eye, the corresponding square was filled with its mean color. Viewing these images forced the visual system to weight the input from each eye equally in order to process the complete scene.

Five days of adapting to patchworks for a few hours each day increased interocular balance, as measured with binocular rivalry, in nonamblyopic observers who had relatively imbalanced eyes. Two other experiments showed that repeated patchwork adaptation produced gains in visual function in adults who were formally diagnosed with amblyopia, improving visual acuity in the amblyopic eye along with other measures of binocular function (Bao et al., 2018).

Conclusions

By allowing observers to interact with systematically altered worlds, AR has extended studies of visual adaptation to durations of hours and days. This enabled systematic investigations of long-term visual adaptation that are difficult to realize with traditional approaches. The methods described here can be extended in many directions. Future work could apply AR to study plasticity in the perception of other visual qualities, such as motion or color (Grush et al., 2015). It would also be useful to incorporate neuroimaging techniques into AR rigs to allow continuous measurement of the neural changes that produce long-term adaptation (Engel, Mesik, & Vergeer, 2018). AR also provides a natural way to study the long-term plasticity of interactions between the senses (Bai, Bao, Zhang, & Jiang, 2019).

AR technology continues to improve rapidly, with both cameras and HMDs becoming lighter and of higher resolution and image-processing hardware showing parallel gains. We expect the potential scientific value of AR to also rise in the future.

Footnotes

Action Editor

Randall W. Engle served as action editor for this article.

ORCID iD

Min Bao

Declaration of Conflicting Interests

M. Bao holds a patent on an invention for using an altered-reality system to rebalance ocular dominance. M. Bao and S. A. Engel declare no other potential conflicts of interest with respect to the authorship or the publication of this article.

Funding

This research was supported by the National Natural Science Foundation of China (Grants 31571112, 31371030, 31871104, and 31830037) and the National Science Foundation (Grant BCS1558308).

References

Anstis

(1992). Visual adaptation to a negative, brightness-reversed world: Some preliminary observations. In Carpenter

G. A.

Grossberg

(Eds.), Neural networks for vision and image processing (pp. 1–14). Cambridge, MA: MIT Press.

Bai

Bao

Zhang

Jiang

(2019). A virtual reality approach identifies flexible inhibition of motion aftereffects induced by head rotation. Behavior Research Methods, 51, 96–107. doi:10.3758/s13428-018-1116-6

Bai

Dong

Bao

(2017). Monocular deprivation of Fourier phase information boosts the deprived eye’s dominance during interocular competition but not interocular phase combination. Neuroscience, 352, 122–130. doi:10.1016/j.neuroscience.2017.03.053

Bao

Dong

Liu

Engel

S. A.

Jiang

(2018). The best of both worlds: Adaptation during natural tasks produces long-lasting plasticity in perceptual ocular dominance. Psychological Science, 29, 14–33. doi:10.1177/0956797617728126

Bao

Engel

S. A.

(2012). Distinct mechanism for long-term contrast adaptation. Proceedings of the National Academy of Sciences, USA, 109, 5898–5903. doi:10.1073/pnas.1113503109

Bao

Fast

Mesik

Engel

(2013). Distinct mechanisms control contrast adaptation over different timescales. Journal of Vision, 13(10), Article 14. doi:10.1167/13.10.14

Bao

Yang

Rios

Engel

S. A.

(2010). Perceptual learning increases the strength of the earliest signals in visual cortex. The Journal of Neuroscience, 30, 15080–15084. doi:10.1523/JNEUROSCI.5703-09.2010

Basgoze

Mackey

A. P.

Cooper

E. A.

(2018). Plasticity and adaptation in adult binocular vision. Current Biology, 28, R1406–R1413. doi:10.1016/j.cub.2018.10.024

Caudell

Mizell

D. W.

(1992, January). Augmented reality: An application of heads-up display technology to manual manufacturing processes. Paper presented at the Twenty-Fifth Hawaii International Conference on System Sciences, Kauai, HI.

10.

Engel

S. A.

Mesik

Vergeer

(2018, May). Long-term contrast deprivation increases neural gain in early visual cortex. Paper presented at the 18th Annual Meeting of the Vision Sciences Society, St. Pete Beach, FL.

11.

Falconbridge

Wozny

Shams

Engel

S. A.

(2009). Adapting to altered image statistics using processed video. Vision Research, 49, 1757–1764. doi:10.1016/j.visres.2009.03.027

12.

Gao

T. Y.

Guo

C. X.

Babu

R. J.

Black

J. M.

Bobier

W. R.

Chakraborty

. . . BRAVO Study Team. (2018). Effectiveness of a binocular video game vs placebo video game for improving visual functions in older children, teenagers, and adults with amblyopia: A randomized clinical trial. JAMA Ophthalmology, 136, 172–181. doi:10.1001/jamaophthalmol.2017.6090

13.

Grush

Jaswal

Knoepfler

Brovold

(2015). Visual adaptation to a remapped spectrum: Lessons for enactive theories of color perception and constancy, the effect of color on aesthetic judgments, and the memory color effect. In Metzinger

Windt

J. M.

(Eds.), Open MIND (pp. 1–16). Frankfurt am Main, Germany: MIND Group.

14.

Haak

K. V.

Fast

Bao

Lee

Engel

S. A.

(2014). Four days of visual contrast deprivation reveals limits of neuronal adaptation. Current Biology, 24, 2575–2579. doi:10.1016/j.cub.2014.09.027

15.

Hess

R. F.

Mansouri

Thompson

(2010). A new binocular approach to the treatment of amblyopia in adults well beyond the critical period of visual development. Restorative Neurology and Neuroscience, 28, 793–802. doi:10.3233/RNN-2010-0550

16.

Holmes

J. M.

Repka

M. X.

Kraker

R. T.

Clarke

M. P.

(2006). The treatment of amblyopia. Strabismus, 14(1), 37–42.

17.

Karni

Sagi

(1991). Where practice makes perfect in texture discrimination: Evidence for primary visual cortex plasticity. Proceedings of the National Academy of Sciences, USA, 88, 4966–4970.

18.

Kwon

Legge

G. E.

Fang

Cheong

A. M.

(2009). Adaptive changes in visual cortex following prolonged contrast reduction. Journal of Vision, 9(2), Article 20. doi:10.1167/9.2.20

19.

Levi

D. M.

Polat

(1996). Neural plasticity in adults with amblyopia. Proceedings of the National Academy of Sciences, USA, 93, 6830–6834.

20.

Thompson

Deng

Chan

L. Y.

Hess

R. F.

(2013). Dichoptic training enables the adult amblyopic brain to learn. Current Biology, 23, 308–309. doi:10.1016/j.cub.2013.01.059

21.

Liu

X. Y.

Zhang

J. Y.

(2018). Dichoptic training in adults with amblyopia: Additional stereoacuity gains over monocular training. Vision Research, 152, 84–90. doi:10.1016/j.visres.2017.07.002

22.

Lunghi

Burr

D. C.

Morrone

M. C.

(2011). Brief periods of monocular deprivation disrupt ocular balance in human adult visual cortex. Current Biology, 21, 538–539. doi:10.1016/j.cub.2011.06.004

23.

Lunghi

Burr

D. C.

Morrone

M. C.

(2013). Long-term effects of monocular deprivation revealed with binocular rivalry gratings modulated in luminance and in color. Journal of Vision, 13(6), Article 1. doi:10.1167/13.6.1

24.

Maniglia

Seitz

A. R.

(2018). Towards a whole brain model of perceptual learning. Current Opinion in Behavioral Sciences, 20, 47–55. doi:10.1016/j.cobeha.2017.10.004

25.

Mei

Dong

Bao

(2015). Spontaneous recovery of effects of contrast adaptation without awareness. Frontiers in Psychology, 6, Article 1464. doi:10.3389/fpsyg.2015.01464

26.

Mei

Yuan

Liu

Pan

Bao

(2018). Spontaneous recovery and time course of biological motion adaptation. Vision Research, 149, 40–46. doi:10.1016/j.visres.2018.06.001

27.

Mesik

Bao

Engel

S. A.

(2013). Spontaneous recovery of motion and face aftereffects. Vision Research, 89, 72–78. doi:10.1016/j.visres.2013.07.004

28.

Neitz

Carroll

Yamauchi

Neitz

Williams

D. R.

(2002). Color perception is mediated by a plastic neural mechanism that is adjustable in adults. Neuron, 35, 783–792.

29.

Peli

Luo

Bowers

Rensing

(2007). Applications of augmented vision head-mounted systems in vision rehabilitation. Journal of the Society for Information Display, 15, 1037–1045. doi:10.1889/1.2825088

30.

Sagi

(2011). Perceptual learning in vision research. Vision Research, 51, 1552–1566. doi:10.1016/j.visres.2010.10.019

31.

Schwartz

Maquet

Frith

(2002). Neural correlates of perceptual learning: A functional MRI study of visual texture discrimination. Proceedings of the National Academy of Sciences, USA, 99, 17137–17142. doi:10.1073/pnas.242414599

32.

Schweinhart

A. M.

Shafto

Essock

E. A.

(2017). Distribution of content in recently-viewed scenes whitens perception. Journal of Vision, 17(3), Article 8. doi:10.1167/17.3.8

33.

Shibata

Sagi

Watanabe

(2014). Two-stage model in perceptual learning: Toward a unified theory. Annals of the New York Academy of Sciences, 1316, 18–28. doi:10.1111/nyas.12419

34.

Stratton

G. M.

(1897). Vision without inversion of the retinal image. Psychological Review, 4, 341–360.

35.

Tregillus

K. E. M.

Werner

J. S.

Webster

M. A.

(2016). Adjusting to a sudden “aging” of the lens. Journal of the Optical Society of America A: Optics Image Science and Vision, 33, A129–A136. doi:10.1364/Josaa.33.00a129

36.

Webster

M. A.

(2015). Visual adaptation. Annual Review of Vision Science, 1, 547–567. doi:10.1146/annurev-vision-082114-035509

37.

Wiesel

T. N.

Hubel

D. H.

(1963). Single-cell responses in striate cortex of kittens deprived of vision in one eye. Journal of Neurophysiology, 26, 1003–1017.

38.

Xiao

L. Q.

Zhang

J. Y.

Wang

Klein

S. A.

Levi

D. M.

(2008). Complete transfer of perceptual learning across retinal locations enabled by double training. Current Biology, 18, 1922–1926. doi:10.1016/j.cub.2008.10.030

39.

Yehezkel

Sagi

Sterkin

Belkin

Polat

(2010). Learning to adapt: Dynamics of readaptation to geometrical distortions. Vision Research, 50, 1550–1558. doi:10.1016/j.visres.2010.05.014

40.

Zhang

Bao

Kwon

Engel

S. A.

(2009). Effects of orientation-specific visual deprivation induced with altered reality. Current Biology, 19, 1956–1960. doi:10.1016/j.cub.2009.10.018

41.

Zhou

Reynaud

Hess

R. F.

(2014). Real-time modulation of perceptual eye dominance in humans. Proceedings of the Royal Society B: Biological Sciences, 281(1795). doi:10.1098/rspb.2014.1717

Augmented Reality as a Tool for Studying Visual Plasticity: 2009 to 2018

Abstract

Keywords

Why Use AR in Research of Visual Plasticity?

AR Displays

Using Altered Reality to Study Long-Term Contrast Adaptation

Using Altered Reality to Study Ocular-Dominance Plasticity

Monocular deprivation

Treating amblyopia with AR

Conclusions

Recommended Reading

Footnotes

Action Editor

ORCID iD

Declaration of Conflicting Interests

Funding

References