Can perception be extended to a “feel of north”? Tests of automaticity with the NaviEar

1.1. Automaticity as a prerequisite of perceptual feel

It is undeniable that sensory substitution and sensory augmentation devices equip their users with new skills (see, e.g., Auvray & Myin, 2009). However, the acquisition of skilled use of a device does not necessarily imply the development of a genuine perceptual feel (Arnold et al., 2017; Auvray & Harris, 2014; Block, 2003; Brown et al., 2011; Deroy & Auvray, 2012). Evidence for the effects of sensory substitution or augmentation consists mainly of performance in very specific discrimination tasks with small sets of stimuli (e.g., Auvray et al., 2007; Bach-y-Rita et al., 1969; Buchs et al., 2021; Chebat et al., 2015; Díaz et al., 2012; Favela et al., 2018; Goeke et al., 2016; Haigh et al., 2013; Lobo et al., 2014; Proulx et al., 2015; Proulx et al., 2008; Travieso et al., 2015). Yet observed performance in this kind of discrimination may be explained by explicit cognitive interpretation rather than by the development of a perceptual feel (Deroy & Auvray, 2012; see also Goeke et al., 2016; Schumann & O'Regan, 2017). Observers may learn by heart that properties specific to the substitution signal are associated with the distinctive features of a small number of stimuli in the substituted modality, for example, that a certain vibrotactile pattern corresponds with a certain visual stimulus shape. Then, observers may explicitly retrieve this associative knowledge to identify the stimulus based on “reading” the substitution signal, similar to reading and interpreting the characters of written text (Deroy & Auvray, 2012). Neither such stimulus knowledge nor retrieval from symbolic memory necessarily reflect the experience of a perceptual feel of the substituted modality, such as the visual feel when seeing the stimulus’ shape (for a recent discussion, see Cohen, 2018; Corns, 2018; Dokic, 2018; Macpherson, 2018; Noordhof, 2018; Renier, 2018; Smith, 2018; Spence, 2018).

Other kinds of evidence for the development of perceptual feel from substitution have been obtained in subjective reports (Guarniero, 1974; Segond et al., 2005; Vazquez-Alvarez et al., 2011; Ward & Meijer, 2010). However, such reports are difficult to assess objectively because of their idiosyncratic and subjective nature (Marcel, 2003; Spence, 2018). This indeed reflects the intrinsic challenge of research on perceptual feel: Its essential aspects are subjective and thus seem to evade objective assessment.

Here we argue that while a direct objective assessment of the content of perceptual feel is methodologically challenging, perceptual feel has a core feature that can be assessed operationally: Automaticity. Take for instance vision. The perceptual interpretation of cues that produce, for example, the perceptual feel of motion, texture, depth, color, or gestalt forms occurs automatically (e.g., Goldstein & Brockmole, 2017; Snowden et al., 2006). Automaticity can also occur with non-perceptual cognitive processes, such as in semantic priming (Hutchison et al., 2013) and reading (e.g., Roembke et al., 2021). So, automaticity is not sufficient to demonstrate perceptual feel. However, under the assumption that genuine perceptual feel is highly automatic, evidence for automaticity is a necessary condition to show the potential existence of perceptual feel. If observers acquire a perceptual feel from training with a sensory substitution device, their response to the substitution signal of the device must be highly automatic. Conversely, if users do not show automaticity, then their interpretation of the substitution signal was not a perceptual feel. In this way, automaticity allows us to distinguish candidate cases of perceptual feel from explicit cognitive evaluation that involves cognitive effort. Figure 1 illustrates the logic of our approach.OPEN IN VIEWER

Tests of automaticity can make some headway toward a more operational study of perceptual feel by excluding cases where perceptual feel was absent. The criteria that characterize automaticity of responding (Ashby & Crossley, 2012; Moors, 2015; Palmeri, 2003) can be applied as an objective criterion to investigate the development of perceptual feel. Automatic responses, first, have the property that the match between stimulation and response is rigid and cannot be changed by a simple decision (rigidity and absence of cognitive control). The rigidity of automatic responses has originally been shown by the seminal experiments of Shiffrin and Schneider (Shiffrin & Schneider, 1977). The resistance of perceptual feel to cognitive control is vividly demonstrated by mandatory fusion (Hillis et al., 2002) and experiments with inverting glasses (e.g., Degenaar, 2014; Lillicrap et al., 2013). Second, automaticity requires little attentional capacity (criterion of efficiency) and is unconscious. While perceptual feel may involve attention and consciousness, the interpretation of the sensory signal itself is unconscious. Examples are the unconscious inferences involved in the color perception of #theDress (for review, see Witzel & Gegenfurtner, 2018) and the independence of visual-haptic cue combination from modality-specific attention (Helbig & Ernst, 2008). Third, automaticity is fast compared to cognitive inferences. The high automaticity of perceptual processes is illustrated by the extreme speed of scene recognition, which may occur in less than 20 ms (Potter et al., 2014). Fourth, automatic performance involves integrating single elements into a coherent whole, such as in chunking (Chase & Simon, 1973) and the automatic perception of musical sequences (Richler et al., 2011). Finally, automatized performance leads to automatic interference, such as in the Stroop effect (Augustinova & Ferrand, 2014; Stroop, 1935). An example of automatic interference specifically in perception are memory color effects (for review, see Witzel & Gegenfurtner, 2018).

The present study investigated whether the perceptual interpretation of an augmented signal can reach levels of automaticity comparable to natural (i.e., non-augmented) perceptual modalities, such as vision or audition. This approach extends the demonstrations of subcognitive integration of augmentation signals (König et al., 2016; Nagel et al., 2005) to test for automaticity as a precondition for demonstrating the emergence of perceptual feel. For this purpose, we introduced a new sensory augmentation device, called NaviEar, and developed a battery with different tests of automaticity.

1.2. The NaviEar

Geomagnetic augmentation signals the direction of geomagnetic north. This approach has been introduced to investigate learning effects in sensory augmentation. The feelSpace belt translates allocentric direction of the torso toward north into vibrations delivered on the waist (Goeke et al., 2016; Kärcher et al., 2012; Kaspar et al., 2014; König et al., 2016; Nagel et al., 2005). Following sensorimotor theory, the rationale is that mastery of sensorimotor contingencies is constitutive of perceptual feel, and hence learning to master these new sensorimotor contingencies about geomagnetic allocentric orientation should ultimately lead to a “feel of north” (and east, south, west, etc.), that is, an immediate impression of being globally oriented when engaging with these artificial sensorimotor contingencies. Results, however, have been mixed (Kaspar et al., 2014; König et al., 2016; Nagel et al., 2005).

To further develop the paradigm of geomagnetic augmentation, here we pick up the suggestion that effective augmentation devices may be informed by the science(s) of sensory integration (Cuturi et al., 2016; Gori et al., 2016). From the perspective of embodied theories of perception, such as sensorimotor theory, signaling orientation to north contingent to self-movement should lead to a perceptual feel of north not available by the natural modalities. However, considering the physiology of human embodiment, signaling the direction of north contingent to the orientation of the torso might not be the most effective way of designing an artificially augmented embodiment to endow users with a feel for their global allocentric orientation. Signaling the direction of north contingent to the orientation of the head may provide a closer link to human orientation. Head orientation aligns a magnetic directional signal with the reference frame of our natural external senses—vision, audition, and smell. From a sensory processing perspective, such implicit alignment may reduce the need for challenging reference frame transformations between these natural senses and the artificial north signal. Further, movements of the head occur much more frequently and are more fine-grained than movements of the body as a whole (Einhauser et al., 2007; Schumann et al., 2008). From a learning perspective, the head thus provides a much richer and more frequent movement source for enacting novel sensorimotor contingencies, which should aid their active exploration and learning. In addition, previous studies have shown that head movements play a key role in navigation by ear (Heller et al., 2009; Vazquez-Alvarez et al., 2011). From an ecological psychology perspective, these observations may highlight an important role of head movements in probing directional information that may further aid active exploration and learning of magnetic-directional sensorimotor contingencies. Hence, displaying magnetic allocentric information contingent to the orientation of the head rather than of the body might improve both the integration of the novel allocentric signal into the existing modalities, as well as observers’ exploration and training behavior. Indeed, in rodents, it has been shown that blind rats can use head-centered cues from an implanted geomagnetic compass for navigation (Norimoto & Ikegaya, 2015). And in humans, a self-report (Barry, 2010) described signs of automatized interpretation and perceptual experience of cardinal directions with a “compass hat.”

Also, the tactile modality, while frequently used, may pose limits to the integration of an artificial directional signal. The tactile modality has been advocated for practical reasons (e.g., Lenay et al., 2003). However, the tactile system naturally receives signals that have a proximal origin and are experienced as such. Hence, there may be a tactile proximity bias that may work toward an egocentric and against an allocentric perceptual interpretation of tactile signals. Further, the large onset asynchronies of affordable eccentric tactile actuators, such as those used in most sensory substitution studies (e.g., Cassinelli et al., 2006; Froese et al., 2012; Kärcher et al., 2012), severely limit the design of refined haptic stimulation profiles suitable to the tactile system. These onset asynchronies also impose a substantial lag between motor action and resulting sensory stimulation, typically much larger than the signal update rate in Virtual Reality applications, and orders of magnitude beyond the lag within the natural exteroceptive receptors. With high lag, a tactile directional augmentation signal may arrive only outside the temporal integration windows of co-occurrence based multisensory integration processes (Parise & Ernst, 2016), challenging the multi-modal integration of artificial information presented via (slow) tactile signals.

For these reasons, a head-centered, auditory sensory augmentation device seemed preferable to promote automatic, sensorimotor integration. This idea is supported by our previous study in the laboratory using head-centred auditory augmentation (Schumann & O’Regan, 2017). We had investigated whether training with an auditory signal conveying head-centered cardinal directions leads to automatic interference with perceived orientation under highly controlled conditions. Participants were seated in a dark room on a motorized rotation chair to precisely control their orientation and prevent interference from visual orientation cues. Results provided evidence that training with a head-centered auditory signal leads to automatic interference with perceived self-rotation. These observations contrast those with a waist-based tactile system (Goeke et al., 2016).

A drawback of such a highly controlled approach is that sensorimotor contingencies are very much limited, for example, to the rotation of the chair. In contrast, people move freely and engage in meaningful activities, such as commuting to work or doing a walk, in everyday life. Following sensorimotor theory, we considered that embedding the augmented signal into meaningful activities may be key to sensorimotor learning because they provide richer experience of sensorimotor contingencies due to freer, more complex movements. They may also improve learning through intrinsic motivation to navigate and self-orient in real life. In addition, we were inspired by the striking effects of acquired automaticity in a field setting demonstrated by riding a bike after reversing its handlebars (Sandlin, 2015). While those effects concerned sensorimotor control, here we were aiming at such effects of acquired automatization on the perceptual interpretation of an augmented signal.

Hence, we developed the NaviEar as a novel geomagnetic sensory augmentation device that can be worn during everyday life activities. The auditory directional signal provided by the NaviEar has been designed analogously to the use of the ears to determine wind direction, a simple and intuitive technique commonly used in sailing (e.g., Isler & Isler, 2006). This novel magnetic-auditory augmentation signal should thus be aligned with the reference frame of the natural exteroceptive senses, presented to a modality that naturally deals with distant sources, updated within the temporal integration windows of multisensory integration, and further easy and intuitive to understand. Thus, we expect users to learn the artificial relationship between head orientation to north and the auditory representation thereof quickly, improving the conditions for automatization and development of a perceptual feel of north when using the device during everyday-life activities outside the lab.

1.3. Tests for automaticity

Figure 1 illustrates the three parts of our experimental approach to test for automaticity: Training to use the NaviEar, tests of automaticity, and a post-experimental questionnaire.

To train participants with the NaviEar, we developed a 5-day-training program to be completed as part of everyday life activities. For users to efficiently achieve mastery of a device, it is important to engage them in relevant tasks (Bertram & Stafford, 2016; Maidenbaum, Abboud, et al., 2014). Allocentric orientation is less important in an urban than in a natural environment, as people can rely on street names and particular buildings to orient themselves. For this reason, we developed a set of training tasks specifically tailored to involve information accessible only through the NaviEar. Changes in performance on these tasks across the training period also allow us to assess general effects of training—which corresponds to evidence for “weak integration” of an augmentation signal according to Nagel et al. (2005).

To test for automaticity, we created a test battery to measure achievements of automaticity after training in the field. The battery included four test tasks that participants completed before (pre-test) and after (post-test) training. Those tests assessed involuntary automatic interference effects, cognitive flexibility, and the integration of the NaviEar signal in movement and orientation.

First, we tested whether responses to the NaviEar become robust to cognitive control in a way that indicates automaticity. To this end, we assessed participants’ ability to adapt to changes in the coding function of the NaviEar, that is, the relationship between the orientation of the head and the sound signal (rotated-signal test).

Second, an automatic interpretation of the signal should involve recognizing complex patterns defined by the sensorimotor contingencies in dynamic sequences (chunking, pattern recognition). To test whether participants can recognize complex patterns in sequences of the NaviEar signal, we developed a distortion-detection test. Here, participants walked over different paths that did or did not contain a section where the orientation-to-sound relationship was distorted and had to indicate whether they noticed an abnormality in the behavior of the NaviEar signal.

Third, in the absence of a stable external visual reference, human observers are unable to walk in a perfectly straight line but instead deviate to either side. This phenomenon is called veering (Guth & LaDuke, 1994). Veering itself happens automatically, as suggested by the fact that observers are unaware of their veering and cannot prevent themselves from veering to accomplish a straight walk. If users interpret the NaviEar signal automatically, the unbiased veridical auditory-directional information should be integrated with vestibular-proprioceptive sensory signals. This should increase the precision of orientation and reduce veering (cf. Kärcher et al., 2012). For this reason, we tested whether veering reduces after training with the NaviEar (veering test).

Finally, if using the NaviEar automatically triggers a feeling of orientation, providing an incorrect NaviEar signal should interfere with task completion. We tested whether modulations of the NaviEar signal involuntarily affect participants’ orientations when walking a series of paths (interference test).

The post-experimental questionnaire had two purposes. One part of the questions solicited participants’ first-person evaluation about the comfort and usability of the NaviEar. Similar to previous approaches (Auvray et al., 2005; Guarniero, 1974; Nagel et al., 2005; Segond et al., 2005; Vazquez-Alvarez et al., 2011; Ward & Meijer, 2010). Another set of questions targeted effects of NaviEar usage on subjective experience. These questions complemented the tests of automaticity: If the necessary condition of automaticity is met, indications of subjective experience may contribute to disambiguate whether the observed case of automaticity may be related to perceptual feel or is a non-perceptual kind of automatic cognition.

2.1. Participants

Recruiting participants that would complete two experimental sessions per week as part of their everyday life activities was a particular challenge of this study. Ten healthy adults (8 women) aged 22–37 years (M = 30.60, SD = 5.72) participated in this study. This sample size seemed acceptable in comparison with previous related studies (n = 12 in Faugloire & Lejeune, 2014; n = 11 in Froese et al., 2012; n = 1 in Kärcher et al., 2012; n = 9 in Kaspar et al., 2014; n = 9 in König et al., 2016; n = 4 Nagel et al., 2005). One participant was the author CW (who was not the experimenter, see Procedure). The others received compensation of €120 for the (at least) 15 h of training tasks and automaticity tests they completed. An overview of participants and the training and tests they completed is provided in Table 1. This study has been approved by the Comité d'Éthique de la Recherche en Santé (CERES) of the Université Paris Descartes (Nr 2015/50).

OPEN IN VIEWER

Table 1.

Participants. The first initial of the participant ID (PP) indicates the gender. Participant cw is the first author (male). “Sess” and days refer to sessions of training and the number of days those sessions involved.

PP	Age	Sess	Days	Rotat	Distort	Veer	Interf
f1	36	8	5	Post	Both	Both	Post
f2	23	7	4	Both	Both	Both	Both
f3 a	37	8	5	Both	Both	Both	Both
f4	35	10	6	Both	Both	Both	Both
f5	22	6	4	Both	Both	Both	Both
f6	28	9	5	Pre	Both	Both	Both
f7	25	8	5	Both	Both	Both	Both
f8	31	8	5	Both	Both	Both	Both
cw	36	8	5	Both	Both	Both	Both
m2	33	9	5	Both	Both	Both	Both

2.2. Apparatus and materials

The NaviEar consists of three parts (see Figure 2): a head-mountable orientation sensor (YEI technology, 3-space Bluetooth sensor (YOST Labs, 1998-2015)), bone-conduction headphones (Aftershokz Sportz M3 AS450), and an Android phone with low latency in orientation to sound mapping (LG G3 16GB). The orientation sensor consisted of three types of sensors: a gyroscope, an accelerometer, and a compass. The raw output of these sensors was processed through a Kalman filter to provide the final orientation reading. The sensor’s recording rate was 100 Hz. Nominal orientation accuracy was on average 1 deg and orientation resolution <0.08 deg (YOST Labs, 1998-2015). Before each use, the sensor was zeroed in on north by comparison with a manual geomagnetic compass. We implemented the NaviEar orientation-to-sound coding as well as the training tasks in a custom-made Android App (for source code, see https://github.com/christophWit/NaviEar). Measured all-round latency of the final system, including the external orientation sensor, was 80–120 ms.OPEN IN VIEWER

2.3. Signal

The sound signal of the NaviEar is designed in analogy to the way wind produces sound in the ears (cf. Figure 3(a)): When oriented toward the wind, the wind produces sound in both ears. In the NaviEar, the direction of the “wind” is determined to be north: When the head is oriented toward north, the NaviEar signal is played to both sides at full volume. When the head is turned clockwise away from north toward east, the signal’s volume on the right side decreases to arrive at complete silence when facing east. At the same time, the left ear continues to face north; the sound on the left is therefore still played at full volume. Turning the head further until south is reached decreases the volume on the left side, with the signal on the right remaining switched off, to end in complete silence on both sides when facing south. The same principle applies when turning counterclockwise from north toward west, with the change of the signal on the two sides now being reversed compared to the eastward turn.OPEN IN VIEWER

The sound simulated the sound of “bubbles,” which was pleasant to hear and stood out against typical noise in an everyday environment. Users adjusted the volume of the phone to a level that was comfortable while enabling them to complete the training tasks in the respective environments. As a result, the minimum intensity was silence, and maximum intensity of the bubble sound depended on user preferences and noise level in the environment. To improve the detection of volume differences between ears, the bubble sound on the left side had a slightly higher pitch than the one played on the right (the sounds are available in the GitHub repository).

The modulation of the volume was related by sigmoid functions to allocentric orientation (cf. Figure 3(b)). This was done to account for the non-linear perception of sound intensity (e.g., Stevens, 1957).

2.4. Procedure

Figure 4(a) illustrates the overall time course of the study. During this time, participants engaged with the training tasks, completed the tests of automaticity before (pre) and after the training (post), and answered a questionnaire in the form of an interview at the end of the study. Experimental sessions were run by author AL.OPEN IN VIEWER

3.1. Training tasks

If participants learned successfully to use the NaviEar and to complete the tasks, performance should increase with training; that is, response times and errors should decrease across training sessions. For each participant, response times were aggregated by median across trials. Errors were calculated as the difference between the target direction and the response by the participant. In the Angle-Difference task, an error is defined as the deviation (absolute difference) of a user’s estimate from the real difference between the angles of the test sounds according to the NaviEar coding function. There was no interesting difference between the results for the different sub-versions. For this reason and to avoid clutter, we averaged response times and errors across subversions. For details on each subversion of the training tasks, consider supplementary Table S3-4 and Figure S5.

Figure 8 illustrates the development of response times (red curves) and errors (blue curves) over training days. For each participant, we calculated correlations between the number of training sessions and response times and errors, respectively, to determine whether they have a negative (decreasing) trend over the days of training. For this, we used Spearman correlations to account for nonlinearities. We determined the regression slopes for the rank-ordered data (i.e., the basis of Spearman correlations) and tested whether slopes were below zero with a t-test. According to these tests, response times and errors in all four training tasks decreased with sessions of training, min. d = −0.87, t (9) = −2.7, all p < .03. The only exception from this were the errors in the Word-to-Pointing task (blue curve in Figure 8(a)), the slopes of which did not differ from zero, d = −0.29, t (9) = −0.9, p = .38. Analyses with nonparametric sign tests instead of t-tests and with t-test for Fisher-transformed Pearson coefficients provide very similar results (see Table S4). These results show that participants became faster with training in all tasks and more precise in three of four tasks.OPEN IN VIEWER

We examined whether the different numbers of training sessions across participants (cf. Table 1) was related to the learning effects, that is, the improvement in terms of decreasing response times and errors. For this purpose, we calculated correlations across the 10 participants between the number of training sessions and the response time and error slopes in each training task. The more negative the Spearman slope, the stronger the improvement across sessions. So, we expected a negative correlation between the amount of training and the slopes. Table S5 in the supplementary material provides detailed statistics. Errors in the Sound-to-Word tasks yielded a positive correlation coefficient close to significance, r (8) = .60, p = .06. This tendency contradicts the prediction. None of the other seven correlations was significant (ps > .15). So, there was no evidence that more training sessions led to higher levels of improvement.

Post-trial ratings were aimed to control for external factors that might affect the performance in the training tasks. Familiarity with the environment and orientation might increase the performance in the three training tasks involving self-orientation (Word-to-Pointing, Sound-to-Pointing, and Sound-to-Word) because participants might know cardinal directions even without the NaviEar. In contrast, loud noise would decrease performance, if it prevented participants to hear the NaviEar signal. For each participant, we calculated slopes of Spearman correlations between subjective ratings and performance across trials. As above, we tested across participants whether slopes differed from zero.

Ratings of orientation were negatively correlated with errors in the Word-to-Pointing, d = −1.3, t (9) = −4.0, p = .003, and the Sound-to-Word task, d = −0.92, t (9) = −2.9, p = .02. The negative Spearman slopes suggest that participants felt more oriented when their responses were more accurate (and vice versa). There were no other significant Spearman slopes between subjective ratings and performance. There were also no significant slopes for correlations between subjective orientation and familiarity across trials (all p > .11), which would relate the effects of orientation to the familiarity with the environment.

Loudness ratings were positively correlated with errors in the Sound-to-Pointing, d = 1.1, t (9) = 3.6, p = .006 and the Sound-to-Word tasks, d = 1.6, t (9) = 5.1, p < .001. The positive correlations indicate that participants made more errors when they reported louder environmental noise in the tasks that require listening to the NaviEar signal. All three types of post-trial ratings were rather flat over training sessions (cf. Figure S7), and there were no significant Spearman slopes between sessions and post-trial ratings for the three training tasks (all p > .15).

3.2. Rotated-signal test

With the rotated-signal test, we wanted to test if the flexibility of participants in interpreting the NaviEar signal decreases across training due to automatization. If this is the case, rotating the spatial orientation relative to the auditory signal should strongly reduce the performance (increase response times and errors) after training compared to the performance in the respective training tasks with the original coding of the training sessions. The change of the code should not affect their performance as strongly before as after the training because participants would not need to counteract any automatic response in the first session before any training.

In order to test this idea, we contrasted the performance (response times and errors) in the rotated-signal tests (circles in Figure 8) to the performance in the corresponding training sessions (curves in Figure 8). For this, we calculated a “rotation contrast,” which is the difference between the performance in the rotated-signal task and the performance in the training tasks of the same day of training (day 1 for the pre-training test, last day for the post-test). A positive rotation contrast indicates that performance was higher (response times and errors lower) in the training than in the rotated-signal task, and vice versa for negative rotation contrasts.

If performance in the rotated-signal test was not affected by the change of the signal, the performance should follow the overall trend of performance in the training tasks (curves in Figure 8). In this case, performance should be slightly better in the rotated-signal test than in the training task because the rotated-signal test is measured after the corresponding training task. As a result, the rotation contrast should tend to be slightly below zero. However, in the case of automatization during training, performance in the rotated-signal test should become relatively bad after training. Hence, the rotation contrast should be positive after training (post).

Figure 9 illustrates the rotation contrast for the two tasks that involved a response to the auditory NaviEar signal, namely, Sound-to-Pointing and Sound-to-Word. Again, we pooled the different sub-versions of these tasks to avoid clutter (for detailed results on each subversion, see supplementary Table S6). To test whether performance in the rotated-signal test differed from performance in the training task, we tested whether rotation contrasts differed from zero through a paired two-tailed t-test across participants. One participant (f1) missed the pre-training, and another participant (f6) the post-training rotated-signal test, reducing the number to nine participants per test.OPEN IN VIEWER

In the pre-training tests (dark bars in Figure 9), none of the rotation contrasts differed from zero, all p > .11 (top part of Table S6). In the post-training tests (light bars in Figure 9), both tasks yielded average rotation contrasts above zero, especially in the Sound-to-Word task (light bars in Figure 9(b)). Higher response times and errors reflect the difficulty of changing from the original to the rotated NaviEar signal. However, only rotation contrasts of response times in the Sound-to-Word task were significantly above zero, d = 0.94, t (8) = 2.8, p = .02, but neither those for response times in the Sound-to-Pointing task, d = 0.25, t (8) = 0.7, p = .48, nor those for errors in both tasks, both p > .32.

If sustained training led to automatization, rotation contrasts should be larger in the post- than in the pre-test. In all tasks, average rotation contrasts for response times and errors were higher in post- than in pre-training tests. We applied a paired two-tailed t-test across participants to compare rotation contrasts in pre- and post-training tests. These comparisons could only include the eight participants with data in both, pre- and post-training tests. Only the difference between pre- and post-training rotation contrasts of response times in the Sound-to-Word task was significant (d = 1.12, t (7) = 3.2, p = .02), but neither those for response times in the Sound-to-Pointing task, d = 0.58, t (7) = 1.6, p = .14, nor those for errors in both tasks (Sound-to-Pointing: d = 0.14, t (7) = 0.4, p = .70; Sound-to-Word: d = 0.21, t (7) = .6, p = .57). After applying a Bonferroni correction for testing similar predictions in two tasks (alpha = .25), the results for response times in the Sound-to-Word task are still just so significant. Hence, these results provide weak evidence for automatization.

The extent of automatization depends on the success of the training sessions, and different participants might have engaged to different degrees with the training. Figure S8 in the Supplementary Material illustrates the pre-/post-comparisons for single participants. In the Sound-to-Pointing task, six of the eight participants produced higher rotation contrasts of response times in post- than in pre-training tests in the two tasks (cf. Figure S8(a)). This was the case for 7 of 8 in the Sound-to-Word task (cf. Figure S8(c)). To test for a relationship between training and rotated-signal tests, we calculated correlations between the Spearman-slopes in the (non-rotated) training tasks (Figure 8) and the rotation contrasts in the post-training tests (light bars in Figure 9(b)). If stronger training effects led to higher automatization, post-training contrasts should be higher when slopes are more negative, implying a negative correlation. However, only the correlation for errors in the Sound-to-Word task was significant, r (7) = −.70, p = .04 (Table S7 (left side) provides detailed statistics). So, there is some evidence for a relationship between training effects and rotated-signal effects.

We also calculated correlations between rotated-signal effects and the number of training sessions across participants. See Table S7 (right side) for detailed statistics. There was no significant correlation with rotated-signal effects on response times and errors in both, the rotated Sound-to-Pointing and Sound-to-Word task, ps > .38.

3.3. Distortion-detection test

The distortion-detection test assessed the degree to which participants integrated and coordinated the NaviEar signal with their orientation. This integration and coordination should allow them to detect the distortions in the NaviEar signal in the respective trials. Most importantly, if participants internalized the coupling between orientation and NaviEar signal through training, they should better detect distorted signals after than before training.

Figure 10 shows the proportions of correct answers before (pre) and after (post) the 5-day training. We used one-tailed t-tests to test whether participants’ accuracy was significantly higher than the chance level of 50% (red dotted line) and whether accuracy differed significantly between pre-training and post-training tests. Accuracy was significantly higher than chance level before, d = 0.73, t (9) = 2.3, p = .02, and after, d = 1.37, t (9) = 4.3, p < .001, training sessions. Accuracy was 17.9% higher for easy (strong distortion) than for difficult (weak distortion) trials, d = -0.93, t(9) = -2.9, p = .02. However, overall performance before (55%) and after training (62%) was not very high. These results show that it was very difficult for participants to detect smooth distortions in the signal. Accuracy after training is significantly higher than before training, d = 0.63, t (9) = 2.0, p = .04. This result implies that participants are slightly better in detecting distortions after than before training. This suggests that participants might have learned the coupling between sound and orientation across training. However, the effect of training seems to be rather weak.OPEN IN VIEWER

There was neither a significant correlation with Spearman slopes for response times and errors in the training tasks, ps > .25, nor with the number of training sessions, r (8) = −0.16, p = .67. The effects in the distortion-detection task did not depend on amount and success of training (for details, see Table S8).

3.4. Veering test

If participants integrate the NaviEar in their orientation behavior, the NaviEar should reduce veering. Moreover, veering should further reduce with training if training produces the internalization and automatization of the NaviEar signal. Veering might reduce in both conditions (with and without signal) across sessions due to general learning of how to accomplish the task. However, if there is specific learning of the NaviEar signal, the reduction of veering should be stronger with than without the NaviEar signal.

Figure 11 illustrates the amount of veering, measured as distances in cm from the straight line (panel a) and angular deviations in degree from the orientation of the straight line (panel b). We calculated a repeated measures analysis of variance with the factors condition (no NaviEar and with NaviEar) and session (pre and post) for distances and orientations, separately. For both distances and orientations trials with NaviEar (dark bars) yielded significantly less veering than without NaviEar (light bars), F (1,9) = 9.66, p = .013, η _p ² = 0.518, and F (1,9) = 12.80, p = .006, η _p ² = 0.587. These results are in line with the predictions: They support the idea that the NaviEar generally reduces veering and improves orientation.OPEN IN VIEWER

Concerning the effect of training, veering in post-training sessions (center groups of bars) did not significantly differ from veering in pre-training sessions (left groups of bars) (distances in panel a: F (1,9) = 4.13, p = .07, η _p ² = 0.315; orientation in panel b: F (1,9) = 0.19, p = .67, η _p ² = 0.021). Moreover, there was no interaction in the predicted direction, according to which the effects of learning would be higher with than without NaviEar, F (1,9) = 2.13, p = .18, η _p ² = 0.191 and F (1,9) = 0.0004, p = .99, η _p ² = 0. Instead, average deviations went to the opposite direction, indicating more learning without than with NaviEar (cf. rightmost group of bars).

Table S9 in the supplementary material provides details on paired, two-tailed t-tests. They showed that veering was significantly lower with than without the NaviEar in both the pre- (d = 0.74, t (9) = 2.3, p = .04) and the post-training session, d = 0.96, t (9) = 3.1, p = .01, when measured in terms of orientation (left and center groups of bars in panel b). This, however, was only the case for the pre- (d = .98, t (9) = 3.1, p = .01), but not for the post-training session, d = 0.38, t (9) = 1.2, p = .26, when veering is determined in terms of distances (left and center groups of bars in panel a). These results support the idea that participants showed less veering with than without NaviEar. However, contrary to our prediction the benefit of using the NaviEar did not increase with training.

We examined the effect of training on veering through correlations between training slopes (of each training task) and veering deviation and orientation with or without the NaviEar. We expected that stronger learning in training tasks (more negative slopes) should be correlated with higher improvement in veering after training (more negative post-pre difference), resulting in positive correlations. Table S10 provides all correlations. When measured in terms of orientation, there was a positive correlation between slopes for errors in the Sound-to-Pointing task and veering effects with the NaviEar on, r (8) = .64, p = .045. Slopes for errors in the Sound-to-Word task yielded a similar tendency that just missed significance, r (8) = .63, p = .0502. There were also significant positive correlations between the numbers of training sessions per participant (cf. Table 1) and their veering effects in the condition with NaviEar for both measures, distance (r (8) = .78, p = .007) and orientation (r (8) = .69, p = .03). There were similar tendencies in the veering condition without NaviEar, but they just missed significance (r (8) = .62, p = .06; r (8) = .59, p = .08). These correlations suggest that participants with higher levels of training and learning in those training tasks tended to improve their veering after training more than did participants with less training and learning. However, it is important to note the inconsistency of those correlations across tasks and measures.

3.5. Interference test

If the interpretation of the NaviEar signal becomes strongly automatized with training it should interfere with the user’s orientation and bias the user to orient in the direction in which the signal is biased (see Figure 7 for illustration). Such interference should occur after training (post-test) when automatization is high, but not or much less before training (pre-test) when automatization is low.

The straight lines and 90-deg turns of each of the 11 paths define a target location and orientation of the participant (arrows in Figure 7). The location has a specific distance from the starting point (horizontal dotted line in Figure 7). A bias would result in a deviation from the target orientation (α in Figure 7), and a difference in the distance from the starting point (other dotted lines in Figure 7). Instead of the theoretical target locations and orientations, we considered trials with an unbiased signal as baseline to account for potential systematic deviations from the target orientation that are not due to the biases in the signal (e.g., when a participant has the tendency to veer to the left). We contrasted measurements with the biased signal to that baseline without signal. We determined the sign of the difference (in angle and distance) so that it was positive when it followed the bias of the signal (like α in Figure 7) and negative when it went in the opposite direction. For each participant, we averaged these deviations across all 11 paths. There was no data for the pre-training session of f1 (cf. Table 1).

Figure 12 illustrates the differences between biased and unbiased trials averaged across participants. We calculated two-tailed paired t-tests to test whether deviations in biased trials differed from zero and whether deviations differed between pre- and post-training sessions. For details, see lower part of Table S9. Neither distances nor orientations differed significantly from zero in both, the pre- and the post-training sessions, min. d = −0.32, t (9)=-1.0, p = .34, and there was no significant difference between pre- and post-training sessions, both p > .79. These results do not provide any support for our predictions and hence the idea that the NaviEar signal automatically interferes with navigation.OPEN IN VIEWER

If interference effects were related to training, we expected stronger interference effects (difference between bias after and before training) when training slopes were more negative (negative correlation). See Table S11 for details. There were no significant correlations. However, slopes for errors in the Sound-to-Pointing (r (7) = −.59, p = .096) and Sound-to-Word (r (7) = −.61, p = .08) tasks yielded marginally, but non-significant negative correlations with interference in orientation. There was no correlation between the number of training sessions and inference effects measured as distance (r (7) = .16, p = .68) or orientation (r (7) = −.36, p = .34).

3.6. Post-experimental questionnaire

Figure 13 illustrates the ratings (numerical items) of the post-experimental questionnaire. Supplementary Table S12 reports the qualitative answers and comments.

To evaluate ratings, we tested with a two-tailed t-test whether ratings differed from 3, the center rating between 1 and 5. Users rated the overall utility of the NaviEar with on average 3.6, which missed significance (utility in Figure 13, d = 0.60, t (9) = 1.9, p = .09). Participants felt that the NaviEar signal provided reliable information about cardinal directions, as indicated by a rating of 4.2 of 5 (reliab in Figure 13, d = 2.42, t (9) = 7.7, p < .001). The comfort rating for the NaviEar in general was 3.6, but the difference from 3 did not reach significance (comfort, d = 0.58, t (9) = 1.8, p = .10). Almost all participants responded that the bubble sound of the NaviEar was very comfortable as shown by the average rating of 4.5 of 5 (snd comf, d = 2.12, t (9) = 6.7; p < .001). As the main drawback in the utility and comfort of the NaviEar, users indicated the effort of charging, mounting, and connecting the three separate items of the NaviEar (mobile phone, headset, and sensor).OPEN IN VIEWER

We also asked users to rate (1) whether they always knew the direction they were heading when wearing and when not wearing the NaviEar (dir know) and (2) how easily they got lost with and without the NaviEar (get lost). Paired two-tailed t-tests showed that users felt they knew better where they were heading (average rating of 3.5 with vs. of 1.4 without NaviEar; d = 3.03, t (9) = 9.6, p < .001) and that they got less often lost when using the NaviEar (M = 2.2 vs. M = 3.2; d = −0.94, t (9) = −3.0, p = .02). These results support the idea that the NaviEar is useful for orientation according to subjective evaluation. However, participants also responded they usually do not think in terms of cardinal directions when they navigate through the city (nesw rel, d = −0.80, t (9) = −2.5, p = .03). They also did not feel they do better in remembering their path through training with the NaviEar (pth integr, d = −1.32, t (9) = −4.2, p = .002).

Concerning the efficiency criterion of automaticity, we also asked participants to rate (1) how much effort or thinking was required to use the NaviEar and to do the training tasks (cog effort), and (2) how integrated the NaviEar signal was into normal hearing without requiring focusing on either the NaviEar signal or usual environmental sound cues (sig integr). They had to give these ratings by comparing retrospectively between their impressions before and after they completed the training tasks. Users felt they needed significantly less cognitive effort to interpret the NaviEar after than before the training (d = −3.51, t (8) = −10.5, p < .001; one answer missing). According to their ratings, the NaviEar signal was almost twice as well integrated after (M = 3.4 of 5) than before (M = 1.7) the training, d = 2.12, t (9) = 6.7, p < .001. In their comments, some participants anticipated that they would become even more automatic in interpreting the signal when given more time for training.

Concerning changes of perceptual feel due to training, several participants reported sudden insights about their orientation (“aha-effects”). For example, they realized that streets and places familiar to them were actually oriented in rather different ways than they had thought. Most interestingly, half of the participants (i.e., 5 out of 10) experienced “aftereffects.” Some of them felt as if they still heard the NaviEar signal (i.e., the bubble sound changing as a function of head orientation); others reported an immediate and accurate awareness of cardinal directions during the minutes after taking off the device.

4.1. Usability and usefulness of the NaviEar

The increase in performance across training sessions (Figure 8) indicates that participants successfully learned how to handle and interpret the NaviEar. The only exception was the errors in the Word-to-Pointing task (blue curve in Figure 8(a)). When comparing the on- and off-condition of the Word-to-Pointing task (Figure S5(b)), the presence of the NaviEar-signal increased accuracy, but no learning took place across sessions. This observation indicates that the NaviEar signal could be cognitively interpreted up from the start, and the cognitive interpretation seemed not to benefit from additional learning.

The second day of training seemed to produce a local increase of errors in the Word-to-Pointing, Sound-to-Pointing and maybe the Angle-Difference task (blue curve at day 2 in Figure 8(a), (b) and (d)). This dip in performance coincides with a decrease of subjective orientation in the post-trial ratings (Figure S7(b)). It may be explained by participants’ confusion when the NaviEar signal was changed back after the rotated-signal test. In all four tasks, response times most steeply reduced after the first training session (cf. day 1 and day 2 in Figure 8). This fast improvement suggests that little training was necessary to reach a high level of behavioral mastery of the NaviEar.

The negative correlations between errors and subjective orientation in the post-trial ratings indicate that participants’ feeling of orientation improved thanks to the use of the NaviEar. Familiarity, as measured through subjective ratings, did not explain those correlations with orientation. However, an increased subjective impression of orientation might also result from the feedback from the tasks, rather than being the result of mastering the augmented signal.

In addition to the results from the training tasks, participants showed less veering with than without NaviEar already in the pre-training session (Figure 11). Performance in the distortion-detection test was already above chance in the pre-training session, even though only slightly (Figure 10). These results further support the idea that participants were able to understand the NaviEar signal and to use it for orientation without sustained training.

Figure S6 in the supplementary material allows for comparing the average results in our training task with those in previous, related studies. These comparisons only give a rough reference for the evaluation of the NaviEar because all those studies used different tasks and devices. Average orientation error in our training tasks in the first session were generally equal or lower than those in our previous study (10.7 to 47.8° in Schumann & O'Regan, 2017). They were also equal or lower than the landmark-pointing performance by a blind person (∼22 to 48° in Kärcher et al., 2012) and thresholds for discriminating self-orientation in a rotation chair (∼32 to 40° in Goeke et al., 2016). In contrast, the orientation errors (8.9–14.6°) in Faugloire and Lejeune (2014) were smaller than errors in our first session, and they were close to errors in the Sound-to-Word and smaller than errors in Word-to-Point and Sound-to-Point tasks in the last session. These observations are rather surprising because the studies of Schumann and O'Regan (2017), Goeke et al. (2016), and Faugloire and Lejeune (2014) were conducted at a fixed location (no movement apart from rotation) in the laboratory. The higher control of interfering factors in the laboratory was expected to produce more accurate and precise measurements. That our measurements under field conditions produced performance better or close to measurements under those controlled conditions highlights the precision of our measurements and an apt use of the NaviEar.

Together, these observations suggest that the NaviEar is easy to interpret and use for orientation. This conclusion is further supported by the positive subjective ratings in the post-experimental questionnaire: Apart from practical issues about mounting and connecting the gear, the ratings indicate that the NaviEar is generally easy and comfortable to use.

The ease and comfort of using the NaviEar suggest that this device might be applicable in real-life contexts, in which allocentric orientation plays an important role, such as in navigation at sea or in landscapes with few points of reference. Beyond that, the NaviEar could be adapted for other practical applications. In particular, an option could be implemented that turns the NaviEar signal into an auditory beacon by tuning it to a target location instead of being fixed to the north. The NaviEar could then serve as an intuitive and highly practical orientation aid when commuting through a city or finding one’s way back to a hotel when visiting a city as a tourist (e.g., Clemenson et al., 2021; Pielot & Boll, 2010). Moreover, it could be of use for the navigation of firefighters in emergencies. A device that constantly updates their position relative to the way out and/or relative to other firefighters could make the difference between life and death (e.g., Harris, 2013). Finally, it might be interesting to investigate the usefulness of the NaviEar as a prosthesis for navigation in blind people (cf. Johnson & Higgins, 2006; Kärcher et al., 2012; Kupers & Ptito, 2004; Maidenbaum, Hanassy, et al., 2014; Poirier et al., 2007). However, since visually impaired users have different skills and needs than sighted users, the potential use of the device with visually impaired users requires more specific needs assessments (Elli et al., 2014; Gori et al., 2016; Pasqualotto et al., 2013).

Many other navigation devices exist for practical applications, including torso-based (e.g., Cosgun et al., 2014; Elliott et al., 2010; Faugloire & Lejeune, 2014; Johnson & Higgins, 2006; Kärcher et al., 2012; Nagel et al., 2005; Pielot & Boll, 2010; Stoll et al., 2015), hand-held (e.g., Froese et al., 2012; Maidenbaum et al., 2014), and head-mounted (e.g., Kerdegari et al., 2016; Kojima et al., 2009; Sohl-Dickstein et al., 2015) devices. Unlike the NaviEar, most of these devices provide tactile rather than auditory stimulation (exceptions are, e.g., Maidenbaum et al., 2014; Sohl-Dickstein et al., 2015; Stoll et al., 2015). The devices also differ in whether they provide information about orientation (Cosgun et al., 2014; Elliott et al., 2010; Faugloire & Lejeune, 2014; Kärcher et al., 2012; Kerdegari et al., 2016; König et al., 2016; Nagel et al., 2005; Pielot & Boll, 2010) or obstacles in the person’s environment (Froese et al., 2012; Johnson & Higgins, 2006; Maidenbaum et al., 2014; Sohl-Dickstein et al., 2015). Comparative studies would be necessary to determine which device is most useful in which particular application context.

4.2. Automaticity

We investigated the level of automaticity achieved after training as a necessary precondition for the development of perceptual feel. Overall, the observed patterns of results are small, and ambiguous with respect to automatization.

On the one hand, a few results might reflect automaticity. Response times in the Sound-to-Word tasks slightly decreased when changing the coding in the rotated-signal test (Figure 9(b)). Average errors (Figure 9(b), right) and performance in the Sound-to-Pointing task (Figure 9(a), left) went in the same direction but were not significant. Participants were slightly better in the distortion-detection task after than before training (Figure 10). In addition, the responses to the post-experimental questionnaire suggested that there may be some effects on the participants’ subjective impression that may possibly reflect automatization (Figure 13, see also Table S12).

On the other hand, other results do not support the idea that the NaviEar became automatic and internalized through training. There was no evidence for interference effects (Figure 12). Although the NaviEar improved veering, this effect did not increase with training, contrary to what we predicted in case of increasing automatization through training (Figure 11). Instead, this pattern of results may stem from participants using the NaviEar like an auditory compass, that is, by cognitively translating the signal into information about orientation (Deroy & Auvray, 2012).

The weak effects in the rotated-signal task could be the result of rapid adjustment based on cognitive interpretation. If users can easily interpret the NaviEar signal without intensive learning, they might be able to almost directly adapt to the rotation of the signal. This would explain why performance in the rotated-signal test was close to the development of performance with the unrotated signal across sessions, at least in the Word-to-Pointing and Sound-to-Pointing tasks (Figure 8(a) and (b)). In addition, subjective reports from the post-experimental questionnaire need to be treated with caution, as argued in the Introduction. This is particularly true because the pre-post questions required several days of hindsight.

At the same time, a lack of significant effects in the rotated-signal and the interference test may be due to the small number of observers and a lack of statistical power. Although, the sample size in our study was comparable to some related previous studies (Cassinelli et al., 2006; Faugloire & Lejeune, 2014; Froese et al., 2012; Kärcher et al., 2012; Kaspar et al., 2014; König et al., 2016; Nagel et al., 2005), it was lower than in our laboratory study, which did observe interference effects (n = 23 and 15 in Schumann & O'Regan, 2017). It is possible that more evidence for automaticity could be obtained if we had the opportunity to measure a larger sample of participants.

However, if a genuine perceptual feel of north had developed through NaviEar training, we would expect strong effects of automatization: Participants should detect distortions with high reliability (∼90%), make more errors in all rotated-signal than in the respective training tasks, and show strong interference after training. Even if the few observed patterns were definitely due to automatization, the magnitude of these effects are in no way close to genuine perceptual feel, which occurs in a few milliseconds (cf. Introduction, e.g., Potter et al., 2014). Hence, our findings rather indicate that participants did not learn and internalize the sensorimotor contingencies conveyed by the NaviEar in a way that would allow for perceptual feel according to sensorimotor theory. In this respect, these results are similar to those obtained with the feelSpace belt that did not find evidence for automatic (or subcognitive) use, either (Kaspar et al., 2014; König et al., 2016; Nagel et al., 2005).

Previous studies did not specifically investigate automaticity but provide data that is potentially interesting in the quest for automaticity. For example, several studies found that skilled use of sensory augmentation devices does not interfere with other tasks that required the user’s cognitive resources (Elliott et al., 2010; Pielot & Boll, 2010; Van Erp & Werkhoven, 2006). While those observations support the practical advantage of the respective devices in the particular context under investigation, the tests in those studies were not specific for automatic processing.

The observation that the interpretation of the vOICe signal might become insensitive to attentional load from a distractor task (Stiles & Shimojo, 2015) provides evidence for automatic processing. Similarly, our study in the lab showed automatic interference of the augmented signal (hearSpace) with perceptual orientation (Schumann & O'Regan, 2017). In contrast to our present study, the first of those studies (Stiles & Shimojo, 2015) used stimulus images chosen to have pre-existing cross-modal mappings, and the hearSpace signal (Schumann & O'Regan, 2017) mimics the natural low-level spatial contingencies of distal sounds. Thus, it is not clear to which degree the observed automaticity reflects learning or preexisting mechanisms. In addition, the above two studies were conducted in the laboratory. It is not clear how their results transfer to the more complex situations and activities in the field.

The lack of automaticity in our study may be because of insufficient or unsuccessful training. In contrast to the experiment in the laboratory (Schumann & O'Regan, 2017), field experiments and training depend on the participants activities and their context, which prevents a control of all potentially interfering factors (cf., e.g., Kärcher et al., 2012; Kaspar et al., 2014; Nagel et al., 2005). Post-trial subjective ratings indicated that loudness of environmental noise reduced performance in training tasks that required listening to the NaviEar signal (Sound-to-Pointing and Sound-to-Word). This interference with training may have reduced learning the sensorimotor contingencies. Although familiarity was not related to training performance, we cannot fully exclude that some participants might have used other kinds of information that would undermine training success. For example, participants could have used the position of the sun (in relation to time) to orient themselves in the Word-to-Pointing and Sound-to-Pointing task without listening to the NaviEar. Nevertheless, the observation that participants performed much better with than without the NaviEar in those tasks (Figure S5) suggests that they did engage with the NaviEar signal, no matter the potential effects of environmental noise and other factors.

Furthermore, participants had completed unequal amounts of training sessions (between 6 and 10, cf. Table 1), making it difficult to compare learning success across individual participants. Generally, the amount of training necessary to reach learning effects can strongly differ across individuals (e.g., Dale et al., 2021; Katz et al., 2021; Traut et al., 2021; Wen et al., 2021). So, a fixed amount of training alone would not guarantee that participants underwent sufficient training to reach automaticity. The question is whether the training was above the minimum needed to expect automatization in all participants.

Unfortunately, we cannot know a priori how much training is needed individually or on average. The training period of 6–10 sessions might have been too short to acquire automatization, especially in the comparatively complex context of the field study (Ashby & Crossley, 2012; Bertram & Stafford, 2016). Correlations between measures of automaticity, numbers of sessions, and learning success provided some indication that amount of training (number of sessions) and learning (slopes) might be related to performance in the veering and the interference test. A definite conclusion cannot be drawn here because results are inconsistent across tasks and measures, and the statistical power is low due to the small number of participants. So, we cannot exclude that the lack of automaticity effects is due to insufficient learning. We might obtain much higher levels of automaticity when users train for weeks, months, or even years (e.g., Sandlin, 2015; Ward & Meijer, 2010).

However, our device conveyed relatively simple signals that are easy to interpret. This idea is supported by the high usability of the NaviEar, as discussed in the previous section. Devices that require the interpretation of more complex signals, for example, when translating visual images into sound (e.g., Bologna et al., 2009) or touch (e.g., Bach-y-Rita et al., 1969), likely require much higher amounts of training to make any potential automatization possible. At the same time, it is also possible that, in contrast to sensory augmentation, automatization in sensory substitution, for example, of vision by sound, could benefit from predisposed brain structures that may be recruited by the substituting signal (e.g., Poirier, De Volder, & Scheiber, 2007).

Another factor that could have hampered automatization in our study is that the geomagnetic allocentric information transmitted by the NaviEar might be of minor importance for orientation in an urban environment. We tried to reinforce participants’ engagement with this novel information through the training tasks. However, it would be beneficial to further improve the training tasks to be more meaningful in the context of everyday life activity (Bertram & Stafford, 2016; Lenay et al., 2003).

4.3. Test battery for automaticity

Although, we focused on the implications of sensorimotor theory, the observation that perception is immediate and automatic does not depend on specific theories about the nature of perceptual experience. The assumption that perception is automatic does neither imply that all automatic responses are perceptual, nor that the development of perceptual experience first transitions through a stage of non-perceptual automaticity, as suggested by our Figure 1. For example, this assumption is in line with representational (Marr, 2010 [1982]), Bayesian (Geisler, 2011; Hillis et al., 2002), ecological (Greeno, 1994), empirical (Purves et al., 2001, 2015), and predictive coding (Friston, 2018; Seth, 2014; Vance & Stokes, 2017) accounts of perception. Hence, the idea of testing for automaticity as a necessary criterion for changes in perceptual experience is not limited to a specific theoretical framework and may be used as a more general, methodological approach.

As already noted in the introduction, to evaluate how participants use the respective device, most studies in sensory substitution and augmentation rely exclusively on simple performance measures (e.g., Bologna et al., 2009; Brown et al., 2011; Durette et al., 2008; Proulx et al., 2008), subjective reports (e.g., Guarniero, 1974; Kaspar et al., 2014; Ward & Meijer, 2010), or both (e.g., Auvray et al., 2007; Bach-y-Rita et al., 1969; Bermejo et al., 2015; Haigh et al., 2013; Nagel et al., 2005; Segond et al., 2005). However, these measures do not allow for firm conclusions about the emergence of perceptual feel through substitution or augmentation devices. Any intensive training leads to some improvement in performance. Such improvements do not necessarily imply automaticity or perceptual feel in the use of the devices. This may also be visible in our results. While first-person subjective reports are important for evaluating perceptual feel, such reports strongly depend on how users interpret the testing situation and the experimenter’s expectations. Nevertheless, many studies have at least implicitly suggested that their measures produced a genuine perceptual feel or changes in perceptual awareness (Auvray et al., 2007; Bach-y-Rita et al., 1969; Bach-y-Rita et al., 1998; Bach-y-Rita & Kercel, 2003; Bermejo et al., 2015; Capelle et al., 1998; Durette et al., 2008; Haigh et al., 2013; Kaspar et al., 2014; König et al., 2016; Kupers & Ptito, 2004; Meijer, 1992; 2015; Nagel et al., 2005; Poirier et al., 2007; Proulx, 2010; Segond et al., 2013; Ward & Meijer, 2010; Ward & Wright, 2014; White et al., 1970). Given the insufficiencies of the measures used, claims and conclusions about changes in perceptual feel or awareness are premature and unjustified (Auvray & Harris, 2014; Block, 2003; Brown et al., 2011; Deroy & Auvray, 2012).

We suggest that studies should show high levels of automaticity as a necessary—but not sufficient—criterion for perceptual feel. As mentioned above (section 4.2), several studies have measured some aspects of automaticity in the use of sensory augmentation and sensory substitution devices (Elliott et al., 2010; Kaspar et al., 2014; König et al., 2016; Nagel et al., 2005; Pielot & Boll, 2010; Schumann & O'Regan, 2017; Stiles & Shimojo, 2015; Van Erp & Werkhoven, 2006). However, to our knowledge, the test battery we developed here is the first that systematically tests not one but a range of criteria for automatization in the use of sensory augmentation and substitution devices.

Our test battery could serve as a model for investigating automaticity with sensory augmentation and sensory substitution devices. For example, following our work an ongoing study (Negen et al., 2021) implemented tests of automaticity to investigate auditory cues to distance, similar to echolocation.

However, the individual tests of the battery we proposed need improvement in their technical implementation. For example, a technical improvement would be to conduct the tests of the battery in a more controlled setting rather than in a park or to implement them in a virtual environment (e.g., Negen et al., 2021) to better control confounding factors and measurement noise. Moreover, it would be important to optimize and benchmark the sensitivity of the tests of automaticity we propose. The single tests in this battery might also need some adjustments to match the specificities of other kinds of sensory augmentation and substitution.

The test battery can also be extended with additional tests. Taking inspiration from the studies mentioned above (Elliott et al., 2010; König et al., 2016; Stiles & Shimojo, 2015), an interesting additional test could involve a distractor task to assess efficiency and cognitive load as a further criterion of automaticity. If the interpretation of the augmenting signal becomes automatized, it should not produce distraction due to dividing attention (e.g., Wickens, 2021). Negen et al. (2021) propose a test for “redundant signal effects” to test whether the augmenting signal (e.g., the auditory cues to distance, or here the NaviEar signal) becomes integrated in the augmented signal (e.g., visual cues about distance or orientation). Such an effect implies that participants are faster to respond to the combination of the augmented and augmenting signals than to each of these signals alone.

Another way to extend the test battery could include tests of distal attribution (Auvray et al., 2005; Hartcher-O’Brien & Auvray, 2014; Siegle & Warren, 2010; Visell, 2008). In the case of the NaviEar, a perceptual feel of north might be considered a case of distal attribution in that the proximal auditory signal in the ear would be attributed to the distal cardinal directions in the environment. However, known measures of distal attribution in sensory substitution rely on subjective verbal reports (for review see, e.g., Hartcher-O’Brien & Auvray, 2014), hence eluding our aim of establishing objective tests of automaticity. Nevertheless, paradigms for measuring perceptual constancy may well serve as objective tests for distal attribution in some sensory substitution or augmentation devices. For example, it has been proposed to test for size constancy in sensory substitution (Renier et al., 2005, 2006). Another example is color constancy, where observers perceive the color of a surface, that is, the distal stimulus, as constant across illumination changes even though the proximal stimulus on the retina changes with illuminations (for review, see, e.g., Witzel & Gegenfurtner, 2018). Using established paradigms of color constancy, one could test whether the interpretation of a substitution signal will reflect the proximal or distal color stimulus after sustained training with color-substitution devices (for review, see Hamilton-Fletcher & Ward, 2013).