Multimodal spatial language maps for robot navigation and manipulation

2.1. Semantic mapping

In recent years, the synergy of the traditional SLAM techniques and the advancements in vision-based semantic understanding has led to augmenting 3D maps with semantic information (McCormac et al., 2017; Salas-Moreno et al., 2013). Stemming from the intuition of augmenting 3D points in the map with 2D segmentation results, previous works focus on either abstracting the map at object-level with a pose graph (McCormac et al., 2018) or an octree (Xu et al., 2019) or modeling the dynamics of objects in the map (Runz et al., 2018). Despite lifting the 3D reconstruction to a semantic level, these methods are restricted to a predefined set of semantic classes. Recent works like LM-Nav (Shah et al., 2023), CoW (Gadre et al., 2023), VLMaps (Huang et al., 2023b), NLMap-SayCan (Chen et al., 2023a), OpenScene (Peng et al., 2023), or CLIP-Fields (Shafiullah et al., 2023) have shown that integrating visual-language features, generated by either pre-trained or fine-tuned models, into a topological graph or an occupancy map enables open-vocabulary object indexing with natural language, freeing the maps from fixed-size semantic categories. Recent approaches also investigate other open-vocabulary map representations such as NeRF (Engelmann et al., 2024; Kerr et al., 2023; Kim et al., 2024), Gaussian Splattings (Qin et al., 2024; Zuo et al., 2024), and Scene Graphs (Gu et al., 2024; Werby et al., 2024). However, these works focus on visual perception to map and move through an environment, overlooking complementary sources of information such as acoustic signals. In contrast, AVLMaps integrate audio, visual, and language cues into a 3D map, equipping the agent with the ability to navigate to multiple types of multimodal goals and effectively disambiguate goals.

2.2. Vision and language navigation

Recently, also Vision-and-Language Navigation (VLN) has received increased attention (Anderson et al., 2018b; Krantz et al., 2020). Further work has focused on learning end-to-end policies that can follow route-based instructions on topological graphs of simulated environments (Anderson et al., 2018b; Fried et al., 2018; Guhur et al., 2021). However, agents trained in this setting do not have low-level planning capabilities and rely heavily on the topological graph, limiting their real-world applicability (Anderson et al., 2021). Moreover, despite extensions to continuous state spaces (Hong et al., 2022; Krantz et al., 2020, 2021), most of these learning-based methods are data-intensive. The recent success of large pretrained vision and language models (Brown et al., 2020; Radford et al., 2021) has spurred a flurry of interest in applying their zero-shot capabilities to open-vocabulary object navigation (Gadre et al., 2023; Shah et al., 2023). LM-Nav (Shah et al., 2023) combines three pre-trained models to navigate via a topological graph in the real world. CoW (Gadre et al., 2023) performs zero-shot language-based object navigation by combining CLIP-based (Radford et al., 2021) saliency maps and traditional exploration methods. However, both methods are limited to navigating to object landmarks and are less capable of understanding finer-grained queries, such as “to the left of the chair” and “in between the TV and the sofa.” In contrast, our method, VLMaps, enables spatial language indexing beyond object-centric goals and can generate open-vocabulary obstacle maps. Our extension, AVLMaps, further enables multimodal spatial concept indexing such as “between the image of a refrigerator and the sound of breaking glass.”

2.3. Multimodal navigation

Recent advances in simulation applications (Chen et al., 2020; Gan et al., 2020a; Kolve et al., 2017; Savva et al., 2019) have boosted research on multimodal navigation in two distinct directions: (i) vision-and-language navigation (VLN) (Anderson et al., 2018b; Krantz et al., 2020) where an agent needs to follow a natural language instruction toward the goal with visual input, and (ii) audio-visual navigation (AVN) (Chen et al., 2020) in which an agent should navigate to the sound source based on information from a binaural sensor and vision. Despite different degrees of success in both directions (Chen et al., 2021a; Fried et al., 2018; Gan et al., 2020b, 2022; Guhur et al., 2021; Younes et al., 2023), less attention has been paid to solving the navigation problem involving vision, language, and audio at the same time. The most relevant concept to our knowledge is from AVLEN (Paul et al., 2022), which extends the AVN with a further query step, introducing a language instruction that helps with navigating to the sound source. In addition, most of the existing methods on AVN focus on approaching the sound without understanding its semantics. In our work, we propose a method to integrate both visual and sound semantics into the same map, enabling a robot to navigate to multimodal goals specified with either goal image or natural language like “go to the sound of baby crying,” “go to the table” or multimodal prompts such as “go to the {image of a table} where the sound of the microwave was heard.”

2.4. Pre-trained zero-shot models in robotics

Recent trends have shown that pre-trained foundation models (Brown et al., 2020; Liang et al., 2023a; Radford et al., 2021) serve as powerful tools for robotic tasks. Most works exploit the inherent perception and reasoning abilities of Vision Language Models (VLMs) or Large Language Models (LLMs) trained with cloud-sourced data to boost the performance of robot tasks including object detection and segmentation (Gu et al., 2021; Kamath et al., 2021; Li et al., 2021), robot manipulation (Chen et al., 2024a; Liang et al., 2023b; Mees et al., 2022a, 2022b, 2023; Rosete-Beas et al., 2022; Shridhar et al., 2022; Zawalski et al., 2024; Zhou et al., 2024), and navigation (Chen et al., 2023b; Gadre et al., 2023; Gu et al., 2024; Hirose et al., 2024; Huang et al., 2023b; Shah et al., 2023; Werby et al., 2024). With access to a growing volume of robot control data annotated with semantic language labels, recent approaches have advanced the training of a Vision-Language-Action (VLA) model, bridging the gap between visual-language comprehension and low-level robot control within a unified foundational model (Brohan et al., 2023; Doshi et al., 2024; Kim et al., 2025; Octo Model Team et al., 2024; O’Neill et al., 2024; Zitkovich et al., 2023). Despite promising results from previous methods, little effort has been made to exploit the audio-language pre-trained models (ALMs) (Elizalde et al., 2023; Guzhov et al., 2022) in robotic tasks. In this work, we leverage the foundation models focusing on different modalities, for example, audio, language, and vision, to create a mapping pipeline to understand multimodal information in the scene and achieve robot navigation given language, image, or audio description queries. Concurrent work ConceptFusion (Jatavallabhula et al., 2023) demonstrates that audio can be used as queries to index locations in a visual-language map. However, it doesn’t support integrating audio information from the observation data into the representation of the scene as we do in this work.

3.1. Building a visual-language map

The key idea behind VLMaps is to fuse pretrained visual-language features with a 3D reconstruction. We achieve this by computing dense pixel-level embeddings from an existing visual-language model (over the video feed of the robot) and by back-projecting them onto the 3D surface of the environment (captured from depth data used for reconstruction with visual odometry). The overview of VLMaps creation is shown on the left of Figure 2.

In our work, we utilize LSeg (Li et al., 2021) as the visual-language model, a language-driven semantic segmentation model that segments the RGB images based on a set of free-form language categories. The LSeg visual encoder maps an image such that the embedding of each pixel lies in the CLIP feature space. In our approach, we fuse the LSeg pixel embeddings with their corresponding 3D map locations. In this way, without explicit manual segmentation labels, we incorporate a powerful language-driven semantic prior that inherits the generalization capabilities of VLMs. The only assumption we make is access to odometry, which is readily available from RGB-D SLAM systems and enables us to build a map from sequences of RGB-D images.

Formally, we define VLMap as M ∈ R H ¯ × W ¯ × C , where H ¯ and W ¯ represent the size of the top-down grid map, and C represents the length of the VLM embedding vector for each grid cell. Together with the scale parameter s (meters per pixel), a VLMap M represents an area with size s H ¯ meters × s W ¯ meters . To build the map, for each RGB-D frame, we back-project all the depth pixels u = (u, v) to form a local depth point cloud that we transform to the world frame, P k = D ( u ) K − 1 u ˜ and P _W = T _Wk P _k where u ˜ = ( u , v , 1 ) , K ∈ R 3 × 3 is the intrinsic matrix of the depth camera, D ( u ) ∈ R is the depth value of the pixel u, T _Wk is the transformation from the world coordinate frame to the kth camera frame, P k ∈ R 3 is the 3D point position in the kth frame, and P W ∈ R 3 is the 3D point position in the world coordinate frame. We then project the point P _W to the ground plane and get the pixel u’s corresponding position on the grid map,

p map x = ⌊ H ¯ 2 + P W x s + 0.5 ⌋ , p map y = ⌊ W ¯ 2 − P W z s + 0.5 ⌋

(1)

Once we build the grid map, we apply LSeg’s visual encoder f ( I ) : R H × W × 3 → R H × W × C to the RGB image I k and generate the pixel-level embedding F k ∈ R H × W × C . Given the RGB-D registration, we project each image pixel u’s embedding q = F k ( u ) ∈ R C to its corresponding grid cell location ( p map x , p map y ) in the top-down grid map. Intuitively, there exist multiple 3D points projecting to the same grid location in the map. Thus, we average their embeddings, M ( p map x , p map y ) = 1 / n ∑ i = 1 n q i where M ( p map x , p map y ) ∈ R C represents the map features at the grid position ( p map x , p map y ) , n represents the total number of points projecting to the grid location ( p map x , p map y ) , and q i ∈ R C denotes the corresponding pixel embedding of each point. We note that these n points might not only come from a single frame, but also from points from multiple frames. Therefore, the resulting features contain the averaged embeddings from multiple views of the same object.

3.2. Localizing open-vocabulary landmarks

We now describe how to localize landmarks in VLMaps with free-form natural language. The overview of the indexing process is shown on the right of Figure 2. Formally, we define the input language list as L = [ l 0 , l 1 , … , l M ] where l _i represents the i-th category in text form, and M represents the number of categories defined by the user. Some examples of the input language list are [“chair,” “sofa,” “table,” “other”] or [“furniture,” “floor,” “other”]. As Li et al. (2021), we apply the pre-trained CLIP text encoder (Radford et al., 2021) to convert such list of texts into a list of vector embeddings [ e 0 , e 1 , … , e M ] , e ∈ R C , which are organized into an embedding matrix E ∈ R M × C , where each row of the matrix represents the embedding of a category. The map embeddings M are also flattened into a matrix Q ∈ R H ¯ W ¯ × C , where each row represents the embedding of a pixel in the top-down grid map. We then compute the pixel-to-category similarity matrix S = Q ⋅ E ^T , where S ∈ R H ¯ W ¯ × M . Each element S _ij in the matrix stores the similarity value between a pixel and a text category, indicating how likely this pixel belongs to the class. By applying the argmax operator along the row direction to S and reshaping the resulting vector to shape H ¯ × W ¯ , we get the final segmentation result R ∈ R H ¯ × W ¯ . Each element R _ij represents the label index of the input language list L at the grid map location (i, j). With the final resulting matrix R, we compute the most related language-based category for every pixel in the grid map.

3.3. Generating open-vocabulary obstacle maps

Building a VLMap enables us to generate obstacle maps that inherit the open-vocabulary nature of the VLMs used (LSeg and CLIP). Specifically, given a list of obstacle categories described with natural language, we can localize those obstacles at runtime to generate a binary map for collision avoidance and/or shortest path planning, as is shown in Figure 3. A prominent use case for this is sharing a VLMap of the same environment between different robots with different embodiments (i.e., cross-embodiment problem (Ganapathi et al., 2022; Zakka et al., 2022)), which may be useful for multi-agent coordination (Wu et al., 2021). For example, a large mobile robot may need to navigate around a table (or other large furniture), while a drone can directly fly over it. By simply providing two different lists of obstacle categories—one for the large mobile robot (that contains “table”), and another for the drone (that does not), we can generate two distinct obstacles maps for the two robots to use, respectively, sourced on-the-fly from the same VLMap.OPEN IN VIEWER

To do so, we first extract an obstacle map O ∈ { 0,1 } H ¯ × W ¯ where each projected position of the depth point cloud in the top-down map is assigned 1, and otherwise 0. To avoid points from the floor or the ceiling, points P _W are filtered out depending on their height,

O i j = 1 , t 1 ≤ P W y ≤ t 2 a n d p map x = i a n d p map y = j 0 , otherwise

(2)

3.4. Zero-shot spatial goal navigation from language

In this section, we describe our approach to long-horizon (spatial) goal navigation, given a set of landmark descriptions specified by natural language instructions such as

Notably different from prior work (Gadre et al., 2023; Shah et al., 2023), VLMaps allow us to reference precise spatial goals such as: “in between the sofa at the TV” or “three meters to the east of the chair.” Specifically, we use a large language model (LLM) to interpret the input natural language commands and break them down into subgoals (Ahn et al., 2022; Shah et al., 2023; Zeng et al., 2023). In contrast to prior work, which may reference these subgoals with language and map to low-level policies with semantic translation (Huang et al., 2022a) or affordances (Ahn et al., 2022; Huang et al., 2022b; Zeng, 2019), we leverage the code-writing capabilities of LLMs to generate executable Python robot code (Brown et al., 2020; Chen et al., 2021b; Liang et al., 2023b; Mees et al., 2023) that can (i) make precise calls to parameterized navigation primitives, and (ii) perform arithmetic when needed. The generated code can directly be executed on the robot with the built-in Python exec function.

Note that recent works (Brown et al., 2020; Chen et al., 2021b; Liang et al., 2023b; Mees et al., 2023) have shown that code-writing language models (e.g., Codex (Chen et al., 2021b)) trained on billions of lines of code from Github can be used to synthesize new simple Python programs from docstrings. In this work, we re-purpose these models for mobile robot planning by priming them with several input examples of natural language commands (formatted as comments) paired with corresponding robot code (via few-shot prompting). The robot code can express functions or logic structures (if-then-else statements or for/while loops) and parameterize API calls (e.g., move_to(target_name) or turn(degrees). The full list is available in Table 1) that map to spatial behaviors specified by the language commands. The full prompt is shown in Figure 4.

OPEN IN VIEWER

Table 1.

Navigation API library for spatial goal navigation in VLMaps.

APIs	Functions
move_to(pos)	Move to a position on the map
move_to_left(object_name)	Move to the left side of the nearest front object
move_to_right(object_name)	Move to the right side of the nearest front object
with_pos_on_left(object_name)	Turn until the object is on the robot’s left side
with_pos_on_right(object_name)	Turn until the object is on the robot’s right side
move_in_between(object_a, object_b)	Move in between two objects
face(object_name)	Turn until the robot’s front is pointing to the object
turn(angle)	Turn right a certain angle. If the angle value is negative, turn left
turn_absolute(angle)	Turn to absolute angle. 0 is north, 90 is east, −90 is west, 180 is south
move_north(object_name)	Move to the north side of the object
move_south(object_name)	Move to the south side of the object
move_east(object_name)	Move to the east side of the object
move_west(object_name)	Move to the west side of the object
move_forward(dist)	Move forward “dist” meters

OPEN IN VIEWER

Figure 4.

The full context prompt (prompt in gray) VLMap used for achieving spatial goal navigation tasks in the experiments.

Spatial goals are defined as positions around the reference object based on spatial descriptions, for example, “in the middle of the counter and the fridge” or “to the left of the sofa”. Traditional object goal navigation methods either directly retrieve the target object’s location on the map and plan to it (Gadre et al., 2023; Shah et al., 2023) or are trained to approach objects as a reactive system (Chaplot et al., 2020). These methods fall short in reaching spatial goals since these goal locations are free space rather than retrievable locations on semantic maps. However, when we know the reference position, those spatial locations can be computed with simple offsets. The navigation primitive functions (APIs) being called by the language model (e.g., move_to_left(‘counter’)) use a pre-generated VLMap to localize the coordinates of the open-vocabulary landmarks (“counter”) in the maps (described in Section 3.2) modified with predefined scripted offsets (to define “left”). We then navigate to these coordinates using an off-the-shelf navigation stack (Quigley, 2009) that takes as input the embodiment-specific obstacle map (generated using the same VLMap, with the process described in Section 3.3). Some examples of spatial goal navigation in the real world are shown in Figure 5.OPEN IN VIEWER

At test time, the LLM is prompted with context examples (in gray) in Figure 4 as well as commands (in green) in Figure 6. It can autonomously re-compose API calls to generate new robot code that not only references the new landmarks mentioned in the language commands (as comments), but also can chain together new sequences of API calls to follow unseen instructions accordingly. The inference process is shown in Figure 6, and generated outputs are highlighted.OPEN IN VIEWER

3.5. Building an audio visual language map

VLMaps provide an intuitive interface for humans to issue natural language commands to robots, enabling open-vocabulary spatial goal navigation. However, interacting with the world is inherently a multimodal experience. Since VLMaps primarily rely on visual information for map construction, it is important to consider other complementary sources of information that can enhance navigation. Motivated by this, we broaden the scope of VLMaps and propose a more general framework, AVLMaps, which enables robots to integrate and interpret multimodal information, including audio, images, and language, within a unified representation (as illustrated in Figure 7). In this paper, we present the extension to audio and image modalities as a case study, showcasing the modularity and extensibility of our multimodal spatial language maps. The AVLMaps framework is flexible and designed to accommodate future integration of additional sensing modalities, such as temperature, tactile feedback, or magnetic fields. The key idea behind AVLMaps is to combine visual localization features, pre-trained visual-language features, and audio-language features with a 3D reconstruction. Given an RGB-D video stream with an audio track and odometry information, we utilize four modules to build a multimodal features database. Given a specific query, each module returns predicted spatial locations on the map in the form of 3D voxel heatmaps. A heatmap can be denoted as H ∈ [ 0,1 ] H ¯ × W ¯ × Z ¯ , where H ¯ , W ¯ , and Z ¯ represent the size of the voxel map and the value in each element represents the probability of being the target position. p = ( x , y , z ) T , { x , y , z ∈ Z ∣ 1 ≤ x ≤ H ¯ , 1 ≤ y ≤ W ¯ , 1 ≤ z ≤ Z ¯ } is a voxel position in the map H .OPEN IN VIEWER

3.5.1. Visual localization module

The overview of the module is shown in Figure 8. The main purpose of this module is to localize a query image in our map. To achieve this goal, we follow a hierarchical localization scheme (Sarlin et al., 2019, 2020). We first compute the NetVLAD (Arandjelovic et al., 2016) global descriptors and SuperPoint (DeTone et al., 2018) local descriptors for all images from the RGB stream during exploration and store them with the corresponding depth and odometry as a reference database. During inference, we compute the global and local descriptors for the query image in the same manner. By searching the nearest neighbor of the query NetVLAD features in the reference database, we can find a reference image as our candidate. Next, we use SuperGLUE (Sarlin et al., 2020) to establish key point correspondences between the query image and the reference image we retrieve with NetVLAD from the database with their local SuperPoint features. With registered depth, we back-project the reference image’s key points into the 3D space and obtain the 3D–2D correspondences for the query key points. In the end, we can apply the Perspective-n-Point method (Fischler and Bolles, 1981) to estimate the query camera pose relative to the reference camera, and thus obtain the global camera pose with the odometry of the reference camera.OPEN IN VIEWER

Figure 8.

The overview of the visual localization module. We follow the hierarchical localization scheme by using NetVLAD and SuperGLUE to localize the query image’s location before generating a heatmap as the indexing result.

In the visual localization module, the predicted global camera location is denoted as p v = ( x v , y v , z v ) T . In the heatmap H v , we define the probability at p _v as 1.0, and the probability linearly decays around this location according to the distance on the top-down map:

H v ( p ) = m a x ( 1.0 − ϵ ⋅ d i s t x y ( p , p v ) , 0 )

(3)

d i s t x y ( p , q ) = ( p x − q x ) 2 + ( p y − q y ) 2

(4)

where ϵ is the decay rate, and dist _xy (p, q) denotes the distance between 3D vectors p and q on the xy-plane.

3.5.2. Object localization module

The overview of the object localization module is shown in Figure 9. This module is abstracted from the VLMaps we introduced in Section 3.1, Section 3.2, and Section 3.3 with some minor changes. The key idea is to exploit an open-vocabulary segmentation method (e.g., LSeg (Li et al., 2022) or OpenSeg (Ghiasi et al., 2022)) for pixel-level feature generation from the RGB image and to associate these features with the back-projected depth pixels in the 3D reconstruction. Different from Section 3.1, we don’t project the 3D points into the top-down plane to create a 2D grid map but maintain a 3D voxel map where each voxel is associated with a visual-language feature. When there are multiple points projected into the same voxel, we store their mean features at the voxel. The inference process is similar to Section 3.2. We define a list of categories in natural language and encode them with the language encoder. We compute the cosine similarity scores between all voxel-wise features and language features and use an argmax operator to select the top-scoring voxels for a certain category in the map. Depending on the application, the top-scoring 3D voxel points for a certain category can be used as the target point cloud for manipulation tasks or can be projected onto a top-down map for navigation purposes.OPEN IN VIEWER

Figure 9.

The overview of the object localization module. Similar to Section 3.1, during mapping, the RGB images in the exploration video are input to a vision-language model, LSeg (Li et al., 2021), to generate pixel-level features. Corresponding pixels are back-projected with depth images and transformed to locations in the global coordinate frame, where the features are associated with. During inference, the query text is encoded by LSeg’s text encoder, and a dot product with the point-level embeddings generates a score for each point. A heatmap is then created based on point scores and distances.

The object localization results are a list of points, denoted as {p _oi = (x _oi , y _oi , z _oi )∣i = 1, …, N} where N is the total number of points for the target object. We define the probabilities for all these locations as 1.0 in heatmap H o , and the probability linearly decays around these locations based on the Euclidean distance:

d min ( p ) = m i n { d i s t ( p , p o i ) ∣ i = 1 , … , N }

(5)

H o ( p ) = m a x ( 1.0 − ϵ ⋅ d min ( p ) , 0 )

(6)

where d_min(p) denotes the minimal distance between p and all object points {p _oi ∣i = 1, …, N}, dist (p, q) denotes the Euclidean distance between p and q.

In AVLMaps, the object localization module is also used to generate a 2D obstacle grid map for path planning, as is shown in Section 3.3. Different from the 2D feature grid map of a VLMap, the visual language map is now in the form of a 3D voxel grid where each occupied voxel is associated with a feature. We prompt the map with a list of free area concepts (e.g., “floor”) and obstacle concepts (e.g., “chair,” “table,” “counter,” and “other”) for a specific embodiment, assigning a score to each concept for every voxel. Each voxel is then labeled with the concept that achieves the highest score, resulting in a 3D semantic voxel map. We merge the voxels labeled with obstacle concepts into a combined obstacle map, and then perform a top-down projection to produce a 2D obstacle grid map. In this grid, all pixel locations corresponding to projected obstacle voxels are marked as occupied, while all others are marked as free and navigable. It is worth noting that beyond object categories, AVLMaps also support the definition of audio or images as obstacles. For example, we can define “the sound of glass breaking,” or a list of images as obstacles, generate their corresponding heatmaps, and treat locations with heat above a certain threshold as obstacles. This capability is useful when the forbidden regions are hard to describe with only object descriptions, like a glass-breaking scene without surrounding objects, or a region specified with a cell phone video.

3.5.3. Area localization module

While the object localization module is good at extracting object segments on the map, it falls short of localizing coarser goals such as regions (e.g., “the area of the kitchen”). This is because the visual encoder for generating pixel-aligned features is obtained by fine-tuning a pre-trained model on a segmentation dataset, leading to the notorious catastrophic forgetting effect. Therefore, the visual encoder is better at segmenting common objects while worse at recognizing general visual concepts (Jatavallabhula et al., 2023). To take advantage of both pre-trained and fine-tuned methods, we propose to build an extra sparse topological CLIP features map similar to (Shah et al., 2023). The idea is to compute the CLIP visual features (Radford et al., 2021) for all images from the RGB stream and associate the features with corresponding poses. During inference, given the language concept like “the area of a bedroom,” we compute the language features with the CLIP language encoder and the image-to-language cosine similarity scores. These similarity scores indicate how likely these images match the language description. The odometry together with the score of each image indicates the predicted location with a confidence value.

The area localization results are a list of position-confidence pairs, denoted as {(p _ai , s _ai )∣i = 1, …, M} where M is the total number of frames in the input RGB-D stream. The scores s _ai are normalized between 0 and 1. We define the probability for each point p _ai on the heatmap H a as its score s _ai , and the probability linearly decays around the point on the xy-plane direction:

H ¯ a ( p ) = max i = 1 , … , M { s a i − ϵ ⋅ d i s t x y ( p , p a i ) }

(7)

H a ( p ) = max ( H ¯ a ( p ) , 0 )

(8)

where the max operator for the curly brackets means taking the highest probability when a location is inside the affected regions for several p _ai .

3.5.4. Audio localization module

The overview of the module is shown in Figure 10. In this module, we utilize the audio information from the input stream. The key idea is to compute the audio-lingual features with audio-language pre-trained models such as wav2clip (Wu et al., 2022), AudioCLIP (Guzhov et al., 2022) or CLAP (Elizalde et al., 2023). We first segment the whole audio clip into several segments with silence detection. Whenever the volume is above a threshold, we mark this time step as the starting point of a segment. Whenever the volume of the sound is not larger than this threshold for a certain duration, we end the segment. In the next step, we compute the audio features for each segment with pre-trained audio-language models and associate the features with the odometry at the specific segment. During inference, given a language description of the sound, like “the sound of door knocks,” we encode the language into language features and compute the matching scores between the language and all audio segments in the same way as in the object localization module. The odometry associated with the top-scoring segment is the predicted location.OPEN IN VIEWER

Figure 10.

The overview of the audio localization module. During mapping, the exploration video’s audio is segmented by silence and encoded using AudioCLIP’s audio encoder. The resulting embeddings are linked to poses based on their timestamps. During inference, the query text is encoded by AudioCLIP’s text encoder, and a dot product with the pose embeddings generates a score for each pose. A heatmap is then created based on pose scores and distances.

The audio localization results are similar to those of the area localization module. The position-score pairs are denoted as {(p _si , s _si )∣i = 1, …, K} where K is the total number of sound segments in the input video stream. The heatmap H s is defined as:

H ¯ s ( p ) = max i = 1 , … , K { s s i − ϵ ⋅ d i s t x y ( p , p s i ) }

(9)

H s ( p ) = max ( H ¯ s ( p ) − ϵ ⋅ d i s t x y ( p , p s i ) , 0 )

(10)

3.6. Cross-modality reasoning

A key advantage of our method is its capability to disambiguate goals with additional information, even from different modalities. The goal of the cross-modality reasoning method is to output a target location of a specific concept (e.g., “the sofa”) given the information of other nearby concepts (e.g., “near the sound of glass breaking”). In the last section, we introduced how we generate heatmaps for concepts of different modalities. Given these heatmaps, we want to further narrow down the target location.

3.6.1. Cross-modal reasoning

The main idea of our cross-modal reasoning method is shown in Figure 11. We treat the predictions from four modules as four modalities. When there are several queries referring to different modalities, we compute the respective heatmaps first and then perform element-wise multiplication among all heatmaps:

H t a r g e t = H 1 ⊙ … ⊙ H L

(11)

where ⊙ is the element-wise multiplication operator, and L is the total number of referred modalities. We extract the position on the target heatmap H target that has the highest probability as the predicted location.OPEN IN VIEWER

Figure 11.

The key idea of cross-modal reasoning is converting the prediction from different modalities into heatmaps, and then fusing them with element-wise multiplication, effectively using complementary multimodal information to resolve ambiguous prompts.

When we compute the heatmaps, there is always a primary heatmap while others are auxiliary ones. For example, in Figure 11, in the query “the sound of baby crying near the sofa,” the heatmap for “the sound of baby crying” is the primary heatmap, while the heatmap for “the sofa” is the auxiliary. We set the decay rate for the primary heatmap higher (e.g., 0.1 in this work for voxel map with 0.05 m voxel size) since we want to know the exact location of the target while tuning the decay rate for the auxiliary heatmap lower (e.g., 0.01) as having a broader effect area to narrow down major targets is desirable. More specifically, a higher decay rate indicates that the relevancy to the concept decreases faster when the location is farther away from the concept locations (heat value decreases 0.1 for every 0.05 m). When there are multiple concepts or modalities mentioned in the target specification, the map with a high decay rate refers to the target concept we want the robot to get closer to. Those maps with low decay rates serve as constraints to select targets in the main map. From another perspective, the decay rates of multiple maps represent the importance weights we assign to their corresponding concepts. The higher the decay rate is, the more important that concept is, and we want the robot to get closer to that concept. When the decay rates of two concepts are the same, the two concepts are equally important, and the fusion of the two concepts’ heatmaps will peak at the middle points between those concepts.

It is worth noting that our cross-modal reasoning method is relatively simple and has the underlying assumption that the heatmaps from different modalities are conditionally independent. Despite this simplification, our method proves to be effective across our experiments. Nonetheless, exploring more advanced cross-modal fusion techniques remains an exciting direction for future work.

3.7. Multimodal goal navigation from language

In the setting of multimodal goal navigation from language, the agent is given language descriptions referring to targets from different modalities (e.g., sounds, images and objects), and is required to plan paths to them. While most of the previous navigation methods focus mainly on a specific type of goal, we unify these tasks with the help of large language models (LLMs). Following a similar spirit to Section 3.4, we use an LLM to interpret the natural language commands and synthesize API calls combined with simple logic structures in the form of executable python code. In the following, we will detail the code generation process for multimodal goal navigation, including (i) the introduction of the API library as a tool set for the LLM to use during code generation, (ii) the mechanism of spatial goal reasoning, (iii) the way we generate multimodal maps with code, (iv) the method to achieve cross-modal reasoning with code, and (v) the conversion of code to navigation commands that robots receive. At the end, we provide the prompt and an example of inference for our method.

3.7.1. Navigation API library

In Table 2, we listed all the APIs we provide to the LLM for potential usage. Compared to VLMaps’ API library in Table 1, we simplify and adapt the library to more modalities for AVLMaps. We provide a list of examples consisting of language instructions and generated code, which demonstrates the usage of those APIs as a context prompt (as in Figure 12) to the LLM when we ask it to generate code during the inference time.

OPEN IN VIEWER

Table 2.

Navigation API library for multimodal goal navigation in AVLMaps.

APIs	Functions
load_image(path)	Load an image as an array from the path
move_to(pos)	Move to a position on the map
get_major_map(img = None, obj = None, sound = None)	Generate a heatmap with high heat decay rate for a specific modality (image, object description, or sound description)
get_map(img = None, obj = None, sound = None)	Generate a heatmap with low heat decay rate for a specific modality (image, object description, or sound description)
get_max_pose_3d(heatmap)	Retrieve the coordinate in a 3D voxel heatmap which has the maximal heat value)

OPEN IN VIEWER

Figure 12.

The full context prompt (prompt in gray) AVLMaps used for achieving all navigation tasks in the experiments.

4.1. Zero-shot spatial goal navigation from language

4.1.3. Tasks collection

In these experiments, we investigate the performance of VLMaps versus other baselines for zero-shot spatial goal navigation from language. Our benchmark consists of 21 trajectories in seven scenes, with manually specified corresponding language instructions for evaluation. Each trajectory contains four different spatial locations as subgoals. Examples of subgoals are “east of the table,” “in between the chair and the sofa,” or “move forward 3 meters.” There are also instructions for the robot to realign itself in reference to nearby objects such as “with the counter on your right.” We only consider a subgoal as having been achieved, when the robot reaches the subgoal location within a range of 1 meter. To evaluate the long-horizon navigation capabilities of the agents, we compute the success rate (SR) of continuously reaching one to four subgoals in a sequence, shown in Table 3. For all map-based methods, including CoW, CLIP Map, ground truth semantic map, and our method, we apply the code generation techniques introduced in Section 3.4. For LM-Nav, we simply use the same parsing method in the original paper (Shah et al., 2023) to break down the language instruction into subgoals.

OPEN IN VIEWER

Table 3.

The success rate (%) of zero-shot spatial goal navigation with language.

Tasks	Number of subgoals in a row
	1	2	3	4
LM-Nav (Shah et al., 2023)	5	5	0	0
CoW (Gadre et al., 2023)	33	5	0	0
CLIP Map	19	0	0	0
VLMaps (ours)	62	33	14	10
GT Map	76	48	33	29

Table 3 summarizes the zero-shot spatial goal navigation success rates. Our method outperforms other baselines in this task. Different from object navigation tasks where agents only need to approach a certain object type within a range, disregarding the relative spatial shift to the object, the language-based spatial goal navigation tasks require the robot to accurately arrive at the described location in reference to the object. This poses a bigger challenge to the landmark localization ability of the method. The low localization ability of CoW and CLIP Map leads to their high failure rates in this task.

4.2. Cross-embodiment navigation

4.2.4. Metrics

We evaluate the Success Rate (SR) and the Success rate weighted by the (normalized inverse) Path Length (SPL) (Anderson et al., 2018a) defined as: S P L = 1 / N ∑ i = 1 N S i l i / m a x ( p i , l i ) where N is the total number of evaluated tasks, S _i ∈ {0, 1} is the binary indicator of success, l _i denotes the ground truth shortest path length, and p _i denotes the actual path length of the agent in navigation. This metric indicates how efficient the actual path is compared to the ground truth shortest path when the navigation task is achieved. In our three setups, the ground truth trajectories for the LoCoBot and the drone are planned on floor-level and on height level of 1.7 meters, respectively.

The results provided in Table 4 show that the average navigation success rates of the ground-map version of the LoCoBot and the drone are similar because the same obstacles map is used for planning. However, there is an obvious gap between their SPL values. This is because when the drone does not have access to a customized obstacle map, it fails to benefit from flying over ground objects to improve the navigation efficiency. In contrast, while achieving similar success rate compared to the drone with a ground map, the drone with a drone map manages to navigate with higher path efficiency, reflected by the increased SPL values. The comparable SPL values for the drone with the drone map and the LoCoBot with the ground map shows that VLMaps help to generalize the navigation efficiency among different embodiments. An example of the multi-embodiment object navigation task is shown in Figure 14, where by defining a more efficient obstacles map, the drone flies over the sofa and reaches the laptop target directly, while the LoCoBot has to move aside first to avoid colliding with the sofa.

OPEN IN VIEWER

Table 4.

The success rate (%) and SPL of multi-embodiment object goal navigation with language.

Tasks	Number of subgoals in a row								IndependentSubgoals
	1		2		3		4
	SR	SPL	SR	SPL	SR	SPL	SR	SPL	SR
LoCoBot (ground map)	53	49.0	28	17.8	14	6.7	6	2.5	52.3
Drone (ground map)	53	41.8	28	15.5	14	5.3	6	2.0	53.3
Drone (drone map)	56	45.4	30	16.3	17	7.0	7	2.5	55.0

OPEN IN VIEWER

Figure 14.

VLMaps enable different embodiments to define their own obstacle maps for navigation. The left image shows the top-down view of an environment. The middle columns show the observations of agents during navigation. The images on the right demonstrate the obstacles maps generated for different embodiments and the corresponding navigation paths.

4.3. Multimodal navigation simulation setup

4.4. Multimodal goal navigation

4.4.1. Sound goal navigation

We first test AVLMaps in sound goal navigation tasks. We collect 200 sequences of sound goals in 10 different scenes. In each sequence, there are four sound categories that require the robot to reach. The results are shown in Table 5. We generate AudioCLIP (Guzhov et al., 2022) features with our audio localization module and match all audio with the target sound category in the embedding space, similar to a text-to-audio retrieval setup. Then, the agent plans a path to the audio position. We tested different ranges of sound categories inserted into the map. The full list of sound categories in each major class can be found in the link. ¹ The results show that our agent manages to recognize sound goals and navigate with a 77.5% success rate.

OPEN IN VIEWER

Table 5.

The success rate (%) of sound goal navigation with AVLMaps.

Tasks	Number of subgoals in a row				IndependentSubgoals
	1	2	3	4
Domestic sound (10 categories)	59.5	33.0	15.5	7.0	62.5
+ Human sound (20 categories)	69.5	47.0	36.5	23.0	72.38
+ Animal sound (30 categories)	74.5	58.5	45.5	33.0	77.5

4.4.2. Visual and object goals navigation

We then test AVLMaps with visual and object goal navigation tasks. The agent is given an image and two object categories in the language in one sequence of tasks and asked to navigate to the image goal and two object goals in sequence. The success rate for 200 sequences of tasks in 10 scenes is reported in Table 6. The results show that our method enables the agent to navigate to goals from different modalities.

OPEN IN VIEWER

Table 6.

The success rate (%) of multimodal goal navigation with AVLMaps. The agent is required to navigate to one visual goal and two object goals in sequence.

Tasks	Number of subgoals in a row			IndependentSubgoals
	1	2	3
AVLMaps (Ours)	71.5	40.5	25.0	47.4

4.5. Cross-modal goal indexing

When we refer to a goal with language, it is likely that the goal can be found in more than one place in the environment. A major strength of our method is that it can disambiguate goals with multimodal information. In this experiment, we will show the cross-modal goal reasoning capability of AVLMaps.

4.5.1. Area-object goal indexing

In this setup, we use an area description to disambiguate the object goal. We collected 100 indexing tasks in 10 scenes. Each task consists of an object category and a region category (e.g., “living room,” “kitchen,” “dining room,” “bathroom,” etc.). The agent needs to predict the correct object location which is inside the region. The top-one recall with different distance tolerance is reported in Table 7. We notice that VLMaps (Huang et al., 2023b) struggles to find the goal in the correct region because VLMaps integrates visual-language features from the encoder fine-tuned on the instance segmentation dataset, improving its segmentation performance on common objects while dropping its ability to recognize more general concepts like regions. In contrast, ConceptFusion integrates pre-trained CLIP features into the map without fine-tuning, enabling it to recognize general concepts including regions, and thus the indexing results are improved.

OPEN IN VIEWER

Table 7.

The recall (%) of area-object cross-modal indexing.

Method	Recall@1 (%)				Average min. distance (m)
	<0.5 m	<1 m	<1.5 m	<2 m
Baseline (VLMaps)	5.56	7.78	13.33	17.78	8.22
+ ConceptFusion	12.22	13.33	16.67	21.11	7.60
+ CLIP sparse (Ours)	15.56	24.44	31.11	35.56	6.17
+ GT region map	37.78	44.44	55.56	61.11	2.62

4.5.2. Object-sound goal indexing

In this setting, we use object goals to disambiguate sound goals. We collected 119 indexing tasks, each of which consist of a sound category and a nearby object category. Each sound category in a scene can be heard at more than one location, introducing ambiguity to the localization scenario. The recall is reported in Table 8. With the combination of object and audio localization modules, our method largely increases the recall rate for localizing the correct sound goal position in ambiguous scenarios.

OPEN IN VIEWER

Table 8.

The recall (%) of object-sound cross-modal indexing.

Method	Recall@1 (%)				Average min. distance (m)
	<0.5 m	<1 m	<1.5 m	<2 m
Baseline (wav2clip)	8.40	10.08	10.92	14.29	8.52
Baseline (AudioCLIP)	26.05	35.29	36.97	42.01	5.04
VLMaps + wav2clip	24.37	30.25	33.61	38.66	6.27
VLMaps + AudioCLIP (Ours)	53.78	65.55	67.23	70.59	2.74

4.5.3. Visual-object goal indexing

In visual-object goal indexing tasks, visual clues are used to resolve ambiguity. Given an object category and an image, our method can localize the correct object near the image position with over 60% of recall for 0.5 m distance tolerance, as is shown in Table 9.

OPEN IN VIEWER

Table 9.

The recall (%) of visual-object cross-modal indexing.

Method	Recall@1 (%)				Average min. distance (m)
	<0.5 m	<1 m	<1.5 m	<2 m
VLMaps w/o visual module	7.55	9.43	11.32	11.94	11.22
VLMaps w/ visual module (Ours)	62.26	66.67	70.44	72.32	3.11

4.6. Multimodal ambiguous goal navigation

In this part, we test our method with ambiguous goal navigation tasks. We collect 119 sequences of tasks. In each task, the agent is required to navigate to an ambiguous sound goal (i.e., “move to the sound of baby crying near the sofa”) and an ambiguous object goal (i.e., “move to the counter near {image of the kitchen}”) sequentially. The category of the object near the sound and the image taken near the object are provided. These tasks require the agent to reason across different modalities to accurately localize the target. We consider two single-modality baselines: VLMaps (Huang et al., 2023b) and AudioCLIP (Guzhov et al., 2022) and one multimodal baseline. The multimodal baseline uses VLMaps as the object localization module, wav2clip (Wu et al., 2022) as the audio localization module and the same visual localization module as our method. The results are shown in Table 10. We observe that AVLMaps navigates to cross-modal goals with a 24.2% higher success rate to ambiguous sound goals and with a 2.1% higher success rate to ambiguous object goals compared to the alternative multimodal baseline.

OPEN IN VIEWER

Table 10.

The success rate (%) of multimodal ambiguous goal navigation with AVLMaps. The agent is required to navigate to one ambiguous sound goal and one ambiguous object goal sequentially.

Tasks	Number of subgoals in a row		Sound	Object
	1	2	Goals	Goals
VLMaps	-	-	-	27.1
AudioCLIP	-	-	16.9	-
VLMaps + wav2clip	22.0	12.7	22.0	53.4
VLMaps + AudioCLIP (Ours)	46.2	28.6	46.2	55.5

4.7. Scaling experiment

Since AVLMaps is highly modular, each module can be upgraded with advanced foundation models that generate similar audio-language features or visual-language features. In this section, we explore whether AVLMaps can evolve with the advancement in foundation model research. We follow similar settings of multimodal goal navigation as in Section 4.4 to test the AVLMaps modules supported by different foundation models. In this experiment, the robot needs to navigate to a sound goal, a visual goal, and two object goals in a sequence and we report the in-a-row navigation success rate as well as the success rate for each type of goal. In this experiment, we mainly focus on analyzing how the performance scales with improved Audio Language Models and Vision Language Models. We fixed the visual localization module as NetVLAD and SuperGLUE.

4.7.3. Results

We report the results in Table 11. The first two rows compare the success rates of different Audio Language Models (ALMs) using the same VLM. We observe a significant 10% improvement in navigating to sound goals when replacing AudioCLIP with CLAP. This improvement is largely due to CLAP’s more extensive pretraining on larger, more diverse datasets with advanced augmentation techniques (Elizalde et al., 2023). From the second to the final rows, we compared three VLMs that generate dense visual-language features but fixed CLAP as the ALM. However, recent VLMs did not offer clear benefits. In fact, the SAM + CLIP combination significantly reduced performance. Upon closer inspection, we found that while SAM + CLIP performed well in previous work (Werby et al., 2024), it requires extensive hyper-parameter tuning to adapt to different scenes. The quality of SAM’s masks is highly sensitive to its parameters, and CLIP struggles to interpret masked regions, as highlighted by OVSeg (Liang et al., 2023a). OVSeg addressed this by training learnable mask prompts, and improving 2D semantic segmentation through ensemble techniques. However, these methods are designed to enhance segmentation performance, not visual-language feature generation. Like LSeg, OVSeg is fine-tuned on similar datasets, offering limited improvements in dense pixel-level visual-language features. As a result, AVLMaps’ object localization module, which relies on these features, sees minimal benefit from OVSeg. However, we believe that pixel-level visual language features can be improved with access to more high-quality segmentation datasets in the future, boosting the performance of the object localization module in AVLMaps.

OPEN IN VIEWER

Table 11.

The success rate (%) of multimodal goal navigation with different foundation models. The agent is required to navigate to one sound goal, one visual goal, and two object goals in sequence. The right-most three columns indicate the independent success rate of navigating to specific types of goals.

ALM	VLM	Number of subgoals in a row				Sound	Object
		1	2	3	4
AudioCLIP	LSeg	71.0	59.0	29.0	17.0	71.0	41.8
CLAP	LSeg	81.0	69.0	36.5	19.0	81.0	52.0
CLAP	SAM + CLIP	80.0	68.5	29.5	12.0	80.0	40.5
CLAP	OVSeg	81.0	69.0	37.0	21.0	81.0	53.5

4.8. Real world experiment for mobile robot

To answer the question of how AVLMaps applies to real-world environments and benefits multimodal navigation, we designed a mobile navigation experiment in which the robot must locate sound, image, and object-based goals within an environment containing duplicate objects from certain categories such as chairs, backpacks, shelves and so on. This setup demonstrates how AVLMaps enables the robot to retrieve multimodal concepts and disambiguate goals by leveraging information from additional modalities.

4.9. Real world experiment for table-top goal reaching

In the previous section, we demonstrated how AVLMaps empower a robot to interpret multimodal goals in room-scale mobile navigation tasks. Here, we extend our investigation to assess how AVLMaps benefit a fixed-based manipulator in real-world table-top tasks, which require a more detailed semantic understanding of the scene. In this setup, the robot manipulator must approach multimodal goals with a stricter tolerance for error (within 10 cm). Additionally, we explore AVLMaps’ potential for application across robots with varied embodiments.

5.1. Transient sound reaction

One of the main challenges with our AVLMaps is that sound is inherently transient, while the map relies on pre-exploration and offline mapping to embed the information for navigation tasks. Even if sounds are associated with specific locations during the exploration phase, those sounds might disappear or shift to new locations by the time of the inference and navigation. Therefore, AVLMaps struggles to support reactive navigation toward transient sound goals. Previous work on vision and audio navigation (Chen et al., 2020; Gan et al., 2020b; Younes et al., 2023) has focused on enabling robots to respond to transient sounds using biaural and visual observations. However, these methods are limited to transient goals and have been tested only in simulated environments. We argue that both transient sounds and previously heard sounds are essential for intelligent navigation. Transient sounds serve as important cues for immediate action, while past sounds provide valuable references for narrowing down possible targets based on prior experiences. For example, when instructed to go to “the café where you heard the song,” we can recall the specific café and adjust our navigation accordingly. A robot must be able to understand and navigate toward both kinds of sound goals to achieve human-level intelligence in sound-based navigation. However, no current system integrates these dual capabilities. We believe this gap presents an exciting opportunity for future research.

5.2. Sound localization

While AVLMaps manages to comprehend the semantic meaning of sounds, it struggles to localize the sound sources. In this paper, the audio localization module assumes monaural audio input, lacking in the ability to utilize binaural audio to localize sound sources with triangulation like humans. More specifically, a sound can be heard in all locations inside a room, but we only associate its features with the robot’s current location. Encouragingly, several concurrent works are actively addressing the sounds source localization, including efforts to learn the real-world acoustic sound field (Chen et al., 2024b), reconstruct the acoustic properties of environments (Wang et al., 2024), and develop more acoustically realistic simulation environments (Chen et al., 2022).

5.3. Dynamic scenes

Another challenge faced by our multimodal spatial language maps (both VLMaps and AVLMaps) is their inability to handle dynamic scenes. One form of dynamics involves in-view dynamics, such as walking humans and moving articulated objects during exploration. These dynamic entities easily corrupt the object map, leaving behind point artifacts that trace their movement trajectories. The semantic features of the moving objects may be erroneously associated with these artifact points, leading to inaccurate representations of their true locations. Another form of dynamics involves long-term dynamics, such as object relocations between the exploration and inference phases. These relocations can invalidate the pre-built map, necessitating an updating mechanism to ensure accurate navigation. Currently, our multimodal spatial language maps lack mechanisms to address both types of dynamics. To explore potential solutions, we investigated approaches to mitigating the impact of dynamics. Prior works have addressed in-view dynamics by removing or tracking certain classes of semantic masks during mapping (Runz et al., 2018; Xu et al., 2019). For long-term dynamics, recent research has proposed learning-based object association methods to update the locations of relocated objects in the map (Huang et al., 2025; Yan et al., 2025). These solutions could be integrated into our mapping pipeline to enhance its robustness.

5.4. Extension to more modalities

In this paper, we have demonstrated the feasibility of integrating information from multiple modalities into a unified map representation. More broadly, our method provides a framework that can be extended to additional modalities, such as odor, temperature, magnetic fields, infrared imagery, and point clouds. In our implementation, we selected audio, language, and vision because they closely mirror human perceptual capabilities. Viewed from a broader perspective, AVLMaps can be regarded as a case study in building a multimodal spatial memory system, with inherent flexibility to incorporate more modalities. Extending the system to a new modality X involves three steps: (i) implementing the “X localization module” which includes mapping (embedding and storing the data into a representation) and retrieval (generating a heatmap based on the query data and the map) functions for the new modality, (ii) implementing the get_major_map(X_input=None) and get_map(X_input=None) to accept a new modality’s data as input and return 3D heatmaps with high and low decay rates as in Section 3.7, and (iii) implementing a context prompt as in Figure 12 for the new modality, including an instruction involving the new modality, and the expected generated code. Although AVLMaps only focuses on audio, language, and vision, its simplicity, flexibility, and scalability open up promising directions for future research.