We survey applications of pretrained foundation models in robotics. Traditional deep learning models in robotics are trained on small datasets tailored for specific tasks, which limits their adaptability across diverse applications. In contrast, foundation models pretrained on internet-scale data appear to have superior generalization capabilities, and in some instances display an emergent ability to find zero-shot solutions to problems that are not present in the training data. Foundation models may hold the potential to enhance various components of the robot autonomy stack, from perception to decision-making and control. For example, large language models can generate code or provide common sense reasoning, while vision-language models enable open-vocabulary visual recognition. However, significant open research challenges remain, particularly around the scarcity of robot-relevant training data, safety guarantees and uncertainty quantification, and real-time execution. In this survey, we study recent papers that have used or built foundation models to solve robotics problems. We explore how foundation models contribute to improving robot capabilities in the domains of perception, decision-making, and control. We discuss the challenges hindering the adoption of foundation models in robot autonomy and provide opportunities and potential pathways for future advancements. The GitHub project corresponding to this paper can be found here: https://github.com/robotics-survey/Awesome-Robotics-Foundation-Models.
AndersonPWuQTeneyD, et al. (2018) Vision-and-language navigation: interpreting visually-grounded navigation instructions in real environments. In: CVPR, pp. 3674–3683. IEEE.
2.
ArandjelovicRGronatPToriiA, et al. (2016) NetVLAD: CNN architecture for weakly supervised place recognition. In: CVPR, pp. 5297–5307. IEEE.
3.
BahlSMendoncaRChenL, et al. (2023) Affordances from human videos as a versatile representation for robotics. In: CVPR. IEEE.
4.
BakerBAkkayaIZhokovP, et al. (2022) Video PreTraining (VPT): Learning to act by watching unlabeled online videos. In: NeurIPS. IEEE.
5.
BaltrusaitisTAhujaCMorencyLP (2019) Multimodal machine learning: a survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence41(2): 423–443. DOI: 10.1109/TPAMI.2018.2798607.
6.
BangguoYHamidrezaKMingC (2023) LL3MVN: leveraging large language models for visual target navigation. arXiv preprint arXiv:2304.05501.
7.
BommasaniRHudsonDAAdeliE, et al. (2021) On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
8.
BonattiRVempralaSMaS, et al. (2023) PACT: perception-action causal transformer for autoregressive robotics pretraining. In: IROS. IEEE.
9.
BousmalisKVezzaniGRaoD, et al. (2023) RoboCat: a self-improving foundation agent for robotic manipulation. arXiv preprint arXiv:2306.11706.
10.
BoxGEJenkinsGMReinselGC, et al. (2015) Time Series Analysis: Forecasting and Control. Hoboken, New Jersey: John Wiley & Sons.
11.
BrohanABrownNCarbajalJ, et al. (2022) Rt-1: robotics transformer for real-world control at scalearXiv Preprint arXiv:2212.06817.
12.
BrohanABrownNCarbajalJ, et al. (2023a) RT-1: robotics transformer for real-world control at scale. In: RSS.
13.
BrohanAChebotarYFinnC, et al. (2023b) Do as I can, not as I say: grounding language in robotic affordances. In: CoRL, pp. 287–318. PMLR.
CaronMTouvronHMisraI, et al. (2021) Emerging properties in self-supervised vision transformers. In: ICCV. IEEE.
19.
ChangAXFunkhouserTGuibasL, et al. (2015) ShapeNet: an information-rich 3D model repository. arXiv preprint arXiv:1512.03012.
20.
ChangADaiAFunkhouserT, et al. (2017) Matterport3D: learning from RGB-D data in indoor environments. In: 3DV, pp. 667–676. DOI: 10.1109/3DV.2017.00081.
21.
ChenTKornblithSNorouziM, et al. (2020) A simple framework for contrastive learning of visual representations. In: ICML.
22.
ChenMTworekJJunH, et al. (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
23.
ChenWHuSTalakR, et al. (2022a) Leveraging large language models for robot 3D scene understanding. arXiv preprint arXiv:2209.05629.
24.
ChenXWangXChangpinyoS, et al. (2022b) PaLI: a jointly-scaled multilingual language-image model. In NeurIPS. IEEE.
25.
ChenJLiGKumarS, et al. (2023a) How to not train your dragon: training-free embodied object goal navigation with semantic frontiers. arXiv preprint arXiv:2305.16925.
26.
ChenTTippurMWuS, et al. (2023b) Visual dexterity: in-hand reorientation of novel and complex object shapes. Science Robotics8(84): eadc9244.
27.
ChenXDjolongaJPadlewskiP, et al. (2023c) PaLI-X: on scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565.
28.
ChenYArkinJZhangY, et al. (2023d) AutoTAMP: autoregressive task and motion planning with llms as translators and checkers. arXiv preprint arXiv:2306.06531.
29.
ChenYGandhiRZhangY, et al. (2023e) NL2TL: transforming natural languages to temporal logics using large language models. arXiv preprint arXiv:2305.07766.
30.
ChenZKiamiSGuptaA, et al. (2023f) GenAug: retargeting behaviors to unseen situations via generative augmentation. In: RSS.
31.
ChenBXuZKirmaniS, et al. (2024) Spatialvlm: endowing vision-language models with spatial reasoning capabilities. In CVPR. IEEE.
32.
ChengHKSchwingAG (2022) XMem: long-term video object segmentation with an atkinson-shiffrin memory model. In: ECCV, pp. 640–658. Springer.
33.
ChowdheryANarangSDevlinJ, et al. (2022) PaLM: scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
34.
ClarkeS (2019) Robot Learning for Manipulation of Granular Materials Using Vision and Sound. Master’s Thesis. Carnegie Mellon University.
35.
DaiZYangZYangY, et al. (2019) Transformer-XL: attentive language models beyond a fixed-length context. In: ACL.
36.
DamenDDoughtyHFarinellaGM, et al. (2018) Scaling egocentric vision: the EPIC-KITCHENS dataset. In: ECCV. Springer.
37.
DasariSEbertFTianS, et al. (2019) RoboNet: large-scale multi-robot learning. In: CoRL.
38.
DasguptaIKaeser-ChenCMarinoK, et al. (2022) Collaborating with language models for embodied reasoning. In: Second Workshop on Language and Reinforcement Learning.
39.
DehghaniMDjolongaJMustafaB, et al. (2023) Scaling vision transformers to 22 billion parameters. In: ICML.
40.
DeitkeMHanWHerrastiA, et al. (2020) RoboTHOR: an open simulation-to-real embodied AI platform. In: CVPR. IEEE.
41.
DeToneDMalisiewiczTRabinovichA (2018) Superpoint: self-supervised interest point detection and description. In: CVPR deep learning for visual SLAM workshop. IEEE.
42.
DevlinJChangMWLeeK, et al. (2018) BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
43.
DingWXuCAriefM, et al. (2023) A survey on safety-critical driving scenario generation—a methodological perspective. IEEE ITSC99: 1–19.
44.
DongQLiLDaiD, et al. (2022) A survey for in-context learning. arXiv preprint arXiv:2301.00234.
45.
DosovitskiyABeyerLKolesnikovA, et al. (2021) An image is worth 16x16 words: transformers for image recognition at scale. In: ICLR.
46.
DriessDXiaFSajjadiMSM, et al. (2023) PaLM-E: an embodied multimodal language model. arXiv preprint arXiv:2303.03378.
47.
DuYLiJTangT, et al. (2023a) Zero-shot visual question answering with language model feedback. arXiv preprint arXiv:2305.17006.
48.
DuYWatkinsOWangZ, et al. (2023b) Guiding pretraining in reinforcement learning with large language models. In: ICML, pp. 8657–8677. PMLR.
49.
DuYYangMDaiB, et al. (2023c) Learning universal policies via text-guided video generation. In: NeurIPS.
50.
DuYYangMFlorenceP, et al. (2023d) Video language planning. arXiv preprint arXiv:2310.10625.
51.
DuanYSchulmanJChenX, et al. (2016) RLˆ2: fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779.
EhsaniKGuptaTHendrixR, et al. (2024) Spoc: imitating shortest paths in simulation enables effective navigation and manipulation in the real world. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16238–16250. IEEE.
54.
EngelbrechtHASchieleG (2014) Transforming minecraft into a research platform. In: IEEE CCNC. IEEE.
55.
FanLWangGJiangY, et al. (2022) MineDojo: building open-ended embodied agents with internet-scale knowledge. In: NeurIPS Datasets and Benchmarks Track.
56.
FangHSWangCGouM, et al. (2020) Graspnet-1billion: a large-scale benchmark for general object grasping. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11444–11453. IEEE.
57.
FaridAVeerSMajumdarA (2021) Task-driven out-of-distribution detection with statistical guarantees for robot learning. In: CoRL.
58.
FaridASnyderDRenAZ, et al. (2022a) Failure prediction with statistical guarantees for vision-based robot control. arXiv preprint arXiv:2202.05894.
59.
FaridAVeerSIvanovicB, et al. (2022b) Task-relevant failure detection for trajectory predictors in autonomous vehicles. In: CoRL.
60.
FengXLuoYWangZ, et al. (2023) ChessGPT: bridging policy learning and language modeling. arXiv preprint arXiv:2306.09200.
61.
FuLDattaGHuangH, et al. (2024) A touch, vision, and language dataset for multimodal alignment. arXiv preprint arXiv:2402.13232.
62.
GadreSYWortsmanMIlharcoG, et al. (2023) CoWs on pasture: baselines and benchmarks for language-driven zero-shot object navigation. In: CVPR, pp. 23171–23181. IEEE.
63.
GanguliDLovittLKernionJ, et al. (2022) Red teaming language models to reduce harms: methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858.
64.
GaoRSiZChangYY, et al. (2022) Objectfolder 2.0: a multisensory object dataset for sim2real transfer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10598–10608. IEEE.
65.
GaoJSarkarBXiaF, et al. (2023) Physically grounded vision-language models for robotic manipulation. In: CoRL.
66.
GarrettCRChitnisRHolladayR, et al. (2021) Integrated task and motion planning. Annual review of control, robotics, and autonomous systems4: 265–293.
67.
GervetTChintalaSBatraD, et al. (2023) Navigating to objects in the real world. arXiv preprint arXiv:2212.00922.
68.
GoodwinWHavoutisIPosnerI (2022a) You only look at one: category-level object representations for pose estimation from a single example. In: CoRL.
HongYZhenHChenP, et al. (2023) 3D-LLM: injecting the 3D world into large language models. arXiv preprint arXiv:2307.12981.
80.
HsuKCHuHFisacJF (2023a) The safety filter: a unified view of safety-critical control in autonomous systems. arXiv preprint arXiv:2309.05837.
81.
HsuKCRenAZNguyenDP, et al. (2023b) Sim-to-Lab-to-Real: safe reinforcement learning with shielding and generalization guarantees. Artificial Intelligence314: 103811.
82.
HuARussellLYeoH, et al. (2023a) GAIA-1: a generative world model for autonomous driving. arXiv preprint arXiv:2309.17080.
83.
HuYXieQJainV, et al. (2023b) Toward general-purpose robots via foundation models: a survey and meta-analysis. arXiv preprint arXiv:2312.08782.
84.
HuXWangYFanL, et al. (2024) Semantic anything in 3d Gaussians. arXiv preprint arXiv:2401.17857.
85.
HuangJXieSSunJ, et al. (2021) Learning a decision module by imitating driver’s control behaviors. In: CoRL, pp. 1–10. PMLR.
86.
HuangWAbbeelPPathakD, et al. (2022a) Language models as zero-shot planners: extracting actionable knowledge for embodied agents. In: ICML.
87.
HuangWXiaFXiaoT, et al. (2022b) Inner monologue: embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608.
88.
HuangCMeesOZengA, et al. (2023a) Audio visual language maps for robot navigation. arXiv preprint arXiv:2303.07522.
89.
HuangCMeesOZengA, et al. (2023b) Visual language maps for robot navigation. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 10608–10615. IEEE.
90.
HuangWWangCZhangR, et al. (2023c) VoxPoser: composable 3D value maps for robotic manipulation with language models. In: CoRL.
91.
JannerMDuYTenenbaumJ, et al. (2022) Planning with diffusion for flexible behavior synthesis. In: ICML.
92.
JiaZLiuFThumuluriV, et al. (2023) Chain-of-thought predictive control. arXiv preprint arXiv:2304.00776.
93.
JiangYGuptaAZhangZ, et al. (2023) VIMA: general robot manipulation with multimodal prompts. In: ICML.
94.
KapelyukhIVosyliusVJohnsE (2023) DALL-E-Bot: introducing web-scale diffusion models to robotics. IEEE Robotics and Automation Letters8(7): 3956–3963. DOI: 10.1109/LRA.2023.3272516.
95.
KaramchetiSNairSChenAS, et al. (2023) Language-driven representation learning for robotics. In: RSS.
96.
KerblBKopanasGLeimkuehlerT, et al. (2023) 3D Gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics42(4): 1–14.
97.
KerrJKimCMGoldbergK, et al. (2023) LERF: language embedded radiance fields. In: ICCV, pp. 19729–19739.
98.
KhanSNaseerMHayatM, et al. (2022) Transformers in vision: a survey. ACM Computing Surveys54(10s): 1–41.
99.
KirillovAMintunERaviN, et al. (2023) Segment anything. In: ICCV, pp. 4015–4026.
100.
KobayashiSMatsumotoESitzmannV (2022) Decomposing NeRF for editing via feature field distillation. In: NeurIPS.
101.
KumarAFuZPathakD, et al. (2021) Rma: rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04034.
102.
KusanoKDBeattyKSchnelleS, et al. (2022) Collision avoidance testing of the waymo automated driving system. arXiv preprint arXiv:2212.08148.
103.
KwonMXieSMBullardK, et al. (2023) Reward design with language models. In: ICLR.
104.
LevesqueHDavisEMorgensternL (2012) The Winograd schema challenge. In: KR.
105.
LiCXiaFMartín-MartínR, et al. (2021) iGibson 2.0: object-centric simulation for robot learning of everyday household tasks. In: CoRL.
106.
LiBWeinbergerKQBelongieS, et al. (2022a) Language-driven semantic segmentation. In: ICLR.
107.
LiCZhangRWongJ, et al. (2022b) BEHAVIOR-1K: a benchmark for embodied AI with 1,000 everyday activities and realistic simulation. In: CoRL.
108.
LiJLiDXiongC, et al. (2022c) BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In: ICML.
109.
LiLHZhangPZhangH, et al. (2022d) Grounded language-image pre-training. In: CVPR,pp. 10965–10975.
110.
LiYFanHHuR, et al. (2023) Scaling language-image pre-training via masking. In: CVPR.
111.
LiangJHuangWXiaF, et al. (2023) Code as Policies: language model programs for embodied control. In: ICRA, pp. 9493–9500. IEEE.
112.
LiaoGLiJBaoZ, et al. (2024) Clip-gs: clip-informed gaussian splatting for real-time and view-consistent 3d semantic understanding. arXiv preprint arXiv:2404.14249.
113.
LinJDuYWatkinsO, et al. (2023a) Learning to model the world with language. arXiv preprint arXiv:2308.01399.
114.
LinJTangJTangH, et al. (2023b) AWQ: activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978.
115.
LinKAgiaCMigimatsuT, et al. (2023c) Text2Motion: from natural language instructions to feasible plans. In: Special Issue: Large Language Models in Robotics. Autonomous Robots.
116.
LiuPJSalehMPotE, et al. (2018) Generating Wikipedia by summarizing long sequences. In: ICLR.
117.
LiuYOttMGoyalN, et al. (2019) RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
118.
LiuMZhuYCaiH, et al. (2023a) Partslip: low-shot part segmentation for 3d point clouds via pretrained image-language models. CVPR: 21736–21746.
119.
LiuPYuanWFuJ, et al. (2023b) Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Computing Surveys55(9): 1–35.
120.
LiuSZengZRenT, et al. (2023c) Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499.
121.
LiuWDuYHermansT, et al. (2023d) StructDiffusion: language-guided creation of physically-valid structures using unseen objects. In: RSS.
122.
LuoRZhaoSKuckJ, et al. (2022) Sample-efficient safety assurances using conformal prediction. In: WAFR.
123.
LynchCSermanetP (2021) Language conditioned imitation learning over unstructured data. In: Robotics: Science and Systems.
124.
LynchCKhansariMXiaoT, et al. (2020) Learning latent plans from play. In: CoRL, pp. 1113–1132. PMLR.
125.
MaSVempralaSWangW, et al. (2022) COMPASS: contrastive multimodal pretraining for autonomous systems. In: IROS, pp. 1000–1007. IEEE.
126.
MaYJKumarVZhangA, et al. (2023a) LIV: language-image representations and rewards for robotic control. In: ICML.
127.
MaYJSodhaniSJayaramanD, et al. (2023b) VIP: towards universal visual reward and representation via value-implicit pre-training. In:ICLR.
128.
MahmoudiehPPathakDDarrellT (2022) Zero-shot reward specification via grounded natural language. In ICML, pp. 14743–14752. PMLR.
129.
MajumdarAShrivastavaALeeS, et al. (2020) Improving vision-and-language navigation with image-text pairs from the web. In: ECCV, pp. 259–274. Springer.
130.
MandiZBharadhwajHMoensV, et al. (2022) CACTI: a framework for scalable multi-task multi-scene visual imitation learning. arXiv preprint arXiv:2212.05711.
131.
MargolisGBAgrawalP (2023) Walk these ways: tuning robot control for generalization with multiplicity of behavior. In: Conference on Robot Learning, pp. 22–31. PMLR.
132.
MeesOBorja-DiazJBurgardW (2023) Grounding language with visual affordances over unstructured data. In: ICRA, pp. 11576–11582. IEEE.
133.
MildenhallBSrinivasanPPTancikM, et al. (2021) Nerf: representing scenes as neural radiance fields for view synthesis. Communications of the ACM65(1): 99–106.
134.
MindererMGritsenkoAStoneA, et al. (2022) Simple open-vocabulary object detection with vision transformers. In: ECCV, pp. 728–755. Springer.
135.
MirchandaniSXiaFFlorenceP, et al. (2023) Large language models as general pattern machines. arXiv preprint arXiv:2307.04721.
136.
MuYZhangQHuM, et al. (2023) EmbodiedGPT: vision-language pre-training via embodied chain of thought. arXiv preprint arXiv:2305.15021.
137.
NairSMitchellEChenK, et al. (2022a) Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. In: CoRL, pp. 1303–1315. PMLR.
138.
NairSRajeswaranAKumarV, et al. (2022b) R3M: a universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601.
OquabMDarcetTMoutakanniT, et al. (2023) DINOv2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193.
142.
PadalkarAPooleyAJainA, et al. (2023) Open X-Embodiment: robotic learning datasets and RT-X models. arXiv preprint arXiv:2310.08864.
143.
PaloNDByravanAHasencleverL, et al. (2023) Towards a unified agent with foundation models. In: Workshop on Reincarnating Reinforcement Learning at ICLR 2023.
144.
ParkJSO’BrienJCCaiCJ, et al. (2023a) Generative Agents: interactive simulacra of human behavior. In: ACM Symposium on User Interface Software and Technology. ACM.
145.
ParkYJWangHArdeshirS, et al. (2023b) Representation reliability and its impact on downstream tasks. arXiv preprint arXiv:2306.00206.
146.
PerezEHuangSSongF, et al. (2022) Red teaming language models with language models. In: EMNLP.
147.
PuigXRaKBobenM, et al. (2018) VirtualHome: simulating household activities via programs. In: CVPR, pp. 8494–8502.
148.
PuigXUndersanderESzotA, et al. (2023) Habitat 3.0: a co-habitat for humans, avatars and robots. arXiv preprint arXiv:2310.13724.
149.
QinMLiWZhouJ, et al. (2023) Langsplat: 3d language Gaussian splatting. arXiv preprint arXiv:2312.16084.
150.
QiuJLiLSunJ, et al. (2023) Large ai models in health informatics: applications, challenges, and the future. IEEE Journal of Biomedical and Health Informatics27(12): 6074–6087. DOI: 10.1109/JBHI.2023.3316750.
RadfordAWuJChildR, et al. (2019) Language Models Are Unsupervised Multitask Learners. OpenAI Blog.
153.
RadfordAKimJWHallacyC, et al. (2021a) Learning transferable visual models from natural language supervision. In: MeilaMZhangT (eds) ICML, Proceedings of Machine Learning Research, Vol. 139, 8748–8763. PMLR.
154.
RadfordAKimJWHallacyC, et al. (2021b) Learning transferable visual models from natural language supervision. In: ICML, pp. 8748–8763.
155.
RadosavovicIXiaoTJamesS, et al. (2023) Real-world robot learning with masked visual pre-training. In: CoRL, pp. 416–426. PMLR.
156.
RaffelCShazeerNRobertsA, et al. (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research21(1): 5485–5551.
157.
RameshAPavlovMGohG, et al. (2021a) Zero-shot text-to-image generation. In: ICML, pp. 8821–8831. PMLR.
158.
RameshAPavlovMGohG, et al. (2021b) Zero-shot text-to-image generation. In: MeilaMZhangT (eds) Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 139, pp. 8821–8831. PMLR.
159.
RameshADhariwalPNicholA, et al. (2022) Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125.
160.
RanftlRBochkovskiyAKoltunV (2021) Vision transformers for dense prediction. In: ICCV, pp. 12179–12188. PMLR.
161.
RashidASharmaSKimCM, et al. (2023) Language embedded radiance fields for zero-shot task-oriented grasping. In: 7th Annual Conference on Robot Learning (CoRL), pp. 178–200. PMLR.
162.
ReedSZolnaKParisottoE, et al. (2022) A generalist agent. arXiv preprint arXiv:2205.06175.
163.
ReidMYamadaYGuSS (2022) Can Wikipedia help offline reinforcement learning?arXiv preprint arXiv:2201.12122.
164.
RenAZDixitABodrovaA, et al. (2023) Robots that ask for help: uncertainty alignment for large language model planners. In: CoRL.
165.
RombachRBlattmannALorenzD, et al. (2022) High-resolution image synthesis with latent diffusion models. In: CVPR.
166.
SavvaMKadianAMaksymetsO, et al. (2019) Habitat: a platform for embodied AI research. In: ICCV.
167.
SennrichRHaddowBBirchA (2016) Neural machine translation of rare words with subword units. In: ACL.
168.
SermanetPDingTZhaoJ, et al. (2023) Robovqa: multimodal long-horizon reasoning for robotics. arXiv:2311.00899.
169.
ShafiullahNMMPaxtonCPintoL, et al. (2023) CLIP-fields: weakly supervised semantic fields for robotic memory. In: RSS.
170.
ShahRKumarV (2021) RRL: resnet as representation for reinforcement learning. In: ICML.
171.
ShahSDeyDLovettC, et al. (2017) AirSim: high-fidelity visual and physical simulation for autonomous vehicles. In: Field and Service Robotics.
172.
ShahDOsińskiBLevineS, et al. (2023a) LM-Nav: robotic navigation with large pre-trained models of language, vision, and action. In: CoRL, pp. 492–504. PMLR.
173.
ShahDSridharADashoraN, et al. (2023b) ViNT: a foundation model for visual navigation. In: CoRL.
ShorinwaOTuckerJSmithA, et al. (2024) Splat-mover: multi-stage, open-vocabulary robotic manipulation via editable Gaussian splatting.
179.
ShridharMManuelliLFoxD (2022) CLIPort: what and where pathways for robotic manipulation. In: CoRL, pp. 894–906. PMLR.
180.
ShridharMManuelliLFoxD (2023) Perceiver-Actor: a multi-task transformer for robotic manipulation. In: CoRL, pp. 785–799. PMLR.
181.
SimeonovADuYTagliasacchiA, et al. (2022) Neural Descriptor Fields: SE(3)-equivariant object representations for manipulation. In: ICRA.
182.
SinghIBlukisVMousavianA, et al. (2023) ProgPrompt: generating situated robot task plans using large language models. In: ICRA, pp. 11523–11530. IEEE.
183.
SinhaRSharmaABanerjeeS, et al. (2022) A system-level view on out-of-distribution data in robotics. arXiv preprint arXiv:2212.14020.
184.
Sohl-DicksteinJWeissEMaheswaranathanN, et al. (2015) Deep unsupervised learning using nonequilibrium thermodynamics. In: ICML.
185.
SohnK (2016) Improved deep metric learning with multi-class n-pair loss objective. In: NeurIPS.
186.
SongYErmonS (2019) Generative modeling by estimating gradients of the data distribution. In: NeurIPS.
187.
StoneAXiaoTLuY, et al. (2023) Open-world object manipulation using pre-trained vision-language model. arXiv preprint arXiv:2303.00905.
188.
SunJSunHHanT, et al. (2021a) Neuro-symbolic program search for autonomous driving decision module design. In: CoRL, pp. 21–30. PMLR.
189.
SunJYuLDongP, et al. (2021b) Adversarial inverse reinforcement learning with self-attention dynamics model. IEEE Robotics and Automation Letters6(2): 1880–1886.
190.
SunJHuangDALuB, et al. (2022a) PlaTe: visually-grounded planning with transformers in procedural tasks. IEEE Robotics and Automation Letters7(2): 4924–4930.
191.
SunJKousikSFridovich-KeilD, et al. (2022b) Self-supervised traffic advisors: distributed, multi-view traffic prediction for smart cities. In: IEEE ITSC.
192.
SunMYanWAbbeelP, et al. (2022c) Quantifying uncertainty in foundation models via ensembles. In: NeurIPS Workshop on Robustness in Sequence Modeling.
193.
SunJJiangYQiuJ, et al. (2023a) Conformal prediction for uncertainty-aware planning with diffusion dynamics model. In: NeurIPS.
194.
SunJKousikSFridovich-KeilD, et al. (2023b) Connected autonomous vehicle motion planning with video predictions from smart, self-supervised infrastructure. arXiv preprint arXiv:2309.07504.
SunJXuYDingM, et al. (2023d) NeRF-Loc: transformer-based object localization within neural radiance fields. IEEE Robotics and Automation Letters8(8): 5244–5250. DOI: 10.1109/LRA.2023.3293308.
197.
SunJZhengCXieE, et al. (2023e) Reasoning with foundation models: concepts, methodologies, and outlook. Zenodo Preprint. DOI: 10.5281/zenodo.10298866.
198.
SunYMaSMadaanR, et al. (2023f) SMART: self-supervised multi-task pretraining with control transformers. In: ICLR.
199.
SzotACleggAUndersanderE, et al. (2021) Habitat 2.0: training home assistants to rearrange their habitat. In: NeurIPS.
200.
TarasovDKurenkovVKolesnikovS (2022) Prompts and pre-trained language models for offline reinforcement learning. In: ICLR Workshop on Generalizable Policy Learning in Physical World.
201.
TeamOELStookeAMahajanA, et al. (2021) Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808.
202.
TeamAABauerJBaumliK, et al. (2023) Human-timescale adaptation in an open-ended task space. In: ICML.
TongSJonesESteinhardtJ (2023) Mass-producing failures of multimodal systems with language models. arXiv preprint arXiv:2306.12105.
205.
TouvronHLavrilTIzacardG, et al. (2023a) LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
206.
TouvronHLavrilTIzacardG, et al. (2023b) LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
207.
TschernezkiVLainaILarlusD, et al. (2022) Neural Feature Fusion Fields: 3D distillation of self-supervised 2D image representations. In: 3DV, pp. 443–453. IEEE.
208.
UmeyamaS (1991) Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence13(04): 376–380.
209.
VaswaniAShazeerNParmarN, et al. (2017) Attention is all you need. In: NeurIPS.
210.
VempralaSBonattiRBuckerA, et al. (2023) ChatGPT for robotics: design principles and model abilities. Technical Report MSR-TR-2023-8, Microsoft.
211.
VillegasRBabaeizadehMKindermansPJ, et al. (2023) Phenaki: variable length video generation from open domain textual description. In: ICLR.
212.
VincentJAFeldmanAOSchwagerM (2023) Guarantees on robot system performance using stochastic simulation rollouts. arXiv preprint arXiv:2309.10874.
213.
VincentJANishimuraHItkinaM, et al. (2024) How generalizable is my behavior cloning policy? a statistical approach to trustworthy performance evaluation. arXiv preprint arXiv:2405.05439.
214.
WangASinghAMichaelJ, et al. (2018) GLUE: a multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
215.
WangWZhuDWangX, et al. (2020) TartanAir: a dataset to push the limits of visual SLAM. In: IROS.
216.
WangCChaiMHeM, et al. (2022) CLIP-NeRF: text-and-image driven manipulation of neural radiance fields. In: CVPR, pp. 3835–3844.
217.
WangCFanLSunJ, et al. (2023a) MimicPlay: long-horizon imitation learning by watching human play. In: CoRL.
218.
WangGXieYJiangY, et al. (2023b) Voyager: an open-ended embodied agent with large language models. arXiv preprint arXiv: Arxiv-2305.16291.
219.
WangSSahariaCMontgomeryC, et al. (2023c) Imagen editor and editbench: advancing and evaluating text-guided image inpainting. In: CVPR, pp. 18359–18369.
220.
WebbNSmithDLudwickC, et al. (2020) Waymo’s safety methodologies and safety readiness determinations. arXiv preprint arXiv:2011.00054.
221.
WeiJWangXSchuurmansD, et al. (2022) Chain-of-thought prompting elicits reasoning in large language models. NeurIPS35: 24824–24837.
222.
WenMLinRWangH, et al. (2023) Large sequence models for sequential decision-making: a survey. Frontiers of Computer Science17(6): 176349.
WuZXiongYYuSX, et al. (2018) Unsupervised feature learning via non-parametric instance discrimination. In: CVPR.
225.
XiaFR ZamirAHeZY, et al. (2018) Gibson Env: real-world perception for embodied agents. In: CVPR.
226.
XianZGervetTXuZ, et al. (2023) Towards generalist robots: a promising paradigm via generative simulation. arXiv preprint arXiv:2305.10455.
227.
XiaoTRadosavovicIDarrellT, et al. (2022) Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173.
228.
XiaoTChanHSermanetP, et al. (2023a) Robotic skill acquisition via instruction augmentation with vision-language models. In: RSS.
229.
XiaoXLiuJWangZ, et al. (2023b) Robot learning in the era of foundation models: a survey. arXiv preprint arXiv:2311.14379.
230.
XueLGaoMXingC, et al. (2022) ULIP: learning unified representation of language, image and point cloud for 3D understanding. arXiv preprint arXiv:2212.05171.
231.
XueLYuNZhangS, et al. (2023) ULIP-2: towards scalable multimodal pre-training for 3d understanding. arXiv preprint arXiv:2305.08275.
YenamandraSRamachandranAYadavK, et al. (2023) Homerobot: open vocabulary mobile manipulation. arXiv preprint arXiv:2306.11565.
240.
YiTMostafaDVinhQT, et al. (2023) UL2: unifying language learning paradigms. arXiv preprint arXiv:2205.05131.
241.
YonedaTFangJLiP, et al. (2023) Statler: state-maintaining language models for embodied reasoning. arXiv preprint arXiv:2306.17840.
242.
YuXTangLRaoY, et al. (2022) Point-BERT: pre-training 3D point cloud transformers with masked point modeling. In: CVPR, pp. 19313–19322. Springer.
243.
YuTXiaoTStoneA, et al. (2023) Scaling robot learning with semantically imagined experience. arXiv Preprint arXiv:2302.11550.
244.
YuyingGAnnabellaMLiEL, et al. (2023) Policy adaptation from foundation model feedback. In: CVPR.
245.
ZengAFlorencePTompsonJ, et al. (2020) Transporter networks: rearranging the visual world for robotic manipulation. In: CoRL.
246.
ZengAAttarianMIchterB, et al. (2022) Socratic Models: composing zero-shot multimodal reasoning with language. arXiv.
247.
ZengALiuXDuZ, et al. (2023a) GLM-130B: an open bilingual pre-trained model. In: ICLR.
248.
ZengYJiangCMaoJ, et al. (2023b) CLIP2: contrastive language-image-point pretraining from real-world point cloud data. In: CVPR.
249.
ZhaiXKolesnikovAHoulsbyN, et al. (2022) Scaling vision transformers. In: CVPR.
250.
ZhangRGuoZZhangW, et al. (2022a) PointCLIP: point cloud understanding by CLIP. In: CVPR, pp. 8552–8562.
251.
ZhangYJiangHMiuraY, et al. (2022b) Contrastive learning of medical visual representations from paired images and text. In: MLHC.
252.
ZhangZWangYZhangY, et al. (2022c) Can offline reinforcement learning help natural language understanding?arXiv preprint arXiv:2212.03864.
253.
ZhangCHanDQiaoY, et al. (2023a) Faster segment anything: towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289.
254.
ZhangDLiangDYangH, et al. (2023b) SAM3D: zero-shot 3D object detection via segment anything model. arXiv preprint arXiv:2306.02245.
255.
ZhangXKunduAFunkhouserT, et al. (2023c) Nerflets: local radiance fields for efficient structure-aware 3d scene representation from 2d supervision. In: CVPR, pp. 8274–8284.
256.
ZhaoXDingWAnY, et al. (2023) Fast segment anything. arXiv preprint arXiv:2306.12156.
257.
ZhaoZChenSDingY, et al. (2024) A survey of optimization-based task and motion planning: from classical to learning approaches. arXiv preprint arXiv:2404.02817.
258.
ZhouCLoyCCDaiB (2022) Extract free dense labels from CLIP. In: European Conference on Computer Vision (ECCV), pp. 696–712. Springer.
259.
ZhouSChangHJiangS, et al. (2023) Feature 3dgs: supercharging 3d Gaussian splatting to enable distilled feature fields. arXiv preprint arXiv:2312.03203.
260.
ZhuXChenYTianH, et al. (2023) Ghost in the Minecraft: generally capable agents for open-world environments via large language models with text-based knowledge and memory. arXiv preprint arXiv:2305.17144.
261.
ZitkovichBYuTXuS, et al. (2023) RT-2: vision-language-action models transfer web knowledge to robotic control. In: CoRL.
262.
ZuoXSamangoueiPZhouY, et al. (2024) Fmgs: foundation model embedded 3d Gaussian splatting for holistic 3d scene understanding. arXiv preprint arXiv:2401.01970.
