{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lm-nav-robotic-navigation-with-large-pre","title":"LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action","arxiv_id":"2207.04429","date":"2022-07-10","proceeding":null,"authors":["Dhruv Shah","Blazej Osinski","Brian Ichter","Sergey Levine"],"abstract":"Goal-conditioned policies for robotic navigation can be trained on large, unannotated datasets, providing for good generalization to real-world settings. However, particularly in vision-based settings where specifying goals requires an image, this makes for an unnatural interface. Language provides a more convenient modality for communication with robots, but contemporary methods typically require expensive supervision, in the form of trajectories annotated with language descriptions. We present a system, LM-Nav, for robotic navigation that enjoys the benefits of training on unannotated large datasets of trajectories, while still providing a high-level interface to the user. Instead of utilizing a labeled instruction following dataset, we show that such a system can be constructed entirely out of pre-trained models for navigation (ViNG), image-language association (CLIP), and language modeling (GPT-3), without requiring any fine-tuning or language-annotated robot data. We instantiate LM-Nav on a real-world mobile robot and demonstrate long-horizon navigation through complex, outdoor environments from natural language instructions. For videos of our experiments, code release, and an interactive Colab notebook that runs in your browser, please check out our project page https://sites.google.com/view/lmnav","url_abs":"https://arxiv.org/abs/2207.04429v2","url_pdf":"https://arxiv.org/pdf/2207.04429v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lm-nav-robotic-navigation-with-large-pre","repo_url":"https://github.com/blazejosinski/lm_nav","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.04429","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.04429"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/blazejosinski/lm_nav","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f794c8e3220abf6a","entry":"bytes2im","repo":"blazejosinski/lm_nav","repo_kind":"official","path":"lm_nav/utils.py","file_url":"https://github.com/blazejosinski/lm_nav/blob/HEAD/lm_nav/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f794c8e3220abf6a"}},{"code_sha256_prefix":"016e7731808ea841","entry":"generic_language_model_api_call","repo":"blazejosinski/lm_nav","repo_kind":"official","path":"lm_nav/landmark_extraction.py","file_url":"https://github.com/blazejosinski/lm_nav/blob/HEAD/lm_nav/landmark_extraction.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"016e7731808ea841"}},{"code_sha256_prefix":"ed1426f2a4432b88","entry":"get_hdf5_leaf_names","repo":"blazejosinski/lm_nav","repo_kind":"official","path":"lm_nav/utils.py","file_url":"https://github.com/blazejosinski/lm_nav/blob/HEAD/lm_nav/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed1426f2a4432b88"}},{"code_sha256_prefix":"112c2db62806094e","entry":"im2bytes","repo":"blazejosinski/lm_nav","repo_kind":"official","path":"lm_nav/utils.py","file_url":"https://github.com/blazejosinski/lm_nav/blob/HEAD/lm_nav/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"112c2db62806094e"}},{"code_sha256_prefix":"f584724aa6279142","entry":"remove_article","repo":"blazejosinski/lm_nav","repo_kind":"official","path":"lm_nav/landmark_extraction.py","file_url":"https://github.com/blazejosinski/lm_nav/blob/HEAD/lm_nav/landmark_extraction.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f584724aa6279142"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}