{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/language-driven-representation-learning-for","title":"Language-Driven Representation Learning for Robotics","arxiv_id":"2302.12766","date":"2023-02-24","proceeding":null,"authors":["Siddharth Karamcheti","Suraj Nair","Annie S. Chen","Thomas Kollar","Chelsea Finn","Dorsa Sadigh","Percy Liang"],"abstract":"Recent work in visual representation learning for robotics demonstrates the viability of learning from large video datasets of humans performing everyday tasks. Leveraging methods such as masked autoencoding and contrastive learning, these representations exhibit strong transfer to policy learning for visuomotor control. But, robot learning encompasses a diverse set of problems beyond control including grasp affordance prediction, language-conditioned imitation learning, and intent scoring for human-robot collaboration, amongst others. First, we demonstrate that existing representations yield inconsistent results across these tasks: masked autoencoding approaches pick up on low-level spatial features at the cost of high-level semantics, while contrastive learning approaches capture the opposite. We then introduce Voltron, a framework for language-driven representation learning from human videos and associated captions. Voltron trades off language-conditioned visual reconstruction to learn low-level visual patterns, and visually-grounded language generation to encode high-level semantics. We also construct a new evaluation suite spanning five distinct robot learning problems $\\unicode{x2013}$ a unified platform for holistically evaluating visual representations for robotics. Through comprehensive, controlled experiments across all five problems, we find that Voltron's language-driven representations outperform the prior state-of-the-art, especially on targeted problems requiring higher-level features.","url_abs":"https://arxiv.org/abs/2302.12766v1","url_pdf":"https://arxiv.org/pdf/2302.12766v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"language-driven-representation-learning-for","repo_url":"https://github.com/siddk/voltron-evaluation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-driven-representation-learning-for","repo_url":"https://github.com/siddk/voltron-robotics","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2302.12766","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.12766"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/siddk/voltron-robotics","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/siddk/voltron-evaluation","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"40e9f2fb3e486b05","entry":"build_datamodule","repo":"siddk/voltron-evaluation","repo_kind":"official","path":"voltron_evaluation/grasping/preprocessing.py","file_url":"https://github.com/siddk/voltron-evaluation/blob/HEAD/voltron_evaluation/grasping/preprocessing.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"40e9f2fb3e486b05"}},{"code_sha256_prefix":"f7040983eb9ab964","entry":"build_datamodule","repo":"siddk/voltron-evaluation","repo_kind":"official","path":"voltron_evaluation/langref/preprocessing.py","file_url":"https://github.com/siddk/voltron-evaluation/blob/HEAD/voltron_evaluation/langref/preprocessing.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7040983eb9ab964"}},{"code_sha256_prefix":"2789bdfe6df3cb6a","entry":"get_path","repo":"siddk/voltron-robotics","repo_kind":"official","path":"voltron/preprocessing/core.py","file_url":"https://github.com/siddk/voltron-robotics/blob/HEAD/voltron/preprocessing/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2789bdfe6df3cb6a"}},{"code_sha256_prefix":"d356844b20b60e44","entry":"instantiate_detector","repo":"siddk/voltron-evaluation","repo_kind":"official","path":"voltron_evaluation/langref/adapter.py","file_url":"https://github.com/siddk/voltron-evaluation/blob/HEAD/voltron_evaluation/langref/adapter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d356844b20b60e44"}},{"code_sha256_prefix":"c60a652253161341","entry":"instantiate_segmenter","repo":"siddk/voltron-evaluation","repo_kind":"official","path":"voltron_evaluation/grasping/adapter.py","file_url":"https://github.com/siddk/voltron-evaluation/blob/HEAD/voltron_evaluation/grasping/adapter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c60a652253161341"}},{"code_sha256_prefix":"29134be0dcfd36fc","entry":"serialize_epoch","repo":"siddk/voltron-robotics","repo_kind":"official","path":"voltron/preprocessing/core.py","file_url":"https://github.com/siddk/voltron-robotics/blob/HEAD/voltron/preprocessing/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29134be0dcfd36fc"}},{"code_sha256_prefix":"c8c4e5a3dc4f02b7","entry":"set_global_seed","repo":"siddk/voltron-evaluation","repo_kind":"official","path":"voltron_evaluation/util/random.py","file_url":"https://github.com/siddk/voltron-evaluation/blob/HEAD/voltron_evaluation/util/random.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8c4e5a3dc4f02b7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}