{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-matters-in-language-conditioned-robotic","title":"What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data","arxiv_id":"2204.06252","date":"2022-04-13","proceeding":null,"authors":["Oier Mees","Lukas Hermann","Wolfram Burgard"],"abstract":"A long-standing goal in robotics is to build robots that can perform a wide range of daily tasks from perceptions obtained with their onboard sensors and specified only via natural language. While recently substantial advances have been achieved in language-driven robotics by leveraging end-to-end learning from pixels, there is no clear and well-understood process for making various design choices due to the underlying variation in setups. In this paper, we conduct an extensive study of the most critical challenges in learning language conditioned policies from offline free-form imitation datasets. We further identify architectural and algorithmic techniques that improve performance, such as a hierarchical decomposition of the robot control learning, a multimodal transformer encoder, discrete latent plans and a self-supervised contrastive loss that aligns video and language representations. By combining the results of our investigation with our improved model components, we are able to present a novel approach that significantly outperforms the state of the art on the challenging language conditioned long-horizon robot manipulation CALVIN benchmark. We have open-sourced our implementation to facilitate future research in learning to perform many complex manipulation skills in a row specified with natural language. Codebase and trained models available at http://hulc.cs.uni-freiburg.de","url_abs":"https://arxiv.org/abs/2204.06252v2","url_pdf":"https://arxiv.org/pdf/2204.06252v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-matters-in-language-conditioned-robotic","repo_url":"https://github.com/mees/hulc","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"what-matters-in-language-conditioned-robotic","repo_url":"https://github.com/lukashermann/hulc","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"robot-manipulation","task_name":"Robot Manipulation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2204.06252","atlas_url":"https://app.syntology.ai/?focus=2204.06252","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.06252"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mees/hulc","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lukashermann/hulc","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":3},"by_repo_kind":{"listed":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b008129d68433002","entry":"get_log_dir","repo":"lukashermann/hulc","repo_kind":"listed","path":"hulc/evaluation/run_multiple.py","file_url":"https://github.com/lukashermann/hulc/blob/HEAD/hulc/evaluation/run_multiple.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b008129d68433002"}},{"code_sha256_prefix":"068b88e1544a6518","entry":"intervals","repo":"lukashermann/hulc","repo_kind":"listed","path":"hulc/evaluation/run_multiple.py","file_url":"https://github.com/lukashermann/hulc/blob/HEAD/hulc/evaluation/run_multiple.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"068b88e1544a6518"}},{"code_sha256_prefix":"1dfe6f74142d8825","entry":"load_eval_data","repo":"lukashermann/hulc","repo_kind":"listed","path":"hulc/evaluation/create_plots.py","file_url":"https://github.com/lukashermann/hulc/blob/HEAD/hulc/evaluation/create_plots.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1dfe6f74142d8825"}},{"code_sha256_prefix":"256f624439688a69","entry":"get_epoch","repo":"lukashermann/hulc","repo_kind":"listed","path":"hulc/evaluation/evaluate_policy.py","file_url":"https://github.com/lukashermann/hulc/blob/HEAD/hulc/evaluation/evaluate_policy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"256f624439688a69"}},{"code_sha256_prefix":"8ab9acda1d0ae3ee","entry":"setup_callbacks","repo":"lukashermann/hulc","repo_kind":"listed","path":"hulc/training.py","file_url":"https://github.com/lukashermann/hulc/blob/HEAD/hulc/training.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ab9acda1d0ae3ee"}},{"code_sha256_prefix":"a183b4deb3b42f2f","entry":"setup_logger","repo":"lukashermann/hulc","repo_kind":"listed","path":"hulc/training.py","file_url":"https://github.com/lukashermann/hulc/blob/HEAD/hulc/training.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a183b4deb3b42f2f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}