{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/language-conditioned-imitation-learning-for","title":"Language-Conditioned Imitation Learning for Robot Manipulation Tasks","arxiv_id":"2010.12083","date":"2020-10-22","proceeding":"NeurIPS 2020 12","authors":["Simon Stepputtis","Joseph Campbell","Mariano Phielipp","Stefan Lee","Chitta Baral","Heni Ben Amor"],"abstract":"Imitation learning is a popular approach for teaching motor skills to robots. However, most approaches focus on extracting policy parameters from execution traces alone (i.e., motion trajectories and perceptual data). No adequate communication channel exists between the human expert and the robot to describe critical aspects of the task, such as the properties of the target object or the intended shape of the motion. Motivated by insights into the human teaching process, we introduce a method for incorporating unstructured natural language into imitation learning. At training time, the expert can provide demonstrations along with verbal descriptions in order to describe the underlying intent (e.g., \"go to the large green bowl\"). The training process then interrelates these two modalities to encode the correlations between language, perception, and motion. The resulting language-conditioned visuomotor policies can be conditioned at runtime on new human commands and instructions, which allows for more fine-grained control over the trained policies while also reducing situational ambiguity. We demonstrate in a set of simulation experiments how our approach can learn language-conditioned manipulation policies for a seven-degree-of-freedom robot arm and compare the results to a variety of alternative methods.","url_abs":"https://arxiv.org/abs/2010.12083v1","url_pdf":"https://arxiv.org/pdf/2010.12083v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"language-conditioned-imitation-learning-for","repo_url":"https://github.com/ir-lab/LanguagePolicies","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"robot-manipulation","task_name":"Robot Manipulation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2010.12083","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.12083"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ir-lab/LanguagePolicies","reach":null}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b6efd7ad2314bdda","entry":"BasisModel","repo":"ir-lab/LanguagePolicies","repo_kind":"official","path":"model_src/model.py","file_url":"https://github.com/ir-lab/LanguagePolicies/blob/HEAD/model_src/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6efd7ad2314bdda"}},{"code_sha256_prefix":"d0813eb27538cb5f","entry":"FeedbackController","repo":"ir-lab/LanguagePolicies","repo_kind":"official","path":"model_src/model.py","file_url":"https://github.com/ir-lab/LanguagePolicies/blob/HEAD/model_src/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0813eb27538cb5f"}},{"code_sha256_prefix":"b56a8ca69b4eb709","entry":"GloveEmbeddings","repo":"ir-lab/LanguagePolicies","repo_kind":"official","path":"model_src/model.py","file_url":"https://github.com/ir-lab/LanguagePolicies/blob/HEAD/model_src/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b56a8ca69b4eb709"}},{"code_sha256_prefix":"5961072ab83d693b","entry":"PolicyTranslationModel","repo":"ir-lab/LanguagePolicies","repo_kind":"official","path":"model_src/model.py","file_url":"https://github.com/ir-lab/LanguagePolicies/blob/HEAD/model_src/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5961072ab83d693b"}},{"code_sha256_prefix":"3a4d10e318bf8583","entry":"TopDownAttention","repo":"ir-lab/LanguagePolicies","repo_kind":"official","path":"model_src/model.py","file_url":"https://github.com/ir-lab/LanguagePolicies/blob/HEAD/model_src/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a4d10e318bf8583"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}