{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vision-based-navigation-with-language-based","title":"Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention","arxiv_id":"1812.04155","date":"2018-12-10","proceeding":"CVPR 2019 6","authors":["Khanh Nguyen","Debadeepta Dey","Chris Brockett","Bill Dolan"],"abstract":"We present Vision-based Navigation with Language-based Assistance (VNLA), a\ngrounded vision-language task where an agent with visual perception is guided\nvia language to find objects in photorealistic indoor environments. The task\nemulates a real-world scenario in that (a) the requester may not know how to\nnavigate to the target objects and thus makes requests by only specifying\nhigh-level end-goals, and (b) the agent is capable of sensing when it is lost\nand querying an advisor, who is more qualified at the task, to obtain language\nsubgoals to make progress. To model language-based assistance, we develop a\ngeneral framework termed Imitation Learning with Indirect Intervention (I3L),\nand propose a solution that is effective on the VNLA task. Empirical results\nshow that this approach significantly improves the success rate of the learning\nagent over other baselines in both seen and unseen environments. Our code and\ndata are publicly available at https://github.com/debadeepta/vnla .","url_abs":"http://arxiv.org/abs/1812.04155v4","url_pdf":"http://arxiv.org/pdf/1812.04155v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vision-based-navigation-with-language-based","repo_url":"https://github.com/debadeepta/vnla","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"navigate","task_name":"Navigate"},{"task_slug":"vnla","task_name":"VNLA"},{"task_slug":"vision-based-navigation-with-language-based","task_name":"Vision-based navigation with language-based assistance"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1812.04155","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}