{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vision-and-language-navigation-interpreting","title":"Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments","arxiv_id":"1711.07280","date":"2017-11-20","proceeding":"CVPR 2018 6","authors":["Peter Anderson","Qi Wu","Damien Teney","Jake Bruce","Mark Johnson","Niko Sünderhauf","Ian Reid","Stephen Gould","Anton Van Den Hengel"],"abstract":"A robot that can carry out a natural-language instruction has been a dream\nsince before the Jetsons cartoon series imagined a life of leisure mediated by\na fleet of attentive robot helpers. It is a dream that remains stubbornly\ndistant. However, recent advances in vision and language methods have made\nincredible progress in closely related areas. This is significant because a\nrobot interpreting a natural-language navigation instruction on the basis of\nwhat it sees is carrying out a vision and language process that is similar to\nVisual Question Answering. Both tasks can be interpreted as visually grounded\nsequence-to-sequence translation problems, and many of the same methods are\napplicable. To enable and encourage the application of vision and language\nmethods to the problem of interpreting visually-grounded navigation\ninstructions, we present the Matterport3D Simulator -- a large-scale\nreinforcement learning environment based on real imagery. Using this simulator,\nwhich can in future support a range of embodied vision and language tasks, we\nprovide the first benchmark dataset for visually-grounded natural language\nnavigation in real buildings -- the Room-to-Room (R2R) dataset.","url_abs":"http://arxiv.org/abs/1711.07280v3","url_pdf":"http://arxiv.org/pdf/1711.07280v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vision-and-language-navigation-interpreting","repo_url":"https://github.com/peteanderson80/Matterport3DSimulator","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"vision-and-language-navigation-interpreting","repo_url":"https://github.com/MarSaKi/NvEM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"vision-and-language-navigation-interpreting","repo_url":"https://github.com/YicongHong/Entity-Graph-VLN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"vision-and-language-navigation-interpreting","repo_url":"https://github.com/YicongHong/Recurrent-VLN-BERT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"vision-and-language-navigation-interpreting","repo_url":"https://github.com/batra-mlp-lab/vln-chasing-ghosts","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"vision-and-language-navigation-interpreting","repo_url":"https://github.com/batra-mlp-lab/vln-sim2real","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"vision-and-language-navigation-interpreting","repo_url":"https://github.com/google-research-datasets/RxR","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"CC-BY-4.0"}},{"paper_slug":"vision-and-language-navigation-interpreting","repo_url":"https://github.com/hlr/vln-trans","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"vision-and-language-navigation","task_name":"Vision and Language Navigation"},{"task_slug":"visual-navigation","task_name":"Visual Navigation"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[{"slug":"room-to-room","name":"R2R","full_name":"Room-to-Room"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-navigation-on-room-to-room-1","task":"Visual Navigation","dataset":"R2R","model":"Seq2Seq baseline","rank_in_archive_order":11,"of":11,"metrics":{"spl":"0.18"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1711.07280","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}