{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-evaluation-of-embodied-navigation-agents","title":"On Evaluation of Embodied Navigation Agents","arxiv_id":"1807.06757","date":"2018-07-18","proceeding":null,"authors":["Peter Anderson","Angel Chang","Devendra Singh Chaplot","Alexey Dosovitskiy","Saurabh Gupta","Vladlen Koltun","Jana Kosecka","Jitendra Malik","Roozbeh Mottaghi","Manolis Savva","Amir R. Zamir"],"abstract":"Skillful mobile operation in three-dimensional environments is a primary\ntopic of study in Artificial Intelligence. The past two years have seen a surge\nof creative work on navigation. This creative output has produced a plethora of\nsometimes incompatible task definitions and evaluation protocols. To coordinate\nongoing and future research in this area, we have convened a working group to\nstudy empirical methodology in navigation research. The present document\nsummarizes the consensus recommendations of this working group. We discuss\ndifferent problem statements and the role of generalization, present evaluation\nmeasures, and provide standard scenarios that can be used for benchmarking.","url_abs":"http://arxiv.org/abs/1807.06757v1","url_pdf":"http://arxiv.org/pdf/1807.06757v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-evaluation-of-embodied-navigation-agents","repo_url":"https://github.com/StanfordVL/GibsonSim2RealCallenge","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"on-evaluation-of-embodied-navigation-agents","repo_url":"https://github.com/StanfordVL/GibsonSim2RealChallenge","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"on-evaluation-of-embodied-navigation-agents","repo_url":"https://github.com/StanfordVL/iGibsonChallenge2021","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"on-evaluation-of-embodied-navigation-agents","repo_url":"https://github.com/facebookresearch/habitat-api","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"on-evaluation-of-embodied-navigation-agents","repo_url":"https://github.com/facebookresearch/habitat-challenge","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"on-evaluation-of-embodied-navigation-agents","repo_url":"https://github.com/facebookresearch/habitat-lab","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"on-evaluation-of-embodied-navigation-agents","repo_url":"https://github.com/facebookresearch/soundspaces-challenge","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"on-evaluation-of-embodied-navigation-agents","repo_url":"https://github.com/google-research-datasets/RxR","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"CC-BY-4.0"}},{"paper_slug":"on-evaluation-of-embodied-navigation-agents","repo_url":"https://github.com/iit-pavis/look_around_and_learn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"on-evaluation-of-embodied-navigation-agents","repo_url":"https://github.com/vkartik97/habitat-challenge-two-docker","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1807.06757","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}