{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mid-level-visual-representations-improve","title":"Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies","arxiv_id":"1812.11971","date":"2018-12-31","proceeding":null,"authors":["Alexander Sax","Bradley Emi","Amir R. Zamir","Leonidas Guibas","Silvio Savarese","Jitendra Malik"],"abstract":"How much does having visual priors about the world (e.g. the fact that the\nworld is 3D) assist in learning to perform downstream motor tasks (e.g.\ndelivering a package)? We study this question by integrating a generic\nperceptual skill set (e.g. a distance estimator, an edge detector, etc.) within\na reinforcement learning framework--see Figure 1. This skill set (hereafter\nmid-level perception) provides the policy with a more processed state of the\nworld compared to raw images.\n  We find that using a mid-level perception confers significant advantages over\ntraining end-to-end from scratch (i.e. not leveraging priors) in\nnavigation-oriented tasks. Agents are able to generalize to situations where\nthe from-scratch approach fails and training becomes significantly more sample\nefficient. However, we show that realizing these gains requires careful\nselection of the mid-level perceptual skills. Therefore, we refine our findings\ninto an efficient max-coverage feature set that can be adopted in lieu of raw\nimages. We perform our study in completely separate buildings for training and\ntesting and compare against visually blind baseline policies and\nstate-of-the-art feature learning methods.","url_abs":"http://arxiv.org/abs/1812.11971v3","url_pdf":"http://arxiv.org/pdf/1812.11971v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mid-level-visual-representations-improve","repo_url":"https://github.com/alexsax/midlevel-reps","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1812.11971","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}