{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/entity-abstraction-in-visual-model-based","title":"Entity Abstraction in Visual Model-Based Reinforcement Learning","arxiv_id":"1910.12827","date":"2019-10-28","proceeding":null,"authors":["Rishi Veerapaneni","John D. Co-Reyes","Michael Chang","Michael Janner","Chelsea Finn","Jiajun Wu","Joshua B. Tenenbaum","Sergey Levine"],"abstract":"This paper tests the hypothesis that modeling a scene in terms of entities and their local interactions, as opposed to modeling the scene globally, provides a significant benefit in generalizing to physical tasks in a combinatorial space the learner has not encountered before. We present object-centric perception, prediction, and planning (OP3), which to the best of our knowledge is the first fully probabilistic entity-centric dynamic latent variable framework for model-based reinforcement learning that acquires entity representations from raw visual observations without supervision and uses them to predict and plan. OP3 enforces entity-abstraction -- symmetric processing of each entity representation with the same locally-scoped function -- which enables it to scale to model different numbers and configurations of objects from those in training. Our approach to solving the key technical challenge of grounding these entity representations to actual objects in the environment is to frame this variable binding problem as an inference problem, and we develop an interactive inference algorithm that uses temporal continuity and interactive feedback to bind information about object properties to the entity variables. On block-stacking tasks, OP3 generalizes to novel block configurations and more objects than observed during training, outperforming an oracle model that assumes access to object supervision and achieving two to three times better accuracy than a state-of-the-art video prediction model that does not exhibit entity abstraction.","url_abs":"https://arxiv.org/abs/1910.12827v5","url_pdf":"https://arxiv.org/pdf/1910.12827v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"entity-abstraction-in-visual-model-based","repo_url":"https://github.com/jcoreyes/OP3","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"model-based-reinforcement-learning","task_name":"Model-based Reinforcement Learning"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-discovery","task_name":"Object Discovery"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"variational-inference","task_name":"Variational Inference"},{"task_slug":"video-prediction","task_name":"Video Prediction"},{"task_slug":"model","task_name":"model"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1910.12827","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.12827"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jcoreyes/OP3","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"93d28ca67303e04b","entry":"clip_magnitude","repo":"jcoreyes/OP3","repo_kind":"official","path":"op3/pythonplusplus.py","file_url":"https://github.com/jcoreyes/OP3/blob/HEAD/op3/pythonplusplus.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93d28ca67303e04b"}},{"code_sha256_prefix":"9910e2fc297f8665","entry":"identity","repo":"jcoreyes/OP3","repo_kind":"official","path":"op3/pythonplusplus.py","file_url":"https://github.com/jcoreyes/OP3/blob/HEAD/op3/pythonplusplus.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9910e2fc297f8665"}},{"code_sha256_prefix":"4138766e854db009","entry":"is_numeric","repo":"jcoreyes/OP3","repo_kind":"official","path":"op3/pythonplusplus.py","file_url":"https://github.com/jcoreyes/OP3/blob/HEAD/op3/pythonplusplus.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4138766e854db009"}},{"code_sha256_prefix":"5cb0c55f58279f59","entry":"simple_separated_format","repo":"jcoreyes/OP3","repo_kind":"official","path":"op3/core/tabulate.py","file_url":"https://github.com/jcoreyes/OP3/blob/HEAD/op3/core/tabulate.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cb0c55f58279f59"}},{"code_sha256_prefix":"42612b635bfa2e19","entry":"add_prefix","repo":"jcoreyes/OP3","repo_kind":"official","path":"op3/core/logging.py","file_url":"https://github.com/jcoreyes/OP3/blob/HEAD/op3/core/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42612b635bfa2e19"}},{"code_sha256_prefix":"1029760706cbba76","entry":"append_log","repo":"jcoreyes/OP3","repo_kind":"official","path":"op3/core/logging.py","file_url":"https://github.com/jcoreyes/OP3/blob/HEAD/op3/core/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1029760706cbba76"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}