{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/conditional-object-centric-learning-from-1","title":"Conditional Object-Centric Learning from Video","arxiv_id":"2111.12594","date":"2021-11-24","proceeding":"ICLR 2022 4","authors":["Thomas Kipf","Gamaleldin F. Elsayed","Aravindh Mahendran","Austin Stone","Sara Sabour","Georg Heigold","Rico Jonschkowski","Alexey Dosovitskiy","Klaus Greff"],"abstract":"Object-centric representations are a promising path toward more systematic generalization by providing flexible abstractions upon which compositional world models can be built. Recent work on simple 2D and 3D datasets has shown that models with object-centric inductive biases can learn to segment and represent meaningful objects from the statistical structure of the data alone without the need for any supervision. However, such fully-unsupervised methods still fail to scale to diverse realistic data, despite the use of increasingly complex inductive biases such as priors for the size of objects or the 3D geometry of the scene. In this paper, we instead take a weakly-supervised approach and focus on how 1) using the temporal dynamics of video data in the form of optical flow and 2) conditioning the model on simple object location cues can be used to enable segmenting and tracking objects in significantly more realistic synthetic data. We introduce a sequential extension to Slot Attention which we train to predict optical flow for realistic looking synthetic scenes and show that conditioning the initial state of this model on a small set of hints, such as center of mass of objects in the first frame, is sufficient to significantly improve instance segmentation. These benefits generalize beyond the training distribution to novel objects, novel backgrounds, and to longer video sequences. We also find that such initial-state-conditioning can be used during inference as a flexible interface to query the model for specific objects or parts of objects, which could pave the way for a range of weakly-supervised approaches and allow more effective interaction with trained models.","url_abs":"https://arxiv.org/abs/2111.12594v2","url_pdf":"https://arxiv.org/pdf/2111.12594v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"conditional-object-centric-learning-from-1","repo_url":"https://github.com/google-research/slot-attention-video","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"conditional-object-centric-learning-from-1","repo_url":"https://github.com/a-imamshah/savi-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"conditional-object-centric-learning-from-1","repo_url":"https://github.com/Genera1Z/SAVi-PyTorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"3d-geometry","task_name":"3D geometry"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"object","task_name":"Object"},{"task_slug":"optical-flow-estimation","task_name":"Optical Flow Estimation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"systematic-generalization","task_name":"Systematic Generalization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2111.12594","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.12594"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/a-imamshah/savi-pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Genera1Z/SAVi-PyTorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-research/slot-attention-video","reach":null}],"summary":{"ran":11,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"9ac6a4d6ab4ed206","entry":"BboxEmbedding","repo":"Genera1Z/SAVi-PyTorch","repo_kind":"listed","path":"model.py","file_url":"https://github.com/Genera1Z/SAVi-PyTorch/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ac6a4d6ab4ed206"}},{"code_sha256_prefix":"f1be85158b3d9613","entry":"CorrectorPredictorTuple","repo":"google-research/slot-attention-video","repo_kind":"official","path":"savi/modules/video.py","file_url":"https://github.com/google-research/slot-attention-video/blob/HEAD/savi/modules/video.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f1be85158b3d9613"}},{"code_sha256_prefix":"18be91c2b6f9fa44","entry":"GRUCell","repo":"Genera1Z/SAVi-PyTorch","repo_kind":"listed","path":"model.py","file_url":"https://github.com/Genera1Z/SAVi-PyTorch/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"18be91c2b6f9fa44"}},{"code_sha256_prefix":"7fb19e46e2c683c0","entry":"MLP","repo":"Genera1Z/SAVi-PyTorch","repo_kind":"listed","path":"model.py","file_url":"https://github.com/Genera1Z/SAVi-PyTorch/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7fb19e46e2c683c0"}},{"code_sha256_prefix":"a50cd588d4e75a0a","entry":"MultiheadAttention","repo":"Genera1Z/SAVi-PyTorch","repo_kind":"listed","path":"model.py","file_url":"https://github.com/Genera1Z/SAVi-PyTorch/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a50cd588d4e75a0a"}},{"code_sha256_prefix":"cd8a7e68afbf24a4","entry":"PositionEmbedding2d","repo":"Genera1Z/SAVi-PyTorch","repo_kind":"listed","path":"model.py","file_url":"https://github.com/Genera1Z/SAVi-PyTorch/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd8a7e68afbf24a4"}},{"code_sha256_prefix":"b02a7373121b6ba6","entry":"Processor","repo":"google-research/slot-attention-video","repo_kind":"official","path":"savi/modules/video.py","file_url":"https://github.com/google-research/slot-attention-video/blob/HEAD/savi/modules/video.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b02a7373121b6ba6"}},{"code_sha256_prefix":"6e838b3b41544e46","entry":"SAVi","repo":"google-research/slot-attention-video","repo_kind":"official","path":"savi/modules/video.py","file_url":"https://github.com/google-research/slot-attention-video/blob/HEAD/savi/modules/video.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6e838b3b41544e46"}},{"code_sha256_prefix":"495a44630f8fac14","entry":"SimpleCnn","repo":"Genera1Z/SAVi-PyTorch","repo_kind":"listed","path":"model.py","file_url":"https://github.com/Genera1Z/SAVi-PyTorch/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"495a44630f8fac14"}},{"code_sha256_prefix":"9350a1846d302769","entry":"SlotAttention","repo":"Genera1Z/SAVi-PyTorch","repo_kind":"listed","path":"model.py","file_url":"https://github.com/Genera1Z/SAVi-PyTorch/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9350a1846d302769"}},{"code_sha256_prefix":"3d0d67682d50f148","entry":"TransformBlock","repo":"Genera1Z/SAVi-PyTorch","repo_kind":"listed","path":"model.py","file_url":"https://github.com/Genera1Z/SAVi-PyTorch/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d0d67682d50f148"}},{"code_sha256_prefix":"e3d84f21fbbdff63","entry":"Savi","repo":"Genera1Z/SAVi-PyTorch","repo_kind":"listed","path":"model.py","file_url":"https://github.com/Genera1Z/SAVi-PyTorch/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3d84f21fbbdff63"}},{"code_sha256_prefix":"ca0021be69127472","entry":"lecun_normal_","repo":"Genera1Z/SAVi-PyTorch","repo_kind":"listed","path":"model.py","file_url":"https://github.com/Genera1Z/SAVi-PyTorch/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca0021be69127472"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}