{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/object-segmentation-from-common-fate-motion","title":"Object segmentation from common fate: Motion energy processing enables human-like zero-shot generalization to random dot stimuli","arxiv_id":"2411.01505","date":"2024-11-03","proceeding":null,"authors":["Matthias Tangemann","Matthias Kümmerer","Matthias Bethge"],"abstract":"Humans excel at detecting and segmenting moving objects according to the Gestalt principle of \"common fate\". Remarkably, previous works have shown that human perception generalizes this principle in a zero-shot fashion to unseen textures or random dots. In this work, we seek to better understand the computational basis for this capability by evaluating a broad range of optical flow models and a neuroscience inspired motion energy model for zero-shot figure-ground segmentation of random dot stimuli. Specifically, we use the extensively validated motion energy model proposed by Simoncelli and Heeger in 1998 which is fitted to neural recordings in cortex area MT. We find that a cross section of 40 deep optical flow models trained on different datasets struggle to estimate motion patterns in random dot videos, resulting in poor figure-ground segmentation performance. Conversely, the neuroscience-inspired model significantly outperforms all optical flow models on this task. For a direct comparison to human perception, we conduct a psychophysical study using a shape identification task as a proxy to measure human segmentation performance. All state-of-the-art optical flow models fall short of human performance, but only the motion energy model matches human capability. This neuroscience-inspired model successfully addresses the lack of human-like zero-shot generalization to random dot stimuli in current computer vision models, and thus establishes a compelling link between the Gestalt psychology of human object perception and cortical motion processing in the brain. Code, models and datasets are available at https://github.com/mtangemann/motion_energy_segmentation","url_abs":"https://arxiv.org/abs/2411.01505v1","url_pdf":"https://arxiv.org/pdf/2411.01505v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"object-segmentation-from-common-fate-motion","repo_url":"https://github.com/mtangemann/motion_energy_segmentation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"optical-flow-estimation","task_name":"Optical Flow Estimation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"zero-shot-generalization","task_name":"Zero-shot Generalization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.01505","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.01505"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mtangemann/motion_energy_segmentation","reach":{"status":"ok"}}],"summary":{"unverified":10},"by_repo_kind":{"official":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"1d683e6be2b7ecce","entry":"atan3","repo":"mtangemann/motion_energy_segmentation","repo_kind":"official","path":"motion_energy_segmentation/simoncelli_heeger.py","file_url":"https://github.com/mtangemann/motion_energy_segmentation/blob/HEAD/motion_energy_segmentation/simoncelli_heeger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1d683e6be2b7ecce"}},{"code_sha256_prefix":"3641a4db92739b3c","entry":"blurDn3","repo":"mtangemann/motion_energy_segmentation","repo_kind":"official","path":"motion_energy_segmentation/simoncelli_heeger.py","file_url":"https://github.com/mtangemann/motion_energy_segmentation/blob/HEAD/motion_energy_segmentation/simoncelli_heeger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3641a4db92739b3c"}},{"code_sha256_prefix":"e80b052ebb25be5a","entry":"gaussian_blur","repo":"mtangemann/motion_energy_segmentation","repo_kind":"official","path":"motion_energy_segmentation/simoncelli_heeger.py","file_url":"https://github.com/mtangemann/motion_energy_segmentation/blob/HEAD/motion_energy_segmentation/simoncelli_heeger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e80b052ebb25be5a"}},{"code_sha256_prefix":"6c02bcdd773f694d","entry":"iou","repo":"mtangemann/motion_energy_segmentation","repo_kind":"official","path":"motion_energy_segmentation/evaluate_shape_identification.py","file_url":"https://github.com/mtangemann/motion_energy_segmentation/blob/HEAD/motion_energy_segmentation/evaluate_shape_identification.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6c02bcdd773f694d"}},{"code_sha256_prefix":"ac0061565b0b8fb2","entry":"load_json","repo":"mtangemann/motion_energy_segmentation","repo_kind":"official","path":"motion_energy_segmentation/io_utils.py","file_url":"https://github.com/mtangemann/motion_energy_segmentation/blob/HEAD/motion_energy_segmentation/io_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac0061565b0b8fb2"}},{"code_sha256_prefix":"a964f0bf9f047ba8","entry":"load_video","repo":"mtangemann/motion_energy_segmentation","repo_kind":"official","path":"motion_energy_segmentation/io_utils.py","file_url":"https://github.com/mtangemann/motion_energy_segmentation/blob/HEAD/motion_energy_segmentation/io_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a964f0bf9f047ba8"}},{"code_sha256_prefix":"1f9b5cb8b56522c4","entry":"object_kinematogram","repo":"mtangemann/motion_energy_segmentation","repo_kind":"official","path":"motion_energy_segmentation/transforms.py","file_url":"https://github.com/mtangemann/motion_energy_segmentation/blob/HEAD/motion_energy_segmentation/transforms.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1f9b5cb8b56522c4"}},{"code_sha256_prefix":"3a939b9943ca2fb6","entry":"sequence_loss","repo":"mtangemann/motion_energy_segmentation","repo_kind":"official","path":"motion_energy_segmentation/flow_former_plus_plus/_extern/core/loss.py","file_url":"https://github.com/mtangemann/motion_energy_segmentation/blob/HEAD/motion_energy_segmentation/flow_former_plus_plus/_extern/core/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3a939b9943ca2fb6"}},{"code_sha256_prefix":"b25563140411b0d7","entry":"sequence_loss_smooth","repo":"mtangemann/motion_energy_segmentation","repo_kind":"official","path":"motion_energy_segmentation/flow_former_plus_plus/_extern/core/loss.py","file_url":"https://github.com/mtangemann/motion_energy_segmentation/blob/HEAD/motion_energy_segmentation/flow_former_plus_plus/_extern/core/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b25563140411b0d7"}},{"code_sha256_prefix":"5464e22d6859392a","entry":"smooth_l1_loss","repo":"mtangemann/motion_energy_segmentation","repo_kind":"official","path":"motion_energy_segmentation/flow_former_plus_plus/_extern/core/loss.py","file_url":"https://github.com/mtangemann/motion_energy_segmentation/blob/HEAD/motion_energy_segmentation/flow_former_plus_plus/_extern/core/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5464e22d6859392a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}