{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-to-estimate-hidden-motions-with","title":"Learning to Estimate Hidden Motions with Global Motion Aggregation","arxiv_id":"2104.02409","date":"2021-04-06","proceeding":"ICCV 2021 10","authors":["Shihao Jiang","Dylan Campbell","Yao Lu","Hongdong Li","Richard Hartley"],"abstract":"Occlusions pose a significant challenge to optical flow algorithms that rely on local evidences. We consider an occluded point to be one that is imaged in the first frame but not in the next, a slight overloading of the standard definition since it also includes points that move out-of-frame. Estimating the motion of these points is extremely difficult, particularly in the two-frame setting. Previous work relies on CNNs to learn occlusions, without much success, or requires multiple frames to reason about occlusions using temporal smoothness. In this paper, we argue that the occlusion problem can be better solved in the two-frame case by modelling image self-similarities. We introduce a global motion aggregation module, a transformer-based approach to find long-range dependencies between pixels in the first image, and perform global aggregation on the corresponding motion features. We demonstrate that the optical flow estimates in the occluded regions can be significantly improved without damaging the performance in non-occluded regions. This approach obtains new state-of-the-art results on the challenging Sintel dataset, improving the average end-point error by 13.6% on Sintel Final and 13.7% on Sintel Clean. At the time of submission, our method ranks first on these benchmarks among all published and unpublished approaches. Code is available at https://github.com/zacjiang/GMA","url_abs":"https://arxiv.org/abs/2104.02409v3","url_pdf":"https://arxiv.org/pdf/2104.02409v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-to-estimate-hidden-motions-with","repo_url":"https://github.com/zacjiang/GMA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"learning-to-estimate-hidden-motions-with","repo_url":"https://github.com/open-mmlab/mmflow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"optical-flow-estimation","task_name":"Optical Flow Estimation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/optical-flow-estimation-on-kitti-2015-train","task":"Optical Flow Estimation","dataset":"KITTI 2015 (train)","model":"GMA","rank_in_archive_order":9,"of":19,"metrics":{"EPE":"4.69","F1-all":"17.1"},"uses_additional_data":false},{"leaderboard":"/sota/optical-flow-estimation-on-sintel-clean","task":"Optical Flow Estimation","dataset":"Sintel-clean","model":"GMA","rank_in_archive_order":8,"of":29,"metrics":{"Average End-Point Error":"1.388"},"uses_additional_data":false},{"leaderboard":"/sota/optical-flow-estimation-on-sintel-final","task":"Optical Flow Estimation","dataset":"Sintel-final","model":"GMA","rank_in_archive_order":6,"of":28,"metrics":{"Average End-Point Error":"2.470"},"uses_additional_data":false},{"leaderboard":"/sota/optical-flow-estimation-on-spring","task":"Optical Flow Estimation","dataset":"Spring","model":"GMA","rank_in_archive_order":8,"of":11,"metrics":{"1px total":"7.074"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2104.02409","atlas_url":"https://app.syntology.ai/?focus=2104.02409","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.02409"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zacjiang/GMA","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/open-mmlab/mmflow","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"421e324f0d36348f","entry":"Aggregate","repo":"zacjiang/GMA","repo_kind":"official","path":"core/gma.py","file_url":"https://github.com/zacjiang/GMA/blob/HEAD/core/gma.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"WTFPL","inline_ok":false,"mcp_get_code":{"code_sha256":"421e324f0d36348f"}},{"code_sha256_prefix":"da54ea98547c05a0","entry":"end_point_error","repo":"open-mmlab/mmflow","repo_kind":"listed","path":"mmflow/core/evaluation/metrics.py","file_url":"https://github.com/open-mmlab/mmflow/blob/HEAD/mmflow/core/evaluation/metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"da54ea98547c05a0"}},{"code_sha256_prefix":"986a031d5cc8821b","entry":"end_point_error_map","repo":"open-mmlab/mmflow","repo_kind":"listed","path":"mmflow/core/evaluation/metrics.py","file_url":"https://github.com/open-mmlab/mmflow/blob/HEAD/mmflow/core/evaluation/metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"986a031d5cc8821b"}},{"code_sha256_prefix":"460ed36a715ed976","entry":"optical_flow_outliers","repo":"open-mmlab/mmflow","repo_kind":"listed","path":"mmflow/core/evaluation/metrics.py","file_url":"https://github.com/open-mmlab/mmflow/blob/HEAD/mmflow/core/evaluation/metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"460ed36a715ed976"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}