{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-occlusions-in-video-action-detection-1","title":"On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes","arxiv_id":"2410.19553","date":"2024-10-25","proceeding":"NeurIPS 2023 11","authors":["Rajat Modi","Vibhav Vineet","Yogesh Singh Rawat"],"abstract":"This paper explores the impact of occlusions in video action detection. We facilitate this study by introducing five new benchmark datasets namely O-UCF and O-JHMDB consisting of synthetically controlled static/dynamic occlusions, OVIS-UCF and OVIS-JHMDB consisting of occlusions with realistic motions and Real-OUCF for occlusions in realistic-world scenarios. We formally confirm an intuitive expectation: existing models suffer a lot as occlusion severity is increased and exhibit different behaviours when occluders are static vs when they are moving. We discover several intriguing phenomenon emerging in neural nets: 1) transformers can naturally outperform CNN models which might have even used occlusion as a form of data augmentation during training 2) incorporating symbolic-components like capsules to such backbones allows them to bind to occluders never even seen during training and 3) Islands of agreement can emerge in realistic images/videos without instance-level supervision, distillation or contrastive-based objectives2(eg. video-textual training). Such emergent properties allow us to derive simple yet effective training recipes which lead to robust occlusion models inductively satisfying the first two stages of the binding mechanism (grouping/segregation). Models leveraging these recipes outperform existing video action-detectors under occlusion by 32.3% on O-UCF, 32.7% on O-JHMDB & 2.6% on Real-OUCF in terms of the vMAP metric. The code for this work has been released at https://github.com/rajatmodi62/OccludedActionBenchmark.","url_abs":"https://arxiv.org/abs/2410.19553v1","url_pdf":"https://arxiv.org/pdf/2410.19553v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-occlusions-in-video-action-detection-1","repo_url":"https://github.com/rajatmodi62/occludedactionbenchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"action-detection","task_name":"Action Detection"},{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"video-action-detection","task_name":"Video Action Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.19553","atlas_url":"https://app.syntology.ai/?focus=2410.19553","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.19553"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/rajatmodi62/OccludedActionBenchmark","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rajatmodi62/occludedactionbenchmark","reach":{"status":"ok"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"a4723eab8df61dbe","entry":"attention_pool","repo":"rajatmodi62/OccludedActionBenchmark","repo_kind":"official","path":"codebase/codebase_islands/slowfast/models/attention.py","file_url":"https://github.com/rajatmodi62/OccludedActionBenchmark/blob/HEAD/codebase/codebase_islands/slowfast/models/attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4723eab8df61dbe"}},{"code_sha256_prefix":"156d0bf08e703f3c","entry":"cal_rel_pos_spatial","repo":"rajatmodi62/OccludedActionBenchmark","repo_kind":"official","path":"codebase/codebase_islands/slowfast/models/attention.py","file_url":"https://github.com/rajatmodi62/OccludedActionBenchmark/blob/HEAD/codebase/codebase_islands/slowfast/models/attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"156d0bf08e703f3c"}},{"code_sha256_prefix":"fa1acef3f6b3f71f","entry":"contrastive_forward","repo":"rajatmodi62/OccludedActionBenchmark","repo_kind":"official","path":"codebase/codebase_islands/slowfast/models/contrastive.py","file_url":"https://github.com/rajatmodi62/OccludedActionBenchmark/blob/HEAD/codebase/codebase_islands/slowfast/models/contrastive.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa1acef3f6b3f71f"}},{"code_sha256_prefix":"bcc1cdae3bb3212c","entry":"drop_path","repo":"rajatmodi62/OccludedActionBenchmark","repo_kind":"official","path":"codebase/codebase_islands/slowfast/models/common.py","file_url":"https://github.com/rajatmodi62/OccludedActionBenchmark/blob/HEAD/codebase/codebase_islands/slowfast/models/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bcc1cdae3bb3212c"}},{"code_sha256_prefix":"5fc6c9c916e0ebc7","entry":"get_rel_pos","repo":"rajatmodi62/OccludedActionBenchmark","repo_kind":"official","path":"codebase/codebase_islands/slowfast/models/attention.py","file_url":"https://github.com/rajatmodi62/OccludedActionBenchmark/blob/HEAD/codebase/codebase_islands/slowfast/models/attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5fc6c9c916e0ebc7"}},{"code_sha256_prefix":"1ef3753d70ebb220","entry":"l2_loss","repo":"rajatmodi62/OccludedActionBenchmark","repo_kind":"official","path":"codebase/codebase_islands/slowfast/models/contrastive.py","file_url":"https://github.com/rajatmodi62/OccludedActionBenchmark/blob/HEAD/codebase/codebase_islands/slowfast/models/contrastive.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1ef3753d70ebb220"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}