{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/object-region-video-transformers-1","title":"Object-Region Video Transformers","arxiv_id":"2110.06915","date":"2021-10-13","proceeding":"CVPR 2022 1","authors":["Roei Herzig","Elad Ben-Avraham","Karttikeya Mangalam","Amir Bar","Gal Chechik","Anna Rohrbach","Trevor Darrell","Amir Globerson"],"abstract":"Recently, video transformers have shown great success in video understanding, exceeding CNN performance; yet existing video transformer models do not explicitly model objects, although objects can be essential for recognizing actions. In this work, we present Object-Region Video Transformers (ORViT), an \\emph{object-centric} approach that extends video transformer layers with a block that directly incorporates object representations. The key idea is to fuse object-centric representations starting from early layers and propagate them into the transformer-layers, thus affecting the spatio-temporal representations throughout the network. Our ORViT block consists of two object-level streams: appearance and dynamics. In the appearance stream, an \"Object-Region Attention\" module applies self-attention over the patches and \\emph{object regions}. In this way, visual object regions interact with uniform patch tokens and enrich them with contextualized object information. We further model object dynamics via a separate \"Object-Dynamics Module\", which captures trajectory interactions, and show how to integrate the two streams. We evaluate our model on four tasks and five datasets: compositional and few-shot action recognition on SomethingElse, spatio-temporal action detection on AVA, and standard action recognition on Something-Something V2, Diving48 and Epic-Kitchen100. We show strong performance improvement across all tasks and datasets considered, demonstrating the value of a model that incorporates object representations into a transformer architecture. For code and pretrained models, visit the project page at \\url{https://roeiherz.github.io/ORViT/}","url_abs":"https://arxiv.org/abs/2110.06915v3","url_pdf":"https://arxiv.org/pdf/2110.06915v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"object-region-video-transformers-1","repo_url":"https://github.com/eladb3/orvit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"action-detection","task_name":"Action Detection"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"few-shot-action-recognition","task_name":"Few Shot Action Recognition"},{"task_slug":"few-shot-action-recognition","task_name":"Few-Shot action recognition"},{"task_slug":"object","task_name":"Object"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-recognition-on-ava-v2-2","task":"Action Recognition","dataset":"AVA v2.2","model":"ORViT MViT-B, 16x4 (K400 pretraining)","rank_in_archive_order":34,"of":38,"metrics":{"mAP":"26.6"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-on-diving-48","task":"Action Recognition","dataset":"Diving-48","model":"ORViT TimeSformer","rank_in_archive_order":7,"of":18,"metrics":{"Accuracy":"88.0"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-on-epic-kitchens-100","task":"Action Recognition","dataset":"EPIC-KITCHENS-100","model":"ORViT Mformer-L (ORViT blocks)","rank_in_archive_order":16,"of":32,"metrics":{"Action@1":"45.7","Noun@1":"58.7","Verb@1":"68.4"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"ORViT Mformer-L (ORViT blocks)","rank_in_archive_order":46,"of":123,"metrics":{"GFLOPs":"N/A","Parameters":"N/A","Top-1 Accuracy":"69.5","Top-5 Accuracy":"91.5"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"ORViT Mformer (ORViT blocks)","rank_in_archive_order":56,"of":123,"metrics":{"GFLOPs":"N/A","Parameters":"N/A","Top-1 Accuracy":"67.9","Top-5 Accuracy":"90.5"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.06915","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.06915"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eladb3/orvit","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"listed":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"83c9daa7db244c42","entry":"qkv_attn","repo":"eladb3/orvit","repo_kind":"listed","path":"slowfast/models/attention.py","file_url":"https://github.com/eladb3/orvit/blob/HEAD/slowfast/models/attention.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"83c9daa7db244c42"}},{"code_sha256_prefix":"a4723eab8df61dbe","entry":"attention_pool","repo":"eladb3/orvit","repo_kind":"listed","path":"slowfast/models/attention.py","file_url":"https://github.com/eladb3/orvit/blob/HEAD/slowfast/models/attention.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a4723eab8df61dbe"}},{"code_sha256_prefix":"bf66d6b3f9e8a4ec","entry":"boxes_to_layout","repo":"eladb3/orvit","repo_kind":"listed","path":"slowfast/models/ORViT/layout.py","file_url":"https://github.com/eladb3/orvit/blob/HEAD/slowfast/models/ORViT/layout.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"bf66d6b3f9e8a4ec"}},{"code_sha256_prefix":"13fe414f1312e550","entry":"boxes_to_mask","repo":"eladb3/orvit","repo_kind":"listed","path":"slowfast/models/ORViT/layout.py","file_url":"https://github.com/eladb3/orvit/blob/HEAD/slowfast/models/ORViT/layout.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"13fe414f1312e550"}},{"code_sha256_prefix":"bcc1cdae3bb3212c","entry":"drop_path","repo":"eladb3/orvit","repo_kind":"listed","path":"slowfast/models/ORViT/orvit.py","file_url":"https://github.com/eladb3/orvit/blob/HEAD/slowfast/models/ORViT/orvit.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"bcc1cdae3bb3212c"}},{"code_sha256_prefix":"bd99fa4d54003b3b","entry":"get_loss_func","repo":"eladb3/orvit","repo_kind":"listed","path":"slowfast/models/losses.py","file_url":"https://github.com/eladb3/orvit/blob/HEAD/slowfast/models/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"bd99fa4d54003b3b"}},{"code_sha256_prefix":"26736b43ee6ba1fd","entry":"masks_to_layout","repo":"eladb3/orvit","repo_kind":"listed","path":"slowfast/models/ORViT/layout.py","file_url":"https://github.com/eladb3/orvit/blob/HEAD/slowfast/models/ORViT/layout.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"26736b43ee6ba1fd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}