{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/motion-meets-attention-video-motion-prompts","title":"Motion meets Attention: Video Motion Prompts","arxiv_id":"2407.03179","date":"2024-07-03","proceeding":null,"authors":["Qixiang Chen","Lei Wang","Piotr Koniusz","Tom Gedeon"],"abstract":"Videos contain rich spatio-temporal information. Traditional methods for extracting motion, used in tasks such as action recognition, often rely on visual contents rather than precise motion features. This phenomenon is referred to as 'blind motion extraction' behavior, which proves inefficient in capturing motions of interest due to a lack of motion-guided cues. Recently, attention mechanisms have enhanced many computer vision tasks by effectively highlighting salient visual areas. Inspired by this, we propose a modified Sigmoid function with learnable slope and shift parameters as an attention mechanism to modulate motion signals from frame differencing maps. This approach generates a sequence of attention maps that enhance the processing of motion-related video content. To ensure temporal continuity and smoothness of the attention maps, we apply pair-wise temporal attention variation regularization to remove unwanted motions (e.g., noise) while preserving important ones. We then perform Hadamard product between each pair of attention maps and the original video frames to highlight the evolving motions of interest over time. These highlighted motions, termed video motion prompts, are subsequently used as inputs to the model instead of the original video frames. We formalize this process as a motion prompt layer and incorporate the regularization term into the loss function to learn better motion prompts. This layer serves as an adapter between the model and the video data, bridging the gap between traditional 'blind motion extraction' and the extraction of relevant motions of interest. We show that our lightweight, plug-and-play motion prompt layer seamlessly integrates into models like SlowFast, X3D, and TimeSformer, enhancing performance on benchmarks such as FineGym and MPII Cooking 2.","url_abs":"https://arxiv.org/abs/2407.03179v2","url_pdf":"https://arxiv.org/pdf/2407.03179v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"motion-meets-attention-video-motion-prompts","repo_url":"https://github.com/q1xiangchen/vmps","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"timesformer","method_name":"TimeSformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.03179","atlas_url":"https://app.syntology.ai/?focus=2407.03179","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.03179"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/q1xiangchen/vmps","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"ran_draft_wrong":1,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c0504d333a664585","entry":"construct_optimizer","repo":"q1xiangchen/vmps","repo_kind":"official","path":"timesformer/models/optimizer.py","file_url":"https://github.com/q1xiangchen/vmps/blob/HEAD/timesformer/models/optimizer.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0504d333a664585"}},{"code_sha256_prefix":"c8a06e6ec83e9e27","entry":"extract_layer","repo":"q1xiangchen/vmps","repo_kind":"official","path":"timesformer/models/helpers.py","file_url":"https://github.com/q1xiangchen/vmps/blob/HEAD/timesformer/models/helpers.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8a06e6ec83e9e27"}},{"code_sha256_prefix":"c90b5f606f54d4f7","entry":"get_loss_func","repo":"q1xiangchen/vmps","repo_kind":"official","path":"timesformer/models/losses.py","file_url":"https://github.com/q1xiangchen/vmps/blob/HEAD/timesformer/models/losses.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c90b5f606f54d4f7"}},{"code_sha256_prefix":"54e11386110ff042","entry":"get_same_padding","repo":"q1xiangchen/vmps","repo_kind":"official","path":"timesformer/models/conv2d_same.py","file_url":"https://github.com/q1xiangchen/vmps/blob/HEAD/timesformer/models/conv2d_same.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54e11386110ff042"}},{"code_sha256_prefix":"83ec90904c246b55","entry":"load_state_dict","repo":"q1xiangchen/vmps","repo_kind":"official","path":"timesformer/models/helpers.py","file_url":"https://github.com/q1xiangchen/vmps/blob/HEAD/timesformer/models/helpers.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83ec90904c246b55"}},{"code_sha256_prefix":"e93bd3734824ac34","entry":"m_sigmoid","repo":"q1xiangchen/vmps","repo_kind":"official","path":"timesformer/models/vmps.py","file_url":"https://github.com/q1xiangchen/vmps/blob/HEAD/timesformer/models/vmps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e93bd3734824ac34"}},{"code_sha256_prefix":"a5912e0e800a1929","entry":"pad_same","repo":"q1xiangchen/vmps","repo_kind":"official","path":"timesformer/models/conv2d_same.py","file_url":"https://github.com/q1xiangchen/vmps/blob/HEAD/timesformer/models/conv2d_same.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5912e0e800a1929"}},{"code_sha256_prefix":"6c352461f0cf088d","entry":"rearrange_tensor","repo":"q1xiangchen/vmps","repo_kind":"official","path":"timesformer/models/vmps.py","file_url":"https://github.com/q1xiangchen/vmps/blob/HEAD/timesformer/models/vmps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c352461f0cf088d"}},{"code_sha256_prefix":"5ad0174be68c7fa4","entry":"reverse_rearrange_tensor","repo":"q1xiangchen/vmps","repo_kind":"official","path":"timesformer/models/vmps.py","file_url":"https://github.com/q1xiangchen/vmps/blob/HEAD/timesformer/models/vmps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ad0174be68c7fa4"}},{"code_sha256_prefix":"909f6252536bbcf0","entry":"get_padding_value","repo":"q1xiangchen/vmps","repo_kind":"official","path":"timesformer/models/conv2d_same.py","file_url":"https://github.com/q1xiangchen/vmps/blob/HEAD/timesformer/models/conv2d_same.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"909f6252536bbcf0"}},{"code_sha256_prefix":"102b75f3481035d1","entry":"resume_checkpoint","repo":"q1xiangchen/vmps","repo_kind":"official","path":"timesformer/models/helpers.py","file_url":"https://github.com/q1xiangchen/vmps/blob/HEAD/timesformer/models/helpers.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"102b75f3481035d1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}