{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/video-focalnets-spatio-temporal-focal","title":"Video-FocalNets: Spatio-Temporal Focal Modulation for Video Action Recognition","arxiv_id":"2307.06947","date":"2023-07-13","proceeding":"ICCV 2023 1","authors":["Syed Talal Wasim","Muhammad Uzair Khattak","Muzammal Naseer","Salman Khan","Mubarak Shah","Fahad Shahbaz Khan"],"abstract":"Recent video recognition models utilize Transformer models for long-range spatio-temporal context modeling. Video transformer designs are based on self-attention that can model global context at a high computational cost. In comparison, convolutional designs for videos offer an efficient alternative but lack long-range dependency modeling. Towards achieving the best of both designs, this work proposes Video-FocalNet, an effective and efficient architecture for video recognition that models both local and global contexts. Video-FocalNet is based on a spatio-temporal focal modulation architecture that reverses the interaction and aggregation steps of self-attention for better efficiency. Further, the aggregation step and the interaction step are both implemented using efficient convolution and element-wise multiplication operations that are computationally less expensive than their self-attention counterparts on video representations. We extensively explore the design space of focal modulation-based spatio-temporal context modeling and demonstrate our parallel spatial and temporal encoding design to be the optimal choice. Video-FocalNets perform favorably well against the state-of-the-art transformer-based models for video recognition on five large-scale datasets (Kinetics-400, Kinetics-600, SS-v2, Diving-48, and ActivityNet-1.3) at a lower computational cost. Our code/models are released at https://github.com/TalalWasim/Video-FocalNets.","url_abs":"https://arxiv.org/abs/2307.06947v4","url_pdf":"https://arxiv.org/pdf/2307.06947v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"video-focalnets-spatio-temporal-focal","repo_url":"https://github.com/talalwasim/video-focalnets","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"video-focalnets-spatio-temporal-focal","repo_url":"https://github.com/hayatkhan8660-maker/DVFL-Net","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"video-focalnets-spatio-temporal-focal","repo_url":"https://github.com/innat/Video-FocalNets","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"},{"task_slug":"video-recognition","task_name":"Video Recognition"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-recognition-on-diving-48","task":"Action Recognition","dataset":"Diving-48","model":"Video-FocalNet-B","rank_in_archive_order":2,"of":18,"metrics":{"Accuracy":"90.8"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2307.06947","atlas_url":"https://app.syntology.ai/?focus=2307.06947","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.06947"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hayatkhan8660-maker/DVFL-Net","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/talalwasim/video-focalnets","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/innat/Video-FocalNets","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/TalalWasim/Video-FocalNets","reach":{"status":"ok"}}],"summary":{"ran":17,"ran_honours":1,"ran_draft_wrong":1,"unverified":13},"by_repo_kind":{"official":{"samples":21,"ran":13,"repositories":1},"listed":{"samples":11,"ran":6,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":21,"samples":[{"code_sha256_prefix":"e83dd8c65a734673","entry":"PatchEmbed","repo":"TalalWasim/Video-FocalNets","repo_kind":"official","path":"classification/videofocalnet.py","file_url":"https://github.com/TalalWasim/Video-FocalNets/blob/HEAD/classification/videofocalnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e83dd8c65a734673"}},{"code_sha256_prefix":"304fe25c547d2f00","entry":"SpatioTemporalFocalModulation","repo":"TalalWasim/Video-FocalNets","repo_kind":"official","path":"classification/videofocalnet.py","file_url":"https://github.com/TalalWasim/Video-FocalNets/blob/HEAD/classification/videofocalnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"304fe25c547d2f00"}},{"code_sha256_prefix":"a58723c1a5bda855","entry":"TFDropPath","repo":"innat/Video-FocalNets","repo_kind":"listed","path":"videofocalnet/model.py","file_url":"https://github.com/innat/Video-FocalNets/blob/HEAD/videofocalnet/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a58723c1a5bda855"}},{"code_sha256_prefix":"bef3ab0dcd7067cf","entry":"TFMlp","repo":"innat/Video-FocalNets","repo_kind":"listed","path":"videofocalnet/model.py","file_url":"https://github.com/innat/Video-FocalNets/blob/HEAD/videofocalnet/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bef3ab0dcd7067cf"}},{"code_sha256_prefix":"44a2e84640d35e69","entry":"TFPatchEmbed","repo":"innat/Video-FocalNets","repo_kind":"listed","path":"videofocalnet/model.py","file_url":"https://github.com/innat/Video-FocalNets/blob/HEAD/videofocalnet/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44a2e84640d35e69"}},{"code_sha256_prefix":"bcd922fa8a50f633","entry":"TFSpatioTemporalFocalModulation","repo":"innat/Video-FocalNets","repo_kind":"listed","path":"videofocalnet/model.py","file_url":"https://github.com/innat/Video-FocalNets/blob/HEAD/videofocalnet/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bcd922fa8a50f633"}},{"code_sha256_prefix":"0b5ff5992ed0ad74","entry":"VideoFocalNet","repo":"TalalWasim/Video-FocalNets","repo_kind":"official","path":"classification/videofocalnet.py","file_url":"https://github.com/TalalWasim/Video-FocalNets/blob/HEAD/classification/videofocalnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b5ff5992ed0ad74"}},{"code_sha256_prefix":"e35a4b2e4114b1a5","entry":"build_optimizer","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"optimizer.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/optimizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e35a4b2e4114b1a5"}},{"code_sha256_prefix":"d38f3624367ac8a0","entry":"build_teacher_model","repo":"hayatkhan8660-maker/DVFL-Net","repo_kind":"listed","path":"classification/build.py","file_url":"https://github.com/hayatkhan8660-maker/DVFL-Net/blob/HEAD/classification/build.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d38f3624367ac8a0"}},{"code_sha256_prefix":"652ca38bafaf4f7d","entry":"build_transforms","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"classification/videofocalnet.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/classification/videofocalnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"652ca38bafaf4f7d"}},{"code_sha256_prefix":"4a1a30b09528ee69","entry":"build_transforms4display","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"classification/videofocalnet.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/classification/videofocalnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4a1a30b09528ee69"}},{"code_sha256_prefix":"eec1e7cba51d5e8e","entry":"get_grad_norm","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"utils.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eec1e7cba51d5e8e"}},{"code_sha256_prefix":"f8587b4b569b31e3","entry":"load_checkpoint","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"utils.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8587b4b569b31e3"}},{"code_sha256_prefix":"40e37850e70297c2","entry":"loss_fn_kd","repo":"hayatkhan8660-maker/DVFL-Net","repo_kind":"listed","path":"classification/criterion.py","file_url":"https://github.com/hayatkhan8660-maker/DVFL-Net/blob/HEAD/classification/criterion.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"40e37850e70297c2"}},{"code_sha256_prefix":"4728dc2025227bba","entry":"one_hot","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"datasets/blending.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/datasets/blending.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4728dc2025227bba"}},{"code_sha256_prefix":"b33222a09fc93bec","entry":"set_weight_decay","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"optimizer.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/optimizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b33222a09fc93bec"}},{"code_sha256_prefix":"e866e1196986f728","entry":"shear_x","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"datasets/rand_augment.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/datasets/rand_augment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e866e1196986f728"}},{"code_sha256_prefix":"7b2503c99c9447fb","entry":"shear_y","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"datasets/rand_augment.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/datasets/rand_augment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b2503c99c9447fb"}},{"code_sha256_prefix":"323759f906fbb394","entry":"translate_x_rel","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"datasets/rand_augment.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/datasets/rand_augment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"323759f906fbb394"}},{"code_sha256_prefix":"5da5482bcfc017c2","entry":"BasicLayer","repo":"TalalWasim/Video-FocalNets","repo_kind":"official","path":"classification/videofocalnet.py","file_url":"https://github.com/TalalWasim/Video-FocalNets/blob/HEAD/classification/videofocalnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5da5482bcfc017c2"}},{"code_sha256_prefix":"456892b8f3ad701f","entry":"TFBasicLayer","repo":"innat/Video-FocalNets","repo_kind":"listed","path":"videofocalnet/model.py","file_url":"https://github.com/innat/Video-FocalNets/blob/HEAD/videofocalnet/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"456892b8f3ad701f"}},{"code_sha256_prefix":"80d19d9d058074c1","entry":"TFVideoFocalNet","repo":"innat/Video-FocalNets","repo_kind":"listed","path":"videofocalnet/model.py","file_url":"https://github.com/innat/Video-FocalNets/blob/HEAD/videofocalnet/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"80d19d9d058074c1"}},{"code_sha256_prefix":"7359d74430b2ebd3","entry":"TFVideoFocalNetBlock","repo":"innat/Video-FocalNets","repo_kind":"listed","path":"videofocalnet/model.py","file_url":"https://github.com/innat/Video-FocalNets/blob/HEAD/videofocalnet/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7359d74430b2ebd3"}},{"code_sha256_prefix":"719831d1fc53a0b9","entry":"VideoFocalNetBlock","repo":"TalalWasim/Video-FocalNets","repo_kind":"official","path":"classification/videofocalnet.py","file_url":"https://github.com/TalalWasim/Video-FocalNets/blob/HEAD/classification/videofocalnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"719831d1fc53a0b9"}},{"code_sha256_prefix":"0c32459bef92ff8c","entry":"auto_resume_helper","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"utils.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c32459bef92ff8c"}},{"code_sha256_prefix":"bdac4358ea39d416","entry":"build_model","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"classification/build.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/classification/build.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bdac4358ea39d416"}},{"code_sha256_prefix":"07e8b4b487c0abb6","entry":"build_scheduler","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"lr_scheduler.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/lr_scheduler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"07e8b4b487c0abb6"}},{"code_sha256_prefix":"b0a5beb34716d5a6","entry":"check_keywords_in_name","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"optimizer.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/optimizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b0a5beb34716d5a6"}},{"code_sha256_prefix":"80450600f7f09b0a","entry":"create_logger","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"logger.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80450600f7f09b0a"}},{"code_sha256_prefix":"028190203e877ad1","entry":"find_package_location","repo":"hayatkhan8660-maker/DVFL-Net","repo_kind":"listed","path":"get_local_rank.py","file_url":"https://github.com/hayatkhan8660-maker/DVFL-Net/blob/HEAD/get_local_rank.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"028190203e877ad1"}},{"code_sha256_prefix":"eb702a1f77e3c7bd","entry":"get_config","repo":"talalwasim/video-focalnets","repo_kind":"official","path":"config.py","file_url":"https://github.com/talalwasim/video-focalnets/blob/HEAD/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eb702a1f77e3c7bd"}},{"code_sha256_prefix":"0493984d01a4e886","entry":"get_config","repo":"hayatkhan8660-maker/DVFL-Net","repo_kind":"listed","path":"config.py","file_url":"https://github.com/hayatkhan8660-maker/DVFL-Net/blob/HEAD/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0493984d01a4e886"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}