{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/video-mamba-suite-state-space-model-as-a","title":"Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding","arxiv_id":"2403.09626","date":"2024-03-14","proceeding":null,"authors":["Guo Chen","Yifei HUANG","Jilan Xu","Baoqi Pei","Zhe Chen","Zhiqi Li","Jiahao Wang","Kunchang Li","Tong Lu","LiMin Wang"],"abstract":"Understanding videos is one of the fundamental directions in computer vision research, with extensive efforts dedicated to exploring various architectures such as RNN, 3D CNN, and Transformers. The newly proposed architecture of state space model, e.g., Mamba, shows promising traits to extend its success in long sequence modeling to video modeling. To assess whether Mamba can be a viable alternative to Transformers in the video understanding domain, in this work, we conduct a comprehensive set of studies, probing different roles Mamba can play in modeling videos, while investigating diverse tasks where Mamba could exhibit superiority. We categorize Mamba into four roles for modeling videos, deriving a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12 video understanding tasks. Our extensive experiments reveal the strong potential of Mamba on both video-only and video-language tasks while showing promising efficiency-performance trade-offs. We hope this work could provide valuable data points and insights for future research on video understanding. Code is public: https://github.com/OpenGVLab/video-mamba-suite.","url_abs":"https://arxiv.org/abs/2403.09626v1","url_pdf":"https://arxiv.org/pdf/2403.09626v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"video-mamba-suite-state-space-model-as-a","repo_url":"https://github.com/opengvlab/video-mamba-suite","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"mamba","task_name":"Mamba"},{"task_slug":"moment-retrieval","task_name":"Moment Retrieval"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"3d-cnn","method_name":"3D CNN"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/moment-retrieval-on-charades-sta","task":"Moment Retrieval","dataset":"Charades-STA","model":"video-mamba-suite","rank_in_archive_order":19,"of":25,"metrics":{"R@1 IoU=0.5":"57.18","R@1 IoU=0.7":"36.05"},"uses_additional_data":false},{"leaderboard":"/sota/moment-retrieval-on-qvhighlights","task":"Moment Retrieval","dataset":"QVHighlights","model":"video-mamba-suite","rank_in_archive_order":14,"of":32,"metrics":{"R@1 IoU=0.5":"66.65","R@1 IoU=0.7":"52.19","mAP":"45.18","mAP@0.5":"64.37","mAP@0.75":"46.68"},"uses_additional_data":false},{"leaderboard":"/sota/temporal-action-localization-on-activitynet","task":"Temporal Action Localization","dataset":"ActivityNet-1.3","model":"ActionMamba (InternVideo2-6B)","rank_in_archive_order":2,"of":33,"metrics":{"mAP":"42.02","mAP IOU@0.5":"62.43","mAP IOU@0.75":"43.49","mAP IOU@0.95":"10.23"},"uses_additional_data":false},{"leaderboard":"/sota/temporal-action-localization-on-fineaction","task":"Temporal Action Localization","dataset":"FineAction","model":"ActionMamba(InternVideo2-6B)","rank_in_archive_order":2,"of":9,"metrics":{"mAP":"29.04","mAP IOU@0.5":"45.44","mAP IOU@0.75":"28.82","mAP IOU@0.95":"6.79"},"uses_additional_data":false},{"leaderboard":"/sota/temporal-action-localization-on-hacs","task":"Temporal Action Localization","dataset":"HACS","model":"ActionMamba(InternVideo2-6B)","rank_in_archive_order":2,"of":12,"metrics":{"Average-mAP":"44.56","mAP@0.5":"64.02","mAP@0.75":"45.71","mAP@0.95":"13.34"},"uses_additional_data":false},{"leaderboard":"/sota/temporal-action-localization-on-thumos14","task":"Temporal Action Localization","dataset":"THUMOS’14","model":"ActionMamba(InternVideo2-6B)","rank_in_archive_order":3,"of":42,"metrics":{"Avg mAP (0.3:0.7)":"72.72","mAP IOU@0.3":"86.89","mAP IOU@0.4":"83.09","mAP IOU@0.5":"76.90","mAP IOU@0.6":"65.91","mAP IOU@0.7":"50.82"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2403.09626","atlas_url":"https://app.syntology.ai/?focus=2403.09626","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.09626"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opengvlab/video-mamba-suite","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"unverified":5},"by_repo_kind":{"official":{"samples":13,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bc39b8ed5fefa7eb","entry":"crop_clip","repo":"opengvlab/video-mamba-suite","repo_kind":"official","path":"video-mamba-suite/action-recognition/functional.py","file_url":"https://github.com/opengvlab/video-mamba-suite/blob/HEAD/video-mamba-suite/action-recognition/functional.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc39b8ed5fefa7eb"}},{"code_sha256_prefix":"8467c56ed6949a06","entry":"exponential_descrease","repo":"opengvlab/video-mamba-suite","repo_kind":"official","path":"video-mamba-suite/temporal-action-segmentation/model.py","file_url":"https://github.com/opengvlab/video-mamba-suite/blob/HEAD/video-mamba-suite/temporal-action-segmentation/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8467c56ed6949a06"}},{"code_sha256_prefix":"6aef977b44c0feaf","entry":"get_loss_scale_for_deepspeed","repo":"opengvlab/video-mamba-suite","repo_kind":"official","path":"video-mamba-suite/action-recognition/engines/engine_for_finetuning.py","file_url":"https://github.com/opengvlab/video-mamba-suite/blob/HEAD/video-mamba-suite/action-recognition/engines/engine_for_finetuning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6aef977b44c0feaf"}},{"code_sha256_prefix":"2b53e457d5632b4e","entry":"get_resize_sizes","repo":"opengvlab/video-mamba-suite","repo_kind":"official","path":"video-mamba-suite/action-recognition/functional.py","file_url":"https://github.com/opengvlab/video-mamba-suite/blob/HEAD/video-mamba-suite/action-recognition/functional.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b53e457d5632b4e"}},{"code_sha256_prefix":"f9c6fc8c1ddfbfac","entry":"get_sinusoid_encoding_table","repo":"opengvlab/video-mamba-suite","repo_kind":"official","path":"video-mamba-suite/action-recognition/models/modeling_finetune.py","file_url":"https://github.com/opengvlab/video-mamba-suite/blob/HEAD/video-mamba-suite/action-recognition/models/modeling_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9c6fc8c1ddfbfac"}},{"code_sha256_prefix":"981936c73d3cdd12","entry":"resize_clip","repo":"opengvlab/video-mamba-suite","repo_kind":"official","path":"video-mamba-suite/action-recognition/functional.py","file_url":"https://github.com/opengvlab/video-mamba-suite/blob/HEAD/video-mamba-suite/action-recognition/functional.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"981936c73d3cdd12"}},{"code_sha256_prefix":"f188ca6f84532f7f","entry":"spatial_sampling","repo":"opengvlab/video-mamba-suite","repo_kind":"official","path":"video-mamba-suite/action-recognition/datasets/kinetics.py","file_url":"https://github.com/opengvlab/video-mamba-suite/blob/HEAD/video-mamba-suite/action-recognition/datasets/kinetics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f188ca6f84532f7f"}},{"code_sha256_prefix":"fb98703650f02bf6","entry":"train_class_batch","repo":"opengvlab/video-mamba-suite","repo_kind":"official","path":"video-mamba-suite/action-recognition/engines/engine_for_finetuning_regression.py","file_url":"https://github.com/opengvlab/video-mamba-suite/blob/HEAD/video-mamba-suite/action-recognition/engines/engine_for_finetuning_regression.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb98703650f02bf6"}},{"code_sha256_prefix":"e72f71197ef8b97e","entry":"clip_b16","repo":"opengvlab/video-mamba-suite","repo_kind":"official","path":"video-mamba-suite/action-recognition/models/clip.py","file_url":"https://github.com/opengvlab/video-mamba-suite/blob/HEAD/video-mamba-suite/action-recognition/models/clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e72f71197ef8b97e"}},{"code_sha256_prefix":"055a054d676fb93c","entry":"clip_l14","repo":"opengvlab/video-mamba-suite","repo_kind":"official","path":"video-mamba-suite/action-recognition/models/clip.py","file_url":"https://github.com/opengvlab/video-mamba-suite/blob/HEAD/video-mamba-suite/action-recognition/models/clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"055a054d676fb93c"}},{"code_sha256_prefix":"bdec7f112532de68","entry":"inflate_weight","repo":"opengvlab/video-mamba-suite","repo_kind":"official","path":"video-mamba-suite/action-recognition/models/clip.py","file_url":"https://github.com/opengvlab/video-mamba-suite/blob/HEAD/video-mamba-suite/action-recognition/models/clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bdec7f112532de68"}},{"code_sha256_prefix":"14b3b13b2e7ee844","entry":"tensor_normalize","repo":"opengvlab/video-mamba-suite","repo_kind":"official","path":"video-mamba-suite/action-recognition/datasets/kinetics.py","file_url":"https://github.com/opengvlab/video-mamba-suite/blob/HEAD/video-mamba-suite/action-recognition/datasets/kinetics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14b3b13b2e7ee844"}},{"code_sha256_prefix":"c7976ea27edc377a","entry":"train_class_batch","repo":"opengvlab/video-mamba-suite","repo_kind":"official","path":"video-mamba-suite/action-recognition/engines/engine_for_finetuning.py","file_url":"https://github.com/opengvlab/video-mamba-suite/blob/HEAD/video-mamba-suite/action-recognition/engines/engine_for_finetuning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7976ea27edc377a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}