{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/long-movie-clip-classification-with-state","title":"Long Movie Clip Classification with State-Space Video Models","arxiv_id":"2204.01692","date":"2022-04-04","proceeding":null,"authors":["Md Mohaiminul Islam","Gedas Bertasius"],"abstract":"Most modern video recognition models are designed to operate on short video clips (e.g., 5-10s in length). Thus, it is challenging to apply such models to long movie understanding tasks, which typically require sophisticated long-range temporal reasoning. The recently introduced video transformers partially address this issue by using long-range temporal self-attention. However, due to the quadratic cost of self-attention, such models are often costly and impractical to use. Instead, we propose ViS4mer, an efficient long-range video model that combines the strengths of self-attention and the recently introduced structured state-space sequence (S4) layer. Our model uses a standard Transformer encoder for short-range spatiotemporal feature extraction, and a multi-scale temporal S4 decoder for subsequent long-range temporal reasoning. By progressively reducing the spatiotemporal feature resolution and channel dimension at each decoder layer, ViS4mer learns complex long-range spatiotemporal dependencies in a video. Furthermore, ViS4mer is $2.63\\times$ faster and requires $8\\times$ less GPU memory than the corresponding pure self-attention-based model. Additionally, ViS4mer achieves state-of-the-art results in $6$ out of $9$ long-form movie video classification tasks on the Long Video Understanding (LVU) benchmark. Furthermore, we show that our approach successfully generalizes to other domains, achieving competitive results on the Breakfast and the COIN procedural activity datasets. The code is publicly available at: https://github.com/md-mohaiminul/ViS4mer.","url_abs":"https://arxiv.org/abs/2204.01692v3","url_pdf":"https://arxiv.org/pdf/2204.01692v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"long-movie-clip-classification-with-state","repo_url":"https://github.com/md-mohaiminul/ViS4mer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"video-classification","task_name":"Video Classification"},{"task_slug":"video-recognition","task_name":"Video Recognition"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-classification-on-breakfast","task":"Video Classification","dataset":"Breakfast","model":"ViS4mer","rank_in_archive_order":6,"of":9,"metrics":{"Accuracy (%)":"88.2"},"uses_additional_data":false},{"leaderboard":"/sota/video-classification-on-coin-1","task":"Video Classification","dataset":"COIN","model":"ViS4mer","rank_in_archive_order":6,"of":7,"metrics":{"Accuracy (%)":"88.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2204.01692","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.01692"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/md-mohaiminul/ViS4mer","reach":null}],"summary":{"ran_draft_wrong":7,"ran":2,"ran_fixture":1,"ran_honours":1,"unverified":7},"by_repo_kind":{"official":{"samples":16,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"89b10a8b26bbb7ed","entry":"Activation","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"89b10a8b26bbb7ed"}},{"code_sha256_prefix":"8538f8c63d35c959","entry":"LinearActivation","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8538f8c63d35c959"}},{"code_sha256_prefix":"39159b1078607edf","entry":"SSKernelSlow","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39159b1078607edf"}},{"code_sha256_prefix":"71048363ae476934","entry":"TransposedLinear","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71048363ae476934"}},{"code_sha256_prefix":"49d9214ec79a8edb","entry":"get_initializer","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"49d9214ec79a8edb"}},{"code_sha256_prefix":"61139ec62260b411","entry":"get_logger","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61139ec62260b411"}},{"code_sha256_prefix":"030ade7f4f951898","entry":"krylov","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"030ade7f4f951898"}},{"code_sha256_prefix":"52f5a73f3505ecee","entry":"nplr","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52f5a73f3505ecee"}},{"code_sha256_prefix":"fae8dcc7964cac77","entry":"power","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fae8dcc7964cac77"}},{"code_sha256_prefix":"3dce80b93036d192","entry":"rank_correction","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3dce80b93036d192"}},{"code_sha256_prefix":"87ea80b70e1e70b9","entry":"transition","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87ea80b70e1e70b9"}},{"code_sha256_prefix":"717d9447a001c75e","entry":"HippoSSKernel","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"717d9447a001c75e"}},{"code_sha256_prefix":"8076f48bc17f0bc5","entry":"OptimModule","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8076f48bc17f0bc5"}},{"code_sha256_prefix":"0bbcf5a8797deb01","entry":"S4","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0bbcf5a8797deb01"}},{"code_sha256_prefix":"a0cd8021dee737f7","entry":"SSKernelNPLR","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0cd8021dee737f7"}},{"code_sha256_prefix":"b7f3ea1de74c55c3","entry":"ViS4mer","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b7f3ea1de74c55c3"}},{"code_sha256_prefix":"0c3303405c8151e9","entry":"_broadcast_dims","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c3303405c8151e9"}},{"code_sha256_prefix":"fba66bbdebdb89e4","entry":"cauchy_conj","repo":"md-mohaiminul/ViS4mer","repo_kind":"official","path":"models.py","file_url":"https://github.com/md-mohaiminul/ViS4mer/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fba66bbdebdb89e4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}