{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/embracing-consistency-a-one-stage-approach","title":"Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Grounding","arxiv_id":"2209.13306","date":"2022-09-27","proceeding":null,"authors":["Yang Jin","Yongzhi Li","Zehuan Yuan","Yadong Mu"],"abstract":"Spatio-Temporal video grounding (STVG) focuses on retrieving the spatio-temporal tube of a specific object depicted by a free-form textual expression. Existing approaches mainly treat this complicated task as a parallel frame-grounding problem and thus suffer from two types of inconsistency drawbacks: feature alignment inconsistency and prediction inconsistency. In this paper, we present an end-to-end one-stage framework, termed Spatio-Temporal Consistency-Aware Transformer (STCAT), to alleviate these issues. Specially, we introduce a novel multi-modal template as the global objective to address this task, which explicitly constricts the grounding region and associates the predictions among all video frames. Moreover, to generate the above template under sufficient video-textual perception, an encoder-decoder architecture is proposed for effective global context modeling. Thanks to these critical designs, STCAT enjoys more consistent cross-modal feature alignment and tube prediction without reliance on any pre-trained object detectors. Extensive experiments show that our method outperforms previous state-of-the-arts with clear margins on two challenging video benchmarks (VidSTG and HC-STVG), illustrating the superiority of the proposed framework to better understanding the association between vision and natural language. Code is publicly available at https://github.com/jy0205/STCAT.","url_abs":"https://arxiv.org/abs/2209.13306v2","url_pdf":"https://arxiv.org/pdf/2209.13306v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"embracing-consistency-a-one-stage-approach","repo_url":"https://github.com/jy0205/stcat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"spatio-temporal-video-grounding","task_name":"Spatio-Temporal Video Grounding"},{"task_slug":"video-grounding","task_name":"Video Grounding"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2209.13306","atlas_url":"https://app.syntology.ai/?focus=2209.13306","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.13306"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jy0205/STCAT","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jy0205/stcat","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":10,"ran_draft_wrong":1,"unverified":7},"by_repo_kind":{"official":{"samples":18,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"61a5ef5ec49e4ea8","entry":"BackboneBase","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"61a5ef5ec49e4ea8"}},{"code_sha256_prefix":"2c8652f8cdc39a8a","entry":"FeatureResizer","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2c8652f8cdc39a8a"}},{"code_sha256_prefix":"6a44d56c318a9f2b","entry":"GroupNorm32","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6a44d56c318a9f2b"}},{"code_sha256_prefix":"d79f14a1bc1a6186","entry":"Joiner","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d79f14a1bc1a6186"}},{"code_sha256_prefix":"c7728510fb5fa2dd","entry":"MLP","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7728510fb5fa2dd"}},{"code_sha256_prefix":"60a2fb58c9949c1d","entry":"NestedTensor","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"60a2fb58c9949c1d"}},{"code_sha256_prefix":"fa04a0c6d2415dac","entry":"PositionEmbeddingLearned","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa04a0c6d2415dac"}},{"code_sha256_prefix":"f52bb0cf00c7d60c","entry":"PositionEmbeddingSine","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f52bb0cf00c7d60c"}},{"code_sha256_prefix":"7b9116f0334dab6b","entry":"PositionEmbeddingSineHW","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b9116f0334dab6b"}},{"code_sha256_prefix":"0edb5fe68c13683e","entry":"Roberta","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0edb5fe68c13683e"}},{"code_sha256_prefix":"ebf7ea1f0aba8315","entry":"build_position_encoding","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ebf7ea1f0aba8315"}},{"code_sha256_prefix":"8687762072eb9277","entry":"Backbone","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8687762072eb9277"}},{"code_sha256_prefix":"ad5c02224767d3c7","entry":"GroupNormBackbone","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ad5c02224767d3c7"}},{"code_sha256_prefix":"ffc2b8b986f923af","entry":"STCATNet","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ffc2b8b986f923af"}},{"code_sha256_prefix":"554210938d06c170","entry":"build_decoder","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"554210938d06c170"}},{"code_sha256_prefix":"0a690726c1cb50e7","entry":"build_encoder","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0a690726c1cb50e7"}},{"code_sha256_prefix":"10aac717b87ceafd","entry":"build_text_encoder","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10aac717b87ceafd"}},{"code_sha256_prefix":"f85b47f8c900ff25","entry":"build_vis_encoder","repo":"jy0205/stcat","repo_kind":"official","path":"models/pipeline.py","file_url":"https://github.com/jy0205/stcat/blob/HEAD/models/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f85b47f8c900ff25"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}