{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cvt-slr-contrastive-visual-textual","title":"CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignment","arxiv_id":"2303.05725","date":"2023-03-10","proceeding":"CVPR 2023 1","authors":["Jiangbin Zheng","Yile Wang","Cheng Tan","Siyuan Li","Ge Wang","Jun Xia","Yidong Chen","Stan Z. Li"],"abstract":"Sign language recognition (SLR) is a weakly supervised task that annotates sign videos as textual glosses. Recent studies show that insufficient training caused by the lack of large-scale available sign datasets becomes the main bottleneck for SLR. Most SLR works thereby adopt pretrained visual modules and develop two mainstream solutions. The multi-stream architectures extend multi-cue visual features, yielding the current SOTA performances but requiring complex designs and might introduce potential noise. Alternatively, the advanced single-cue SLR frameworks using explicit cross-modal alignment between visual and textual modalities are simple and effective, potentially competitive with the multi-cue framework. In this work, we propose a novel contrastive visual-textual transformation for SLR, CVT-SLR, to fully explore the pretrained knowledge of both the visual and language modalities. Based on the single-cue cross-modal alignment framework, we propose a variational autoencoder (VAE) for pretrained contextual knowledge while introducing the complete pretrained language module. The VAE implicitly aligns visual and textual modalities while benefiting from pretrained contextual knowledge as the traditional contextual module. Meanwhile, a contrastive cross-modal alignment algorithm is designed to explicitly enhance the consistency constraints. Extensive experiments on public datasets (PHOENIX-2014 and PHOENIX-2014T) demonstrate that our proposed CVT-SLR consistently outperforms existing single-cue methods and even outperforms SOTA multi-cue methods.","url_abs":"https://arxiv.org/abs/2303.05725v4","url_pdf":"https://arxiv.org/pdf/2303.05725v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cvt-slr-contrastive-visual-textual","repo_url":"https://github.com/binbinjiang/cvt-slr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"sign-language-recognition","task_name":"Sign Language Recognition"},{"task_slug":"cross-modal-alignment","task_name":"cross-modal alignment"}],"methods":[{"method_slug":"slr","method_name":"SLR"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2303.05725","atlas_url":"https://app.syntology.ai/?focus=2303.05725","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.05725"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/binbinjiang/CVT-SLR","reach":null}],"summary":{"ran":4,"unverified":4},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"756e910b89bd8197","entry":"MultiHeadedAttention","repo":"binbinjiang/CVT-SLR","repo_kind":"official","path":"cvtslr_model.py","file_url":"https://github.com/binbinjiang/CVT-SLR/blob/HEAD/cvtslr_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"756e910b89bd8197"}},{"code_sha256_prefix":"fd20d6f53d54aa88","entry":"PositionalEmbedding","repo":"binbinjiang/CVT-SLR","repo_kind":"official","path":"cvtslr_model.py","file_url":"https://github.com/binbinjiang/CVT-SLR/blob/HEAD/cvtslr_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd20d6f53d54aa88"}},{"code_sha256_prefix":"bef5f1e882f94aa7","entry":"PositionwiseFeedForward","repo":"binbinjiang/CVT-SLR","repo_kind":"official","path":"cvtslr_model.py","file_url":"https://github.com/binbinjiang/CVT-SLR/blob/HEAD/cvtslr_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bef5f1e882f94aa7"}},{"code_sha256_prefix":"f3ab8e695d75d895","entry":"TransformerEncoderLayer","repo":"binbinjiang/CVT-SLR","repo_kind":"official","path":"cvtslr_model.py","file_url":"https://github.com/binbinjiang/CVT-SLR/blob/HEAD/cvtslr_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3ab8e695d75d895"}},{"code_sha256_prefix":"79aab4196d7e7c9b","entry":"BiLSTMLayer","repo":"binbinjiang/CVT-SLR","repo_kind":"official","path":"cvtslr_model.py","file_url":"https://github.com/binbinjiang/CVT-SLR/blob/HEAD/cvtslr_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"79aab4196d7e7c9b"}},{"code_sha256_prefix":"062777fd94625d8f","entry":"CVTSLRModel","repo":"binbinjiang/CVT-SLR","repo_kind":"official","path":"cvtslr_model.py","file_url":"https://github.com/binbinjiang/CVT-SLR/blob/HEAD/cvtslr_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"062777fd94625d8f"}},{"code_sha256_prefix":"081f4d7ce4351fad","entry":"Encoder","repo":"binbinjiang/CVT-SLR","repo_kind":"official","path":"cvtslr_model.py","file_url":"https://github.com/binbinjiang/CVT-SLR/blob/HEAD/cvtslr_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"081f4d7ce4351fad"}},{"code_sha256_prefix":"879f27bbe64df7e9","entry":"SelfAttentionAdapter","repo":"binbinjiang/CVT-SLR","repo_kind":"official","path":"cvtslr_model.py","file_url":"https://github.com/binbinjiang/CVT-SLR/blob/HEAD/cvtslr_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"879f27bbe64df7e9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}