{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/masa-motion-aware-masked-autoencoder-with","title":"MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition","arxiv_id":"2405.20666","date":"2024-05-31","proceeding":null,"authors":["Weichao Zhao","Hezhen Hu","Wengang Zhou","Yunyao Mao","Min Wang","Houqiang Li"],"abstract":"Sign language recognition (SLR) has long been plagued by insufficient model representation capabilities. Although current pre-training approaches have alleviated this dilemma to some extent and yielded promising performance by employing various pretext tasks on sign pose data, these methods still suffer from two primary limitations: 1) Explicit motion information is usually disregarded in previous pretext tasks, leading to partial information loss and limited representation capability. 2) Previous methods focus on the local context of a sign pose sequence, without incorporating the guidance of the global meaning of lexical signs. To this end, we propose a Motion-Aware masked autoencoder with Semantic Alignment (MASA) that integrates rich motion cues and global semantic information in a self-supervised learning paradigm for SLR. Our framework contains two crucial components, i.e., a motion-aware masked autoencoder (MA) and a momentum semantic alignment module (SA). Specifically, in MA, we introduce an autoencoder architecture with a motion-aware masked strategy to reconstruct motion residuals of masked frames, thereby explicitly exploring dynamic motion cues among sign pose sequences. Moreover, in SA, we embed our framework with global semantic awareness by aligning the embeddings of different augmented samples from the input sequence in the shared latent space. In this way, our framework can simultaneously learn local motion cues and global semantic features for comprehensive sign language representation. Furthermore, we conduct extensive experiments to validate the effectiveness of our method, achieving new state-of-the-art performance on four public benchmarks.","url_abs":"https://arxiv.org/abs/2405.20666v1","url_pdf":"https://arxiv.org/pdf/2405.20666v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"masa-motion-aware-masked-autoencoder-with","repo_url":"https://github.com/sakura2233565548/masa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"sign-language-recognition","task_name":"Sign Language Recognition"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"slr","method_name":"SLR"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.20666","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.20666"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sakura2233565548/masa","reach":null}],"summary":{"ran":7,"ran_fixture":1,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":12,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"0bd39692ab2abbaa","entry":"Block","repo":"sakura2233565548/masa","repo_kind":"official","path":"moco/GCN_Transformer_mask.py","file_url":"https://github.com/sakura2233565548/masa/blob/HEAD/moco/GCN_Transformer_mask.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0bd39692ab2abbaa"}},{"code_sha256_prefix":"04a157889b43b407","entry":"LayerNorm","repo":"sakura2233565548/masa","repo_kind":"official","path":"moco/GCN_Transformer_mask.py","file_url":"https://github.com/sakura2233565548/masa/blob/HEAD/moco/GCN_Transformer_mask.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04a157889b43b407"}},{"code_sha256_prefix":"3a946d8e45ee55b7","entry":"MultiHeadedSelfAttention","repo":"sakura2233565548/masa","repo_kind":"official","path":"moco/GCN_Transformer_mask.py","file_url":"https://github.com/sakura2233565548/masa/blob/HEAD/moco/GCN_Transformer_mask.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3a946d8e45ee55b7"}},{"code_sha256_prefix":"b6decf507ce36bfc","entry":"PositionEncoding","repo":"sakura2233565548/masa","repo_kind":"official","path":"moco/GCN_Transformer_mask.py","file_url":"https://github.com/sakura2233565548/masa/blob/HEAD/moco/GCN_Transformer_mask.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b6decf507ce36bfc"}},{"code_sha256_prefix":"fd6c5d5dbab2c8c5","entry":"PositionWiseFeedForward","repo":"sakura2233565548/masa","repo_kind":"official","path":"moco/GCN_Transformer_mask.py","file_url":"https://github.com/sakura2233565548/masa/blob/HEAD/moco/GCN_Transformer_mask.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd6c5d5dbab2c8c5"}},{"code_sha256_prefix":"6c42fb2fc8195260","entry":"ProjectHead","repo":"sakura2233565548/masa","repo_kind":"official","path":"moco/GCN_Transformer_mask.py","file_url":"https://github.com/sakura2233565548/masa/blob/HEAD/moco/GCN_Transformer_mask.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6c42fb2fc8195260"}},{"code_sha256_prefix":"6b3c921acf42e83c","entry":"Transformer","repo":"sakura2233565548/masa","repo_kind":"official","path":"moco/GCN_Transformer_mask.py","file_url":"https://github.com/sakura2233565548/masa/blob/HEAD/moco/GCN_Transformer_mask.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b3c921acf42e83c"}},{"code_sha256_prefix":"0d7bcec3b6d68575","entry":"accuracy","repo":"sakura2233565548/masa","repo_kind":"official","path":"pretrain.py","file_url":"https://github.com/sakura2233565548/masa/blob/HEAD/pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0d7bcec3b6d68575"}},{"code_sha256_prefix":"6f324d970195cc1d","entry":"gelu","repo":"sakura2233565548/masa","repo_kind":"official","path":"moco/GCN_Transformer_mask.py","file_url":"https://github.com/sakura2233565548/masa/blob/HEAD/moco/GCN_Transformer_mask.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f324d970195cc1d"}},{"code_sha256_prefix":"03f61837429aef9a","entry":"Embed","repo":"sakura2233565548/masa","repo_kind":"official","path":"moco/GCN_Transformer_mask.py","file_url":"https://github.com/sakura2233565548/masa/blob/HEAD/moco/GCN_Transformer_mask.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"03f61837429aef9a"}},{"code_sha256_prefix":"22ed422281aca870","entry":"Model","repo":"sakura2233565548/masa","repo_kind":"official","path":"moco/GCN_Transformer_mask.py","file_url":"https://github.com/sakura2233565548/masa/blob/HEAD/moco/GCN_Transformer_mask.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"22ed422281aca870"}},{"code_sha256_prefix":"e80cb369929174bb","entry":"train","repo":"sakura2233565548/masa","repo_kind":"official","path":"pretrain.py","file_url":"https://github.com/sakura2233565548/masa/blob/HEAD/pretrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e80cb369929174bb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}