{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-online-sign-language-recognition-and","title":"Towards Online Continuous Sign Language Recognition and Translation","arxiv_id":"2401.05336","date":"2024-01-10","proceeding":null,"authors":["Ronglai Zuo","Fangyun Wei","Brian Mak"],"abstract":"Research on continuous sign language recognition (CSLR) is essential to bridge the communication gap between deaf and hearing individuals. Numerous previous studies have trained their models using the connectionist temporal classification (CTC) loss. During inference, these CTC-based models generally require the entire sign video as input to make predictions, a process known as offline recognition, which suffers from high latency and substantial memory usage. In this work, we take the first step towards online CSLR. Our approach consists of three phases: 1) developing a sign dictionary; 2) training an isolated sign language recognition model on the dictionary; and 3) employing a sliding window approach on the input sign sequence, feeding each sign clip to the optimized model for online recognition. Additionally, our online recognition model can be extended to support online translation by integrating a gloss-to-text network and can enhance the performance of any offline model. With these extensions, our online approach achieves new state-of-the-art performance on three popular benchmarks across various task settings. Code and models are available at https://github.com/FangyunWei/SLRT.","url_abs":"https://arxiv.org/abs/2401.05336v2","url_pdf":"https://arxiv.org/pdf/2401.05336v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-online-sign-language-recognition-and","repo_url":"https://github.com/FangyunWei/SLRT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"sign-language-recognition","task_name":"Sign Language Recognition"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.05336","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.05336"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/FangyunWei/SLRT","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"1503e2da6ffabf2d","entry":"pose_inference","repo":"FangyunWei/SLRT","repo_kind":"official","path":"NLA-SLR/gen_pose.py","file_url":"https://github.com/FangyunWei/SLRT/blob/HEAD/NLA-SLR/gen_pose.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1503e2da6ffabf2d"}},{"code_sha256_prefix":"bdd38b897016bb3e","entry":"detection_inference","repo":"FangyunWei/SLRT","repo_kind":"official","path":"NLA-SLR/gen_pose.py","file_url":"https://github.com/FangyunWei/SLRT/blob/HEAD/NLA-SLR/gen_pose.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bdd38b897016bb3e"}},{"code_sha256_prefix":"71c76eb4ecffbe0f","entry":"save_model","repo":"FangyunWei/SLRT","repo_kind":"official","path":"TwoStreamNetwork/training.py","file_url":"https://github.com/FangyunWei/SLRT/blob/HEAD/TwoStreamNetwork/training.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71c76eb4ecffbe0f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}