{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/joint-ctc-attention-based-end-to-end-speech","title":"Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning","arxiv_id":"1609.06773","date":"2016-09-21","proceeding":null,"authors":["Suyoun Kim","Takaaki Hori","Shinji Watanabe"],"abstract":"Recently, there has been an increasing interest in end-to-end speech\nrecognition that directly transcribes speech to text without any predefined\nalignments. One approach is the attention-based encoder-decoder framework that\nlearns a mapping between variable-length input and output sequences in one step\nusing a purely data-driven method. The attention model has often been shown to\nimprove the performance over another end-to-end approach, the Connectionist\nTemporal Classification (CTC), mainly because it explicitly uses the history of\nthe target character without any conditional independence assumptions. However,\nwe observed that the performance of the attention has shown poor results in\nnoisy condition and is hard to learn in the initial training stage with long\ninput sequences. This is because the attention model is too flexible to predict\nproper alignments in such cases due to the lack of left-to-right constraints as\nused in CTC. This paper presents a novel method for end-to-end speech\nrecognition to improve robustness and achieve fast convergence by using a joint\nCTC-attention model within the multi-task learning framework, thereby\nmitigating the alignment issue. An experiment on the WSJ and CHiME-4 tasks\ndemonstrates its advantages over both the CTC and attention-based\nencoder-decoder baselines, showing 5.4-14.6% relative improvements in Character\nError Rate (CER).","url_abs":"http://arxiv.org/abs/1609.06773v2","url_pdf":"http://arxiv.org/pdf/1609.06773v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"joint-ctc-attention-based-end-to-end-speech","repo_url":"https://github.com/Alexander-H-Liu/End-to-end-ASR-Pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"joint-ctc-attention-based-end-to-end-speech","repo_url":"https://github.com/aaaceo890/Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"joint-ctc-attention-based-end-to-end-speech","repo_url":"https://github.com/mnm-rnd/elsa-voice-asr","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"joint-ctc-attention-based-end-to-end-speech","repo_url":"https://github.com/neil-zeng/asr","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"joint-ctc-attention-based-end-to-end-speech","repo_url":"https://github.com/s3prl/End-to-end-ASR-Pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"joint-ctc-attention-based-end-to-end-speech","repo_url":"https://github.com/sooftware/OpenSpeech","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"joint-ctc-attention-based-end-to-end-speech","repo_url":"https://github.com/PaddlePaddle/PaddleSpeech","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"joint-ctc-attention-based-end-to-end-speech","repo_url":"https://github.com/mindspore-courses/heads-on-mindspore/tree/main/6-speech_recognition","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"speech-to-text","task_name":"Speech-to-Text"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1609.06773","atlas_url":"https://app.syntology.ai/?focus=1609.06773","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1609.06773"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PaddlePaddle/PaddleSpeech","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mnm-rnd/elsa-voice-asr","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sooftware/OpenSpeech","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aaaceo890/Attention","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/s3prl/End-to-end-ASR-Pytorch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Alexander-H-Liu/End-to-end-ASR-Pytorch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mindspore-courses/heads-on-mindspore/tree/main/6-speech_recognition","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/neil-zeng/asr","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":10},"by_repo_kind":{"listed":{"samples":10,"ran":0,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"340cb8273953e493","entry":"collect_audio_batch","repo":"Alexander-H-Liu/End-to-end-ASR-Pytorch","repo_kind":"listed","path":"src/data.py","file_url":"https://github.com/Alexander-H-Liu/End-to-end-ASR-Pytorch/blob/HEAD/src/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"340cb8273953e493"}},{"code_sha256_prefix":"bded34b0a51d533c","entry":"collect_audio_batch","repo":"s3prl/End-to-end-ASR-Pytorch","repo_kind":"listed","path":"src/data.py","file_url":"https://github.com/s3prl/End-to-end-ASR-Pytorch/blob/HEAD/src/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bded34b0a51d533c"}},{"code_sha256_prefix":"ca15b9ec596a30d0","entry":"collect_text_batch","repo":"Alexander-H-Liu/End-to-end-ASR-Pytorch","repo_kind":"listed","path":"src/data.py","file_url":"https://github.com/Alexander-H-Liu/End-to-end-ASR-Pytorch/blob/HEAD/src/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca15b9ec596a30d0"}},{"code_sha256_prefix":"c4ec9d88dda3f211","entry":"collect_text_batch","repo":"s3prl/End-to-end-ASR-Pytorch","repo_kind":"listed","path":"src/data.py","file_url":"https://github.com/s3prl/End-to-end-ASR-Pytorch/blob/HEAD/src/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4ec9d88dda3f211"}},{"code_sha256_prefix":"e2ec1a4e3221772d","entry":"create_dataset","repo":"Alexander-H-Liu/End-to-end-ASR-Pytorch","repo_kind":"listed","path":"src/data.py","file_url":"https://github.com/Alexander-H-Liu/End-to-end-ASR-Pytorch/blob/HEAD/src/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2ec1a4e3221772d"}},{"code_sha256_prefix":"3929b38d6fca37e2","entry":"create_dataset","repo":"neil-zeng/asr","repo_kind":"listed","path":"src/data.py","file_url":"https://github.com/neil-zeng/asr/blob/HEAD/src/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3929b38d6fca37e2"}},{"code_sha256_prefix":"ac71d8ab36c2fae0","entry":"create_dataset","repo":"s3prl/End-to-end-ASR-Pytorch","repo_kind":"listed","path":"src/data.py","file_url":"https://github.com/s3prl/End-to-end-ASR-Pytorch/blob/HEAD/src/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac71d8ab36c2fae0"}},{"code_sha256_prefix":"a7310a3d1f046472","entry":"generate_embedding","repo":"Alexander-H-Liu/End-to-end-ASR-Pytorch","repo_kind":"listed","path":"src/bert_embedding.py","file_url":"https://github.com/Alexander-H-Liu/End-to-end-ASR-Pytorch/blob/HEAD/src/bert_embedding.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7310a3d1f046472"}},{"code_sha256_prefix":"9a8e285c706f7fe8","entry":"get_testcases","repo":"s3prl/End-to-end-ASR-Pytorch","repo_kind":"listed","path":"slot_evaluate.py","file_url":"https://github.com/s3prl/End-to-end-ASR-Pytorch/blob/HEAD/slot_evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a8e285c706f7fe8"}},{"code_sha256_prefix":"90a0a3f3574b92a6","entry":"speech_aug_scheduler","repo":"Alexander-H-Liu/End-to-end-ASR-Pytorch","repo_kind":"listed","path":"src/optim.py","file_url":"https://github.com/Alexander-H-Liu/End-to-end-ASR-Pytorch/blob/HEAD/src/optim.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90a0a3f3574b92a6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}