{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stemm-self-learning-with-speech-text-manifold","title":"STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation","arxiv_id":"2203.10426","date":"2022-03-20","proceeding":"ACL 2022 5","authors":["Qingkai Fang","Rong Ye","Lei LI","Yang Feng","Mingxuan Wang"],"abstract":"How to learn a better speech representation for end-to-end speech-to-text translation (ST) with limited labeled data? Existing techniques often attempt to transfer powerful machine translation (MT) capabilities to ST, but neglect the representation discrepancy across modalities. In this paper, we propose the Speech-TExt Manifold Mixup (STEMM) method to calibrate such discrepancy. Specifically, we mix up the representation sequences of different modalities, and take both unimodal speech sequences and multimodal mixed sequences as input to the translation model in parallel, and regularize their output predictions with a self-learning framework. Experiments on MuST-C speech translation benchmark and further analysis show that our method effectively alleviates the cross-modal representation discrepancy, and achieves significant improvements over a strong baseline on eight translation directions.","url_abs":"https://arxiv.org/abs/2203.10426v1","url_pdf":"https://arxiv.org/pdf/2203.10426v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stemm-self-learning-with-speech-text-manifold","repo_url":"https://github.com/ictnlp/stemm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"speech-to-text","task_name":"Speech-to-Text"},{"task_slug":"speech-to-text-translation","task_name":"Speech-to-Text Translation"}],"methods":[{"method_slug":"manifold-mixup","method_name":"Manifold Mixup"},{"method_slug":"mixup","method_name":"Mixup"},{"method_slug":"self-learning","method_name":"Self-Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2203.10426","atlas_url":"https://app.syntology.ai/?focus=2203.10426","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.10426"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ictnlp/stemm","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/ictnlp/STEMM","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"840d9070e4fd8ba6","entry":"find_offsets","repo":"ictnlp/STEMM","repo_kind":"official","path":"fairseq/file_chunker_utils.py","file_url":"https://github.com/ictnlp/STEMM/blob/HEAD/fairseq/file_chunker_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"840d9070e4fd8ba6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}