{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/subword-mapping-and-anchoring-across","title":"Subword Mapping and Anchoring across Languages","arxiv_id":"2109.04556","date":"2021-09-09","proceeding":"Findings (EMNLP) 2021 11","authors":["Giorgos Vernikos","Andrei Popescu-Belis"],"abstract":"State-of-the-art multilingual systems rely on shared vocabularies that sufficiently cover all considered languages. To this end, a simple and frequently used approach makes use of subword vocabularies constructed jointly over several languages. We hypothesize that such vocabularies are suboptimal due to false positives (identical subwords with different meanings across languages) and false negatives (different subwords with similar meanings). To address these issues, we propose Subword Mapping and Anchoring across Languages (SMALA), a method to construct bilingual subword vocabularies. SMALA extracts subword alignments using an unsupervised state-of-the-art mapping technique and uses them to create cross-lingual anchors based on subword similarities. We demonstrate the benefits of SMALA for cross-lingual natural language inference (XNLI), where it improves zero-shot transfer to an unseen language without task-specific data, but only by sharing subword embeddings. Moreover, in neural machine translation, we show that joint subword vocabularies obtained with SMALA lead to higher BLEU scores on sentences that contain many false positives and false negatives.","url_abs":"https://arxiv.org/abs/2109.04556v1","url_pdf":"https://arxiv.org/pdf/2109.04556v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"subword-mapping-and-anchoring-across","repo_url":"https://github.com/georgevern/smala","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"cross-lingual-natural-language-inference","task_name":"Cross-Lingual Natural Language Inference"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.04556","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.04556"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/georgevern/smala","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"6e7c054c599e7013","entry":"get_similarity","repo":"georgevern/smala","repo_kind":"official","path":"extract_alignments.py","file_url":"https://github.com/georgevern/smala/blob/HEAD/extract_alignments.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"6e7c054c599e7013"}},{"code_sha256_prefix":"781409d243fb95c0","entry":"read","repo":"georgevern/smala","repo_kind":"official","path":"extract_alignments.py","file_url":"https://github.com/georgevern/smala/blob/HEAD/extract_alignments.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"781409d243fb95c0"}},{"code_sha256_prefix":"5d9796e5f560047d","entry":"get_mutual_sim","repo":"georgevern/smala","repo_kind":"official","path":"extract_alignments.py","file_url":"https://github.com/georgevern/smala/blob/HEAD/extract_alignments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5d9796e5f560047d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}