{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/r-grams-unsupervised-learning-of-semantic","title":"R-grams: Unsupervised Learning of Semantic Units in Natural Language","arxiv_id":"1808.04670","date":"2018-08-14","proceeding":"WS 2019 5","authors":["Ariel Ekgren","Amaru Cuba Gyllensten","Magnus Sahlgren"],"abstract":"This paper investigates data-driven segmentation using Re-Pair or Byte Pair\nEncoding-techniques. In contrast to previous work which has primarily been\nfocused on subword units for machine translation, we are interested in the\ngeneral properties of such segments above the word level. We call these\nsegments r-grams, and discuss their properties and the effect they have on the\ntoken frequency distribution. The proposed approach is evaluated by\ndemonstrating its viability in embedding techniques, both in monolingual and\nmultilingual test settings. We also provide a number of qualitative examples of\nthe proposed methodology, demonstrating its viability as a language-invariant\nsegmentation procedure.","url_abs":"http://arxiv.org/abs/1808.04670v2","url_pdf":"http://arxiv.org/pdf/1808.04670v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"r-grams-unsupervised-learning-of-semantic","repo_url":"https://github.com/bakirillov/rgramlib","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}