{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mc-2-a-multilingual-corpus-of-minority","title":"MC$^2$: Towards Transparent and Culturally-Aware NLP for Minority Languages in China","arxiv_id":"2311.08348","date":"2023-11-14","proceeding":null,"authors":["Chen Zhang","Mingxu Tao","Quzhe Huang","Jiuheng Lin","Zhibin Chen","Yansong Feng"],"abstract":"Current large language models demonstrate deficiencies in understanding low-resource languages, particularly the minority languages in China. This limitation stems from the scarcity of available pre-training data. To address this accessibility challenge, we present MC$^2$, a Multilingual Corpus of Minority Languages in China, which is the largest open-source corpus of its kind so far. MC$^2$ includes four underrepresented languages: Tibetan, Uyghur, Kazakh, and Mongolian. Notably, we focus on the less common writing systems of Kazakh and Mongolian, i.e., Kazakh Arabic script and traditional Mongolian script, respectively, which have been long neglected in previous corpus construction efforts. Recognizing the prevalence of language contamination within existing corpora, we adopt a quality-centric solution for collecting MC$^2$, prioritizing accuracy while enhancing diversity. Furthermore, we underscore the importance of attending to the multiplicity of writing systems, which is closely related to the cultural awareness of the resulting models. The MC$^2$ corpus and related models are made public to the community.","url_abs":"https://arxiv.org/abs/2311.08348v2","url_pdf":"https://arxiv.org/pdf/2311.08348v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mc-2-a-multilingual-corpus-of-minority","repo_url":"https://github.com/luciusssss/mc2_corpus","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"mc-2-a-multilingual-corpus-of-minority","repo_url":"https://github.com/luciusssss/mlic-eval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.08348","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.08348"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luciusssss/mc2_corpus","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luciusssss/mlic-eval","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1},"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"57c6adbd2fa9bbe8","entry":"convert_dataset_into_prompt_title","repo":"luciusssss/mlic-eval","repo_kind":"listed","path":"infer.py","file_url":"https://github.com/luciusssss/mlic-eval/blob/HEAD/infer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57c6adbd2fa9bbe8"}},{"code_sha256_prefix":"6e02320e67d19916","entry":"convert_dataset_into_prompt_translation","repo":"luciusssss/mlic-eval","repo_kind":"listed","path":"infer.py","file_url":"https://github.com/luciusssss/mlic-eval/blob/HEAD/infer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e02320e67d19916"}},{"code_sha256_prefix":"d9945aa058cbefe7","entry":"read_data","repo":"luciusssss/mc2_corpus","repo_kind":"official","path":"scripts/culturax_to_json.py","file_url":"https://github.com/luciusssss/mc2_corpus/blob/HEAD/scripts/culturax_to_json.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d9945aa058cbefe7"}},{"code_sha256_prefix":"dc6235427befec3f","entry":"remove_special_tokens","repo":"luciusssss/mlic-eval","repo_kind":"listed","path":"infer.py","file_url":"https://github.com/luciusssss/mlic-eval/blob/HEAD/infer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc6235427befec3f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}