{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/massively-multi-cultural-knowledge","title":"Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking","arxiv_id":"2402.09369","date":"2024-02-14","proceeding":null,"authors":["Yi Fung","Ruining Zhao","Jae Doo","Chenkai Sun","Heng Ji"],"abstract":"Pretrained large language models have revolutionized many applications but still face challenges related to cultural bias and a lack of cultural commonsense knowledge crucial for guiding cross-culture communication and interactions. Recognizing the shortcomings of existing methods in capturing the diverse and rich cultures across the world, this paper introduces a novel approach for massively multicultural knowledge acquisition. Specifically, our method strategically navigates from densely informative Wikipedia documents on cultural topics to an extensive network of linked pages. Leveraging this valuable source of data collection, we construct the CultureAtlas dataset, which covers a wide range of sub-country level geographical regions and ethnolinguistic groups, with data cleaning and preprocessing to ensure textual assertion sentence self-containment, as well as fine-grained cultural profile information extraction. Our dataset not only facilitates the evaluation of language model performance in culturally diverse contexts but also serves as a foundational tool for the development of culturally sensitive and aware language models. Our work marks an important step towards deeper understanding and bridging the gaps of cultural disparities in AI, to promote a more inclusive and balanced representation of global cultures in the digital domain.","url_abs":"https://arxiv.org/abs/2402.09369v1","url_pdf":"https://arxiv.org/pdf/2402.09369v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"massively-multi-cultural-knowledge","repo_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[{"method_slug":"aware","method_name":"AWARE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.09369","atlas_url":"https://app.syntology.ai/?focus=2402.09369","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.09369"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":7},"by_repo_kind":{"official":{"samples":14,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2fa0004a087a12a8","entry":"enum_list_of_keys","repo":"yrf1/llm-massivemulticulturenormsknowledge-nclb","repo_kind":"official","path":"src/preproc_data.py","file_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb/blob/HEAD/src/preproc_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fa0004a087a12a8"}},{"code_sha256_prefix":"2d66c77e326695e8","entry":"filter_by_CultureProfileElements","repo":"yrf1/llm-massivemulticulturenormsknowledge-nclb","repo_kind":"official","path":"eval/utils.py","file_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb/blob/HEAD/eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d66c77e326695e8"}},{"code_sha256_prefix":"17f85a76874f51d2","entry":"ln_by_ln_inference","repo":"yrf1/llm-massivemulticulturenormsknowledge-nclb","repo_kind":"official","path":"eval/benchmark_Mistral.py","file_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb/blob/HEAD/eval/benchmark_Mistral.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17f85a76874f51d2"}},{"code_sha256_prefix":"0b5e8fa2c8ed015f","entry":"run_eval_per_s_query","repo":"yrf1/llm-massivemulticulturenormsknowledge-nclb","repo_kind":"official","path":"eval/benchmark_Mistral.py","file_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb/blob/HEAD/eval/benchmark_Mistral.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b5e8fa2c8ed015f"}},{"code_sha256_prefix":"0f928623ca171da3","entry":"run_eval_per_s_query","repo":"yrf1/llm-massivemulticulturenormsknowledge-nclb","repo_kind":"official","path":"eval/benchmark_OpenAI.py","file_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb/blob/HEAD/eval/benchmark_OpenAI.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f928623ca171da3"}},{"code_sha256_prefix":"a558311992567d15","entry":"run_eval_per_s_query","repo":"yrf1/llm-massivemulticulturenormsknowledge-nclb","repo_kind":"official","path":"eval/benchmark_llama.py","file_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb/blob/HEAD/eval/benchmark_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a558311992567d15"}},{"code_sha256_prefix":"90f74a0232938024","entry":"topic_filt_4Wil","repo":"yrf1/llm-massivemulticulturenormsknowledge-nclb","repo_kind":"official","path":"src/preproc_data.py","file_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb/blob/HEAD/src/preproc_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90f74a0232938024"}},{"code_sha256_prefix":"16d32f9178023e5a","entry":"get_init_neg_ChatGPT","repo":"yrf1/llm-massivemulticulturenormsknowledge-nclb","repo_kind":"official","path":"src/construct_mm_benchmark.py","file_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb/blob/HEAD/src/construct_mm_benchmark.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16d32f9178023e5a"}},{"code_sha256_prefix":"03a1c44ac8696099","entry":"ln_by_ln_inference","repo":"yrf1/llm-massivemulticulturenormsknowledge-nclb","repo_kind":"official","path":"eval/benchmark_llama.py","file_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb/blob/HEAD/eval/benchmark_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03a1c44ac8696099"}},{"code_sha256_prefix":"54ae785eb9f2b038","entry":"read_computed_stats","repo":"yrf1/llm-massivemulticulturenormsknowledge-nclb","repo_kind":"official","path":"eval/utils.py","file_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb/blob/HEAD/eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54ae785eb9f2b038"}},{"code_sha256_prefix":"f1d36b9d9d7e8538","entry":"read_from_json_per_c_t","repo":"yrf1/llm-massivemulticulturenormsknowledge-nclb","repo_kind":"official","path":"eval/benchmark_Mistral.py","file_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb/blob/HEAD/eval/benchmark_Mistral.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1d36b9d9d7e8538"}},{"code_sha256_prefix":"6046e31eabdfb031","entry":"read_from_json_per_c_t","repo":"yrf1/llm-massivemulticulturenormsknowledge-nclb","repo_kind":"official","path":"eval/benchmark_Vicuna.py","file_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb/blob/HEAD/eval/benchmark_Vicuna.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6046e31eabdfb031"}},{"code_sha256_prefix":"7e2c5b0ef97143b9","entry":"read_from_json_per_c_t","repo":"yrf1/llm-massivemulticulturenormsknowledge-nclb","repo_kind":"official","path":"eval/benchmark_llama.py","file_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb/blob/HEAD/eval/benchmark_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e2c5b0ef97143b9"}},{"code_sha256_prefix":"cf415e0cdfeaee77","entry":"self_check_neg","repo":"yrf1/llm-massivemulticulturenormsknowledge-nclb","repo_kind":"official","path":"src/construct_mm_benchmark.py","file_url":"https://github.com/yrf1/llm-massivemulticulturenormsknowledge-nclb/blob/HEAD/src/construct_mm_benchmark.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf415e0cdfeaee77"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}