{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/are-multilingual-llms-culturally-diverse","title":"Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings","arxiv_id":"2309.08591","date":"2023-09-15","proceeding":null,"authors":["Chen Cecilia Liu","Fajri Koto","Timothy Baldwin","Iryna Gurevych"],"abstract":"Large language models (LLMs) are highly adept at question answering and reasoning tasks, but when reasoning in a situational context, human expectations vary depending on the relevant cultural common ground. As languages are associated with diverse cultures, LLMs should also be culturally-diverse reasoners. In this paper, we study the ability of a wide range of state-of-the-art multilingual LLMs (mLLMs) to reason with proverbs and sayings in a conversational context. Our experiments reveal that: (1) mLLMs \"know\" limited proverbs and memorizing proverbs does not mean understanding them within a conversational context; (2) mLLMs struggle to reason with figurative proverbs and sayings, and when asked to select the wrong answer (instead of asking it to select the correct answer); and (3) there is a \"culture gap\" in mLLMs when reasoning about proverbs and sayings translated from other languages. We construct and release our evaluation dataset MAPS (MulticultrAl Proverbs and Sayings) for proverb understanding with conversational context for six different languages.","url_abs":"https://arxiv.org/abs/2309.08591v2","url_pdf":"https://arxiv.org/pdf/2309.08591v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"are-multilingual-llms-culturally-diverse","repo_url":"https://github.com/UKPLab/maps","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2309.08591","atlas_url":"https://app.syntology.ai/?focus=2309.08591","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.08591"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/UKPLab/maps","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":8,"unverified":6},"by_repo_kind":{"official":{"samples":14,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"2dbf41f4c8c1c696","entry":"eval_loop","repo":"UKPLab/maps","repo_kind":"official","path":"experiments/ft_standard.py","file_url":"https://github.com/UKPLab/maps/blob/HEAD/experiments/ft_standard.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2dbf41f4c8c1c696"}},{"code_sha256_prefix":"2e6247334c966e94","entry":"generate_statistics_and_report","repo":"UKPLab/maps","repo_kind":"official","path":"experiments/utils.py","file_url":"https://github.com/UKPLab/maps/blob/HEAD/experiments/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2e6247334c966e94"}},{"code_sha256_prefix":"2ca6722a19c1a3f5","entry":"get_group2id","repo":"UKPLab/maps","repo_kind":"official","path":"experiments/src/datasets.py","file_url":"https://github.com/UKPLab/maps/blob/HEAD/experiments/src/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2ca6722a19c1a3f5"}},{"code_sha256_prefix":"5e276571782d6345","entry":"get_right_patterns","repo":"UKPLab/maps","repo_kind":"official","path":"experiments/ft_standard.py","file_url":"https://github.com/UKPLab/maps/blob/HEAD/experiments/ft_standard.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5e276571782d6345"}},{"code_sha256_prefix":"9f242d774bb34423","entry":"load_dataset","repo":"UKPLab/maps","repo_kind":"official","path":"experiments/src/datasets.py","file_url":"https://github.com/UKPLab/maps/blob/HEAD/experiments/src/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9f242d774bb34423"}},{"code_sha256_prefix":"80fb7fccf35faf62","entry":"load_memorized","repo":"UKPLab/maps","repo_kind":"official","path":"experiments/src/datasets.py","file_url":"https://github.com/UKPLab/maps/blob/HEAD/experiments/src/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"80fb7fccf35faf62"}},{"code_sha256_prefix":"6d5ac8ff47b442d6","entry":"masking_last_word","repo":"UKPLab/maps","repo_kind":"official","path":"experiments/src/model_collection.py","file_url":"https://github.com/UKPLab/maps/blob/HEAD/experiments/src/model_collection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6d5ac8ff47b442d6"}},{"code_sha256_prefix":"2aa10a3a19545c99","entry":"prep_groups","repo":"UKPLab/maps","repo_kind":"official","path":"experiments/qa_experiments.py","file_url":"https://github.com/UKPLab/maps/blob/HEAD/experiments/qa_experiments.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2aa10a3a19545c99"}},{"code_sha256_prefix":"0ac57076cb5129a5","entry":"get_k_shot_patterns","repo":"UKPLab/maps","repo_kind":"official","path":"experiments/src/patterns.py","file_url":"https://github.com/UKPLab/maps/blob/HEAD/experiments/src/patterns.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0ac57076cb5129a5"}},{"code_sha256_prefix":"99c79d7bb4be9115","entry":"get_logits","repo":"UKPLab/maps","repo_kind":"official","path":"experiments/qa_experiments.py","file_url":"https://github.com/UKPLab/maps/blob/HEAD/experiments/qa_experiments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"99c79d7bb4be9115"}},{"code_sha256_prefix":"4140916caf64322c","entry":"load_model","repo":"UKPLab/maps","repo_kind":"official","path":"experiments/src/model_collection.py","file_url":"https://github.com/UKPLab/maps/blob/HEAD/experiments/src/model_collection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4140916caf64322c"}},{"code_sha256_prefix":"4f119901b07ef1a4","entry":"perturb_last_word","repo":"UKPLab/maps","repo_kind":"official","path":"experiments/src/model_collection.py","file_url":"https://github.com/UKPLab/maps/blob/HEAD/experiments/src/model_collection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4f119901b07ef1a4"}},{"code_sha256_prefix":"a9921212ad7599b7","entry":"prep_dataest","repo":"UKPLab/maps","repo_kind":"official","path":"experiments/ft_standard.py","file_url":"https://github.com/UKPLab/maps/blob/HEAD/experiments/ft_standard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a9921212ad7599b7"}},{"code_sha256_prefix":"474e04af966d0942","entry":"seq2seq","repo":"UKPLab/maps","repo_kind":"official","path":"experiments/qa_experiments.py","file_url":"https://github.com/UKPLab/maps/blob/HEAD/experiments/qa_experiments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"474e04af966d0942"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}