{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bertaqa-how-much-do-language-models-know","title":"BertaQA: How Much Do Language Models Know About Local Culture?","arxiv_id":"2406.07302","date":"2024-06-11","proceeding":null,"authors":["Julen Etxaniz","Gorka Azkune","Aitor Soroa","Oier Lopez de Lacalle","Mikel Artetxe"],"abstract":"Large Language Models (LLMs) exhibit extensive knowledge about the world, but most evaluations have been limited to global or anglocentric subjects. This raises the question of how well these models perform on topics relevant to other cultures, whose presence on the web is not that prominent. To address this gap, we introduce BertaQA, a multiple-choice trivia dataset that is parallel in English and Basque. The dataset consists of a local subset with questions pertinent to the Basque culture, and a global subset with questions of broader interest. We find that state-of-the-art LLMs struggle with local cultural knowledge, even as they excel on global topics. However, we show that continued pre-training in Basque significantly improves the models' performance on Basque culture, even when queried in English. To our knowledge, this is the first solid evidence of knowledge transfer from a low-resource to a high-resource language. Our analysis sheds light on the complex interplay between language and knowledge, and reveals that some prior findings do not fully hold when reassessed on local topics. Our dataset and evaluation code are available under open licenses at https://github.com/juletx/BertaQA.","url_abs":"https://arxiv.org/abs/2406.07302v2","url_pdf":"https://arxiv.org/pdf/2406.07302v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bertaqa-how-much-do-language-models-know","repo_url":"https://github.com/juletx/bertaqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2406.07302","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.07302"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/juletx/BertaQA","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/juletx/bertaqa","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"ran_draft_wrong":3,"unverified":4},"by_repo_kind":{"official":{"samples":12,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c74ca09cfdbb61f9","entry":"count_lines","repo":"juletx/bertaqa","repo_kind":"official","path":"translate/dataset.py","file_url":"https://github.com/juletx/bertaqa/blob/HEAD/translate/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c74ca09cfdbb61f9"}},{"code_sha256_prefix":"9a4a652e139275a2","entry":"encode_string","repo":"juletx/bertaqa","repo_kind":"official","path":"translate/translate.py","file_url":"https://github.com/juletx/bertaqa/blob/HEAD/translate/translate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a4a652e139275a2"}},{"code_sha256_prefix":"294fccd9026f79d1","entry":"few_shot_messages","repo":"juletx/BertaQA","repo_kind":"official","path":"openai/evaluate.py","file_url":"https://github.com/juletx/BertaQA/blob/HEAD/openai/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"294fccd9026f79d1"}},{"code_sha256_prefix":"77874b775ba08799","entry":"few_shot_messages","repo":"juletx/bertaqa","repo_kind":"official","path":"anthropic/evaluate.py","file_url":"https://github.com/juletx/bertaqa/blob/HEAD/anthropic/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77874b775ba08799"}},{"code_sha256_prefix":"7151aae860cbcdd0","entry":"format_question","repo":"juletx/BertaQA","repo_kind":"official","path":"openai/evaluate.py","file_url":"https://github.com/juletx/BertaQA/blob/HEAD/openai/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7151aae860cbcdd0"}},{"code_sha256_prefix":"4fbde4146ac83d26","entry":"get_texts","repo":"juletx/BertaQA","repo_kind":"official","path":"translate/translate_dataset_nllb.py","file_url":"https://github.com/juletx/BertaQA/blob/HEAD/translate/translate_dataset_nllb.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4fbde4146ac83d26"}},{"code_sha256_prefix":"80b18c3134e34e8a","entry":"get_texts","repo":"juletx/bertaqa","repo_kind":"official","path":"translate/translate_dataset_few_shot.py","file_url":"https://github.com/juletx/bertaqa/blob/HEAD/translate/translate_dataset_few_shot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"80b18c3134e34e8a"}},{"code_sha256_prefix":"c735fa8c6de07b1b","entry":"get_texts","repo":"juletx/bertaqa","repo_kind":"official","path":"translate/translate_dataset_madlad.py","file_url":"https://github.com/juletx/bertaqa/blob/HEAD/translate/translate_dataset_madlad.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c735fa8c6de07b1b"}},{"code_sha256_prefix":"e859752b12f6b7b5","entry":"get_dataset","repo":"juletx/bertaqa","repo_kind":"official","path":"translate/translate_dataset_few_shot.py","file_url":"https://github.com/juletx/bertaqa/blob/HEAD/translate/translate_dataset_few_shot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e859752b12f6b7b5"}},{"code_sha256_prefix":"2be3ed4b02dce2bf","entry":"get_dataset","repo":"juletx/bertaqa","repo_kind":"official","path":"translate/translate_dataset_madlad.py","file_url":"https://github.com/juletx/bertaqa/blob/HEAD/translate/translate_dataset_madlad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2be3ed4b02dce2bf"}},{"code_sha256_prefix":"6d9a2be0e8aaa21a","entry":"get_few_shot_dataset","repo":"juletx/bertaqa","repo_kind":"official","path":"translate/translate_dataset_few_shot.py","file_url":"https://github.com/juletx/bertaqa/blob/HEAD/translate/translate_dataset_few_shot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d9a2be0e8aaa21a"}},{"code_sha256_prefix":"1b85278ca24c90ca","entry":"load_bertaqa","repo":"juletx/bertaqa","repo_kind":"official","path":"anthropic/evaluate.py","file_url":"https://github.com/juletx/bertaqa/blob/HEAD/anthropic/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b85278ca24c90ca"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}