{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-04208","title":"Nsanku: Evaluating Zero-Shot Translation Performance of LLMs for Ghanaian Languages","arxiv_id":"2605.04208","date":"2026-05-05","proceeding":null,"authors":["Stephen E. Moore","Mich-Seth Owusu","Akwasi Asare","Lawrence Adu Gyamfi","Paul Azunre","Joel Budu","Jonathan Asiamah","Elias Dzobo","Kelvin Newman","Edmund O. Benefo","Gerhardt Datsomor","Onesimus Addo Appiah","Ama Branoa Banful","Lucas Woedem Kpatah","Saani Mustapha Deishini","John Ayernor"],"abstract":"Large language models (LLMs) have demonstrated impressive multilingual capabilities for well-resourced languages, yet their performance on low-resource African languages remains poorly understood and largely unevaluated. This paper presents Nsanku, a systematic benchmark that evaluates the zero-shot machine translation performance of 19 open-weight and proprietary LLMs across 43 Ghanaian languages paired with English. Evaluation sentences were sourced from the YouVersion Bible platform, providing 300 sentence pairs per language. Two complementary automatic metrics are employed: Bilingual Evaluation Understudy (BLEU) and Character n-gram F-Score (chrF), alongside an average accuracy score and a cross-language consistency dimension. Nsanku represents the most comprehensive LLM translation evaluation for Ghanaian languages conducted to date. Results show that gemini-2.5-flash achieves the highest overall average score of 26.88 (BLEU: 24.60, chrF: 29.16), followed by claude-sonnet-4-5 at 24.87 (BLEU: 22.46, chrF: 27.28) and gpt-4.1 at 23.20 (BLEU: 21.15, chrF: 25.24). Among open-weight models, kimi-k2-instruct-0905 leads at an average score of 20.87. A critical finding from the consistency analysis is that no model and no language reached the Leaders quadrant of high performance and high consistency simultaneously, indicating that current LLMs are not yet reliably usable for Ghanaian language translation at scale. Siwu achieved the highest per-language average score at 25.73 while Nkonya scored lowest at 11.65. Nsanku establishes a publicly available, community-extensible evaluation infrastructure for African language NLP research.","url_abs":"https://arxiv.org/abs/2605.04208","url_pdf":"https://arxiv.org/pdf/2605.04208","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.04208","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.04208"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/GhanaNLP/nsanku","reach":null}],"summary":{"ran_draft_wrong":4,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"e0172f8087767b4f","entry":"extract_language_pair_from_filename","repo":"GhanaNLP/nsanku","repo_kind":"found_in_text","path":"repo/main-with-similarity-bk.py","file_url":"https://github.com/GhanaNLP/nsanku/blob/HEAD/repo/main-with-similarity-bk.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e0172f8087767b4f"}},{"code_sha256_prefix":"0fca2e7e724039a6","entry":"get_output_filename","repo":"GhanaNLP/nsanku","repo_kind":"found_in_text","path":"repo/main-with-similarity-bk.py","file_url":"https://github.com/GhanaNLP/nsanku/blob/HEAD/repo/main-with-similarity-bk.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0fca2e7e724039a6"}},{"code_sha256_prefix":"dceffe868db232be","entry":"process_csv","repo":"GhanaNLP/nsanku","repo_kind":"found_in_text","path":"repo/main-with-similarity-bk.py","file_url":"https://github.com/GhanaNLP/nsanku/blob/HEAD/repo/main-with-similarity-bk.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dceffe868db232be"}},{"code_sha256_prefix":"866646f65a0ab145","entry":"sample_dataframe","repo":"GhanaNLP/nsanku","repo_kind":"found_in_text","path":"repo/main-with-similarity-bk.py","file_url":"https://github.com/GhanaNLP/nsanku/blob/HEAD/repo/main-with-similarity-bk.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"866646f65a0ab145"}},{"code_sha256_prefix":"15c72c618600d5c2","entry":"run_full_process","repo":"GhanaNLP/nsanku","repo_kind":"found_in_text","path":"repo/main-with-similarity-bk.py","file_url":"https://github.com/GhanaNLP/nsanku/blob/HEAD/repo/main-with-similarity-bk.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15c72c618600d5c2"}},{"code_sha256_prefix":"9612f8c9bfd83ef4","entry":"save_processing_state","repo":"GhanaNLP/nsanku","repo_kind":"found_in_text","path":"repo/main-with-similarity-bk.py","file_url":"https://github.com/GhanaNLP/nsanku/blob/HEAD/repo/main-with-similarity-bk.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9612f8c9bfd83ef4"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}