{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/goldfish-monolingual-language-models-for-350","title":"Goldfish: Monolingual Language Models for 350 Languages","arxiv_id":"2408.10441","date":"2024-08-19","proceeding":null,"authors":["Tyler A. Chang","Catherine Arnett","Zhuowen Tu","Benjamin K. Bergen"],"abstract":"For many low-resource languages, the only available language models are large multilingual models trained on many languages simultaneously. However, using FLORES perplexity as a metric, we find that these models perform worse than bigrams for many languages (e.g. 24% of languages in XGLM 4.5B; 43% in BLOOM 7.1B). To facilitate research that focuses on low-resource languages, we pre-train and release Goldfish, a suite of monolingual autoregressive Transformer language models up to 125M parameters for 350 languages. The Goldfish reach lower FLORES perplexities than BLOOM, XGLM, and MaLA-500 on 98 of 204 FLORES languages, despite each Goldfish model being over 10x smaller. However, the Goldfish significantly underperform larger multilingual models on reasoning benchmarks, suggesting that for low-resource languages, multilinguality primarily improves general reasoning abilities rather than basic text generation. We release models trained on 5MB (350 languages), 10MB (288 languages), 100MB (166 languages), and 1GB (83 languages) of text data where available. The Goldfish models are available as baselines, fine-tuning sources, or augmentations to existing models in low-resource NLP research, and they are further useful for crosslinguistic studies requiring maximally comparable models across languages.","url_abs":"https://arxiv.org/abs/2408.10441v1","url_pdf":"https://arxiv.org/pdf/2408.10441v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"goldfish-monolingual-language-models-for-350","repo_url":"https://github.com/tylerachang/goldfish","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bloom","method_name":"BLOOM"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2408.10441","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.10441"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tylerachang/goldfish","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"0716d75b426ac991","entry":"get_file_examples","repo":"tylerachang/goldfish","repo_kind":"official","path":"eval_code/flores_eval_bigrams.py","file_url":"https://github.com/tylerachang/goldfish/blob/HEAD/eval_code/flores_eval_bigrams.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0716d75b426ac991"}},{"code_sha256_prefix":"43c965489c0553cc","entry":"get_ngram_surprisals","repo":"tylerachang/goldfish","repo_kind":"official","path":"eval_code/flores_eval_bigrams.py","file_url":"https://github.com/tylerachang/goldfish/blob/HEAD/eval_code/flores_eval_bigrams.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"43c965489c0553cc"}},{"code_sha256_prefix":"5bc6614356e962e2","entry":"get_ngram_surprisals_with_backoff","repo":"tylerachang/goldfish","repo_kind":"official","path":"eval_code/flores_eval_bigrams.py","file_url":"https://github.com/tylerachang/goldfish/blob/HEAD/eval_code/flores_eval_bigrams.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5bc6614356e962e2"}},{"code_sha256_prefix":"c96db622e1f75fe6","entry":"score_text","repo":"tylerachang/goldfish","repo_kind":"official","path":"example_score_text.py","file_url":"https://github.com/tylerachang/goldfish/blob/HEAD/example_score_text.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c96db622e1f75fe6"}},{"code_sha256_prefix":"c362562812238de9","entry":"load_model","repo":"tylerachang/goldfish","repo_kind":"official","path":"eval_code/belebele_eval_goldfish.py","file_url":"https://github.com/tylerachang/goldfish/blob/HEAD/eval_code/belebele_eval_goldfish.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c362562812238de9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}