{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/findings-of-the-babylm-challenge-sample","title":"Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora","arxiv_id":"2504.08165","date":"2025-04-10","proceeding":null,"authors":["Alex Warstadt","Aaron Mueller","Leshem Choshen","Ethan Wilcox","Chengxu Zhuang","Juan Ciro","Rafael Mosquera","Bhargavi Paranjape","Adina Williams","Tal Linzen","Ryan Cotterell"],"abstract":"Children can acquire language from less than 100 million words of input. Large language models are far less data-efficient: they typically require 3 or 4 orders of magnitude more data and still do not perform as well as humans on many evaluations. These intensive resource demands limit the ability of researchers to train new models and use existing models as developmentally plausible cognitive models. The BabyLM Challenge is a communal effort in which participants compete to optimize language model training on a fixed data budget. Submissions are compared on various evaluation tasks targeting grammatical ability, downstream task performance, and generalization. Participants can submit to up to three tracks with progressively looser data restrictions. From over 30 submissions, we extract concrete recommendations on how best to train data-efficient language models, and on where future efforts should (and perhaps should not) focus. The winning submissions using the LTG-BERT architecture (Samuel et al., 2023) outperformed models trained on trillions of words. Other submissions achieved strong results through training on shorter input sequences or training a student model on a pretrained teacher. Curriculum learning attempts, which accounted for a large number of submissions, were largely unsuccessful, though some showed modest improvements.","url_abs":"https://arxiv.org/abs/2504.08165v1","url_pdf":"https://arxiv.org/pdf/2504.08165v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"findings-of-the-babylm-challenge-sample","repo_url":"https://github.com/babylm/evaluation-pipeline","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2504.08165","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.08165"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/babylm/evaluation-pipeline","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran":1,"unverified":7},"by_repo_kind":{"official":{"samples":9,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0928f497e20fb443","entry":"mean","repo":"babylm/evaluation-pipeline","repo_kind":"official","path":"lm_eval/api/metric.py","file_url":"https://github.com/babylm/evaluation-pipeline/blob/HEAD/lm_eval/api/metric.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0928f497e20fb443"}},{"code_sha256_prefix":"518b2201c66c2ad3","entry":"pop_stddev","repo":"babylm/evaluation-pipeline","repo_kind":"official","path":"lm_eval/api/metric.py","file_url":"https://github.com/babylm/evaluation-pipeline/blob/HEAD/lm_eval/api/metric.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"518b2201c66c2ad3"}},{"code_sha256_prefix":"48fb64bd72378059","entry":"get_result","repo":"babylm/evaluation-pipeline","repo_kind":"official","path":"lm_eval/models/openai_completions.py","file_url":"https://github.com/babylm/evaluation-pipeline/blob/HEAD/lm_eval/models/openai_completions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48fb64bd72378059"}},{"code_sha256_prefix":"de4e47a5e5126171","entry":"hash_args","repo":"babylm/evaluation-pipeline","repo_kind":"official","path":"lm_eval/api/model.py","file_url":"https://github.com/babylm/evaluation-pipeline/blob/HEAD/lm_eval/api/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de4e47a5e5126171"}},{"code_sha256_prefix":"910a58b3f0fe9967","entry":"make_table","repo":"babylm/evaluation-pipeline","repo_kind":"official","path":"lm_eval/evaluator.py","file_url":"https://github.com/babylm/evaluation-pipeline/blob/HEAD/lm_eval/evaluator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"910a58b3f0fe9967"}},{"code_sha256_prefix":"103f3ce94e4d19f3","entry":"make_task_dict","repo":"babylm/evaluation-pipeline","repo_kind":"official","path":"collect_results.py","file_url":"https://github.com/babylm/evaluation-pipeline/blob/HEAD/collect_results.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"103f3ce94e4d19f3"}},{"code_sha256_prefix":"99a9bbebcd3ca50f","entry":"mean_pooling","repo":"babylm/evaluation-pipeline","repo_kind":"official","path":"transformers_modified/t5.py","file_url":"https://github.com/babylm/evaluation-pipeline/blob/HEAD/transformers_modified/t5.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99a9bbebcd3ca50f"}},{"code_sha256_prefix":"f6e58cdc3909f465","entry":"sample_stddev","repo":"babylm/evaluation-pipeline","repo_kind":"official","path":"lm_eval/api/metric.py","file_url":"https://github.com/babylm/evaluation-pipeline/blob/HEAD/lm_eval/api/metric.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6e58cdc3909f465"}},{"code_sha256_prefix":"6e2bb9c187be8efa","entry":"stop_sequences_criteria","repo":"babylm/evaluation-pipeline","repo_kind":"official","path":"lm_eval/models/huggingface.py","file_url":"https://github.com/babylm/evaluation-pipeline/blob/HEAD/lm_eval/models/huggingface.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e2bb9c187be8efa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}