{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/emergent-abilities-in-reduced-scale","title":"Emergent Abilities in Reduced-Scale Generative Language Models","arxiv_id":"2404.02204","date":"2024-04-02","proceeding":null,"authors":["Sherin Muckatira","Vijeta Deshpande","Vladislav Lialin","Anna Rumshisky"],"abstract":"Large language models can solve new tasks without task-specific fine-tuning. This ability, also known as in-context learning (ICL), is considered an emergent ability and is primarily seen in large language models with billions of parameters. This study investigates if such emergent properties are strictly tied to model size or can be demonstrated by smaller models trained on reduced-scale data. To explore this, we simplify pre-training data and pre-train 36 causal language models with parameters varying from 1 million to 165 million parameters. We show that models trained on this simplified pre-training data demonstrate enhanced zero-shot capabilities across various tasks in simplified language, achieving performance comparable to that of pre-trained models six times larger on unrestricted language. This suggests that downscaling the language allows zero-shot learning capabilities to emerge in models with limited size. Additionally, we find that these smaller models pre-trained on simplified data demonstrate a power law relationship between the evaluation loss and the three scaling factors: compute, dataset size, and model size.","url_abs":"https://arxiv.org/abs/2404.02204v1","url_pdf":"https://arxiv.org/pdf/2404.02204v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"emergent-abilities-in-reduced-scale","repo_url":"https://github.com/text-machine-lab/mini_gpt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.02204","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.02204"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/text-machine-lab/mini_gpt","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"ran":8,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":13,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"text-machine-lab/mini_gpt","repo_kind":"official","path":"src/modeling_llama.py","file_url":"https://github.com/text-machine-lab/mini_gpt/blob/HEAD/src/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"ff55f4ed6ff9941f","entry":"fixed_pos_embedding","repo":"text-machine-lab/mini_gpt","repo_kind":"official","path":"src/pos_emb_classes.py","file_url":"https://github.com/text-machine-lab/mini_gpt/blob/HEAD/src/pos_emb_classes.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff55f4ed6ff9941f"}},{"code_sha256_prefix":"36a211a87651fca0","entry":"get_color_for_par","repo":"text-machine-lab/mini_gpt","repo_kind":"official","path":"src/plot_utils.py","file_url":"https://github.com/text-machine-lab/mini_gpt/blob/HEAD/src/plot_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36a211a87651fca0"}},{"code_sha256_prefix":"1530e960806b04c2","entry":"get_flops_hoffman","repo":"text-machine-lab/mini_gpt","repo_kind":"official","path":"src/plot_utils.py","file_url":"https://github.com/text-machine-lab/mini_gpt/blob/HEAD/src/plot_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1530e960806b04c2"}},{"code_sha256_prefix":"253925b01572a7c8","entry":"get_flops_kaplan","repo":"text-machine-lab/mini_gpt","repo_kind":"official","path":"src/plot_utils.py","file_url":"https://github.com/text-machine-lab/mini_gpt/blob/HEAD/src/plot_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"253925b01572a7c8"}},{"code_sha256_prefix":"1e973bfb2c3ff69e","entry":"group_texts","repo":"text-machine-lab/mini_gpt","repo_kind":"official","path":"src/data_collation.py","file_url":"https://github.com/text-machine-lab/mini_gpt/blob/HEAD/src/data_collation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e973bfb2c3ff69e"}},{"code_sha256_prefix":"bab8f65d92178226","entry":"init_weights","repo":"text-machine-lab/mini_gpt","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/text-machine-lab/mini_gpt/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bab8f65d92178226"}},{"code_sha256_prefix":"16b2dd199b000a25","entry":"preprocess_function","repo":"text-machine-lab/mini_gpt","repo_kind":"official","path":"src/data_collation.py","file_url":"https://github.com/text-machine-lab/mini_gpt/blob/HEAD/src/data_collation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"16b2dd199b000a25"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"text-machine-lab/mini_gpt","repo_kind":"official","path":"src/modeling_llama.py","file_url":"https://github.com/text-machine-lab/mini_gpt/blob/HEAD/src/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"70016ef148e9a350","entry":"truncate_frequency","repo":"text-machine-lab/mini_gpt","repo_kind":"official","path":"src/pos_emb_classes.py","file_url":"https://github.com/text-machine-lab/mini_gpt/blob/HEAD/src/pos_emb_classes.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70016ef148e9a350"}},{"code_sha256_prefix":"7d7798779fd0d3e6","entry":"apply_helical_pos_emb","repo":"text-machine-lab/mini_gpt","repo_kind":"official","path":"src/modeling_llama.py","file_url":"https://github.com/text-machine-lab/mini_gpt/blob/HEAD/src/modeling_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d7798779fd0d3e6"}},{"code_sha256_prefix":"060de65e98d13c2f","entry":"configure_logger","repo":"text-machine-lab/mini_gpt","repo_kind":"official","path":"src/logging_config.py","file_url":"https://github.com/text-machine-lab/mini_gpt/blob/HEAD/src/logging_config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"060de65e98d13c2f"}},{"code_sha256_prefix":"65b1731cd3980981","entry":"create_data_collator","repo":"text-machine-lab/mini_gpt","repo_kind":"official","path":"src/data_collation.py","file_url":"https://github.com/text-machine-lab/mini_gpt/blob/HEAD/src/data_collation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"65b1731cd3980981"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}