{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/auxiliary-task-demands-mask-the-capabilities","title":"Auxiliary task demands mask the capabilities of smaller language models","arxiv_id":"2404.02418","date":"2024-04-03","proceeding":null,"authors":["Jennifer Hu","Michael C. Frank"],"abstract":"Developmental psychologists have argued about when cognitive capacities such as language understanding or theory of mind emerge. These debates often hinge on the concept of \"task demands\" -- the auxiliary challenges associated with performing a particular evaluation -- that may mask the child's underlying ability. The same issues arise when measuring the capacities of language models (LMs): performance on a task is a function of the model's underlying knowledge, combined with the model's ability to interpret and perform the task given its available resources. Here, we show that for analogical reasoning, reflective reasoning, word prediction, and grammaticality judgments, evaluation methods with greater task demands yield lower performance than evaluations with reduced demands. This \"demand gap\" is most pronounced for models with fewer parameters and less training data. Our results illustrate that LM performance should not be interpreted as a direct indication of intelligence (or lack thereof), but as a reflection of capacities seen through the lens of researchers' design choices.","url_abs":"https://arxiv.org/abs/2404.02418v2","url_pdf":"https://arxiv.org/pdf/2404.02418v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"auxiliary-task-demands-mask-the-capabilities","repo_url":"https://github.com/jennhu/lm-task-demands","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2404.02418","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.02418"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jennhu/lm-task-demands","reach":{"status":"ok"}}],"summary":{"ran":11,"unverified":1},"by_repo_kind":{"official":{"samples":12,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"2124e54740298882","entry":"clean_response","repo":"jennhu/lm-task-demands","repo_kind":"official","path":"src/metrics/crt.py","file_url":"https://github.com/jennhu/lm-task-demands/blob/HEAD/src/metrics/crt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2124e54740298882"}},{"code_sha256_prefix":"54df85a235f3a419","entry":"evaluate","repo":"jennhu/lm-task-demands","repo_kind":"official","path":"src/metrics/blimp.py","file_url":"https://github.com/jennhu/lm-task-demands/blob/HEAD/src/metrics/blimp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"54df85a235f3a419"}},{"code_sha256_prefix":"b372c2e8aeb3cdf2","entry":"evaluate","repo":"jennhu/lm-task-demands","repo_kind":"official","path":"src/metrics/dgl.py","file_url":"https://github.com/jennhu/lm-task-demands/blob/HEAD/src/metrics/dgl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b372c2e8aeb3cdf2"}},{"code_sha256_prefix":"72bbe7d3ff89dea1","entry":"evaluate","repo":"jennhu/lm-task-demands","repo_kind":"official","path":"src/metrics/digit_mat.py","file_url":"https://github.com/jennhu/lm-task-demands/blob/HEAD/src/metrics/digit_mat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"72bbe7d3ff89dea1"}},{"code_sha256_prefix":"7d0ac95d172d351a","entry":"evaluate","repo":"jennhu/lm-task-demands","repo_kind":"official","path":"src/metrics/lambada.py","file_url":"https://github.com/jennhu/lm-task-demands/blob/HEAD/src/metrics/lambada.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d0ac95d172d351a"}},{"code_sha256_prefix":"79e34474e177966d","entry":"format_as_money","repo":"jennhu/lm-task-demands","repo_kind":"official","path":"src/metrics/crt.py","file_url":"https://github.com/jennhu/lm-task-demands/blob/HEAD/src/metrics/crt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"79e34474e177966d"}},{"code_sha256_prefix":"5cb1f9676a3537c0","entry":"get_file_safe_model_name","repo":"jennhu/lm-task-demands","repo_kind":"official","path":"configs/make_configs.py","file_url":"https://github.com/jennhu/lm-task-demands/blob/HEAD/configs/make_configs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5cb1f9676a3537c0"}},{"code_sha256_prefix":"ee0ff824ff43e5a4","entry":"get_metalinguistic_prompt","repo":"jennhu/lm-task-demands","repo_kind":"official","path":"src/metrics/lambada.py","file_url":"https://github.com/jennhu/lm-task-demands/blob/HEAD/src/metrics/lambada.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee0ff824ff43e5a4"}},{"code_sha256_prefix":"c9d64be95334ae5b","entry":"get_metalinguistic_prompt_paired","repo":"jennhu/lm-task-demands","repo_kind":"official","path":"src/metrics/blimp.py","file_url":"https://github.com/jennhu/lm-task-demands/blob/HEAD/src/metrics/blimp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c9d64be95334ae5b"}},{"code_sha256_prefix":"29db9cb7090f2405","entry":"make_prompt","repo":"jennhu/lm-task-demands","repo_kind":"official","path":"src/metrics/crt.py","file_url":"https://github.com/jennhu/lm-task-demands/blob/HEAD/src/metrics/crt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"29db9cb7090f2405"}},{"code_sha256_prefix":"e12034ea5a9846ea","entry":"make_yaml","repo":"jennhu/lm-task-demands","repo_kind":"official","path":"configs/make_configs.py","file_url":"https://github.com/jennhu/lm-task-demands/blob/HEAD/configs/make_configs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e12034ea5a9846ea"}},{"code_sha256_prefix":"a0f56d2545f6d9d5","entry":"logits_to_dist","repo":"jennhu/lm-task-demands","repo_kind":"official","path":"src/metrics/lambada.py","file_url":"https://github.com/jennhu/lm-task-demands/blob/HEAD/src/metrics/lambada.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a0f56d2545f6d9d5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}