{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/number-cookbook-number-understanding-of","title":"Number Cookbook: Number Understanding of Language Models and How to Improve It","arxiv_id":"2411.03766","date":"2024-11-06","proceeding":null,"authors":["Haotong Yang","Yi Hu","Shijia Kang","Zhouchen Lin","Muhan Zhang"],"abstract":"Large language models (LLMs) can solve an increasing number of complex reasoning tasks while making surprising mistakes in basic numerical understanding and processing (such as 9.11 > 9.9). The latter ability is essential for tackling complex arithmetic and mathematical problems and serves as a foundation for most reasoning tasks, but previous work paid little attention to it or only discussed several restricted tasks (like integer addition). In this paper, we comprehensively investigate the numerical understanding and processing ability (NUPA) of LLMs. Firstly, we introduce a benchmark covering four common numerical representations and 17 distinct numerical tasks in four major categories, resulting in 41 meaningful combinations in total. These tasks are derived from primary and secondary education curricula, encompassing nearly all everyday numerical understanding and processing scenarios, and the rules of these tasks are very simple and clear. Through the benchmark, we find that current LLMs fail frequently in many of the tasks. To study the problem, we train small models with existing and potential techniques for enhancing NUPA (such as tokenizers, PEs, and number formats), comprehensively evaluating their effectiveness using our testbed. We also finetune practical-scale LLMs on our proposed NUPA tasks and find that 1) naive finetuning can improve NUPA a lot on many but not all tasks, and 2) surprisingly, techniques designed to enhance NUPA prove ineffective for finetuning pretrained models. We further explore the impact of chain-of-thought techniques on NUPA. Our work provides a more detailed and comprehensive understanding of NUPA in LLMs. Our benchmark and code are released at https://github.com/GraphPKU/number_cookbook.","url_abs":"https://arxiv.org/abs/2411.03766v2","url_pdf":"https://arxiv.org/pdf/2411.03766v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"number-cookbook-number-understanding-of","repo_url":"https://github.com/graphpku/number_cookbook","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.03766","atlas_url":"https://app.syntology.ai/?focus=2411.03766","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.03766"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/GraphPKU/number_cookbook","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/graphpku/number_cookbook","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"f736c789212c5664","entry":"decon","repo":"GraphPKU/number_cookbook","repo_kind":"official","path":"src/datagen/benchmark_generate.py","file_url":"https://github.com/GraphPKU/number_cookbook/blob/HEAD/src/datagen/benchmark_generate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f736c789212c5664"}},{"code_sha256_prefix":"002fbfbbb4ab46aa","entry":"change_model_config","repo":"graphpku/number_cookbook","repo_kind":"official","path":"src/number_model/utils.py","file_url":"https://github.com/graphpku/number_cookbook/blob/HEAD/src/number_model/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"002fbfbbb4ab46aa"}},{"code_sha256_prefix":"6c792dface69fba6","entry":"process_batch","repo":"graphpku/number_cookbook","repo_kind":"official","path":"src/number_model/utils.py","file_url":"https://github.com/graphpku/number_cookbook/blob/HEAD/src/number_model/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"6c792dface69fba6"}},{"code_sha256_prefix":"95f7dbea1688a08f","entry":"readable_model_size","repo":"graphpku/number_cookbook","repo_kind":"official","path":"src/number_model/utils.py","file_url":"https://github.com/graphpku/number_cookbook/blob/HEAD/src/number_model/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"95f7dbea1688a08f"}},{"code_sha256_prefix":"274d7bd6f3115cb3","entry":"to_int","repo":"graphpku/number_cookbook","repo_kind":"official","path":"src/datagen/data_utils.py","file_url":"https://github.com/graphpku/number_cookbook/blob/HEAD/src/datagen/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"274d7bd6f3115cb3"}},{"code_sha256_prefix":"bf77b7b6b3302960","entry":"to_list","repo":"graphpku/number_cookbook","repo_kind":"official","path":"src/datagen/data_utils.py","file_url":"https://github.com/graphpku/number_cookbook/blob/HEAD/src/datagen/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"bf77b7b6b3302960"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}