{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-the-numerical-reasoning-skills-of","title":"Arithmetic-Based Pretraining -- Improving Numeracy of Pretrained Language Models","arxiv_id":"2205.06733","date":"2022-05-13","proceeding":null,"authors":["Dominic Petrak","Nafise Sadat Moosavi","Iryna Gurevych"],"abstract":"State-of-the-art pretrained language models tend to perform below their capabilities when applied out-of-the-box on tasks that require understanding and working with numbers. Recent work suggests two main reasons for this: (1) popular tokenisation algorithms have limited expressiveness for numbers, and (2) common pretraining objectives do not target numeracy. Approaches that address these shortcomings usually require architectural changes or pretraining from scratch. In this paper, we propose a new extended pretraining approach called Arithmetic-Based Pretraining that jointly addresses both in one extended pretraining step without requiring architectural changes or pretraining from scratch. Arithmetic-Based Pretraining combines contrastive learning to improve the number representation, and a novel extended pretraining objective called Inferable Number Prediction Task to improve numeracy. Our experiments show the effectiveness of Arithmetic-Based Pretraining in three different tasks that require improved numeracy, i.e., reading comprehension in the DROP dataset, inference-on-tables in the InfoTabs dataset, and table-to-text generation in the WikiBio and SciGen datasets.","url_abs":"https://arxiv.org/abs/2205.06733v2","url_pdf":"https://arxiv.org/pdf/2205.06733v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-the-numerical-reasoning-skills-of","repo_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"improving-the-numerical-reasoning-skills-of","repo_url":"https://github.com/ukplab/starsem2023-arithmetic-based-pretraining","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"table-to-text-generation","task_name":"Table-to-Text Generation"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2205.06733","atlas_url":"https://app.syntology.ai/?focus=2205.06733","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.06733"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ukplab/starsem2023-arithmetic-based-pretraining","reach":null}],"summary":{"ran":12,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":16,"ran":13,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"db28bccfc5903f08","entry":"MNRLoss","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"losses.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/losses.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"db28bccfc5903f08"}},{"code_sha256_prefix":"83a07f606672603b","entry":"char_level_representation_encoding","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"char_level_representation.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/char_level_representation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"83a07f606672603b"}},{"code_sha256_prefix":"112d014b2004b7e4","entry":"convert_text_scientific_notation","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"scientific_notation.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/scientific_notation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"112d014b2004b7e4"}},{"code_sha256_prefix":"a3826392847cbef3","entry":"count_trainable_parameters","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"callbacks.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/callbacks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a3826392847cbef3"}},{"code_sha256_prefix":"c2d5c83a83cd96ef","entry":"get_checkpoint_callback","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"callbacks.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/callbacks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c2d5c83a83cd96ef"}},{"code_sha256_prefix":"5aa0f4cc91bef627","entry":"get_early_stopping_callback","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"callbacks.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/callbacks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5aa0f4cc91bef627"}},{"code_sha256_prefix":"f28ffee2040b376b","entry":"get_lines","repo":"ukplab/starsem2023-arithmetic-based-pretraining","repo_kind":"official","path":"evaluation/evaluate.py","file_url":"https://github.com/ukplab/starsem2023-arithmetic-based-pretraining/blob/HEAD/evaluation/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f28ffee2040b376b"}},{"code_sha256_prefix":"eae00458a5958ff0","entry":"get_lines","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eae00458a5958ff0"}},{"code_sha256_prefix":"3966b063074a6009","entry":"get_scientific_notation","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"scientific_notation.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/scientific_notation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3966b063074a6009"}},{"code_sha256_prefix":"0327ea5057921c8d","entry":"is_float","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0327ea5057921c8d"}},{"code_sha256_prefix":"78a995a45343b076","entry":"masked_loss","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"losses.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/losses.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"78a995a45343b076"}},{"code_sha256_prefix":"05764486881f3364","entry":"rebuild_string_in_new_notation","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"scientific_notation.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/scientific_notation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"05764486881f3364"}},{"code_sha256_prefix":"75996269cea01a0c","entry":"trim_batch","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"75996269cea01a0c"}},{"code_sha256_prefix":"abd5798bdcfa8197","entry":"add_args","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"args.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/args.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"abd5798bdcfa8197"}},{"code_sha256_prefix":"4bdcc6fd40a2753c","entry":"char_level_representation_impl","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"char_level_representation.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/char_level_representation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4bdcc6fd40a2753c"}},{"code_sha256_prefix":"bd8cd57d2905efde","entry":"pad_or_trim","repo":"ukplab/emnlp2022-reasoning-aware-pretraining","repo_kind":"official","path":"char_level_representation.py","file_url":"https://github.com/ukplab/emnlp2022-reasoning-aware-pretraining/blob/HEAD/char_level_representation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bd8cd57d2905efde"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}