{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/transformers-can-do-arithmetic-with-the-right","title":"Transformers Can Do Arithmetic with the Right Embeddings","arxiv_id":"2405.17399","date":"2024-05-27","proceeding":null,"authors":["Sean McLeish","Arpit Bansal","Alex Stein","Neel Jain","John Kirchenbauer","Brian R. Bartoldson","Bhavya Kailkhura","Abhinav Bhatele","Jonas Geiping","Avi Schwarzschild","Tom Goldstein"],"abstract":"The poor performance of transformers on arithmetic tasks seems to stem in large part from their inability to keep track of the exact position of each digit inside of a large span of digits. We mend this problem by adding an embedding to each digit that encodes its position relative to the start of the number. In addition to the boost these embeddings provide on their own, we show that this fix enables architectural modifications such as input injection and recurrent layers to improve performance even further. With positions resolved, we can study the logical extrapolation ability of transformers. Can they solve arithmetic problems that are larger and more complex than those in their training data? We find that training on only 20 digit numbers with a single GPU for one day, we can reach state-of-the-art performance, achieving up to 99% accuracy on 100 digit addition problems. Finally, we show that these gains in numeracy also unlock improvements on other multi-step reasoning tasks including sorting and multiplication.","url_abs":"https://arxiv.org/abs/2405.17399v2","url_pdf":"https://arxiv.org/pdf/2405.17399v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"transformers-can-do-arithmetic-with-the-right","repo_url":"https://github.com/mcleish7/arithmetic","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":null,"task_name":"Position"}],"methods":[{"method_slug":"mend","method_name":"MEND"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.17399","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.17399"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mcleish7/arithmetic","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":12,"unverified":1},"by_repo_kind":{"official":{"samples":13,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6904fbecf8a80e45","entry":"check_checkpointing","repo":"mcleish7/arithmetic","repo_kind":"official","path":"pretrain.py","file_url":"https://github.com/mcleish7/arithmetic/blob/HEAD/pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6904fbecf8a80e45"}},{"code_sha256_prefix":"93142aa14ecbc7be","entry":"count_digits","repo":"mcleish7/arithmetic","repo_kind":"official","path":"dataset_analysis.py","file_url":"https://github.com/mcleish7/arithmetic/blob/HEAD/dataset_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93142aa14ecbc7be"}},{"code_sha256_prefix":"cff91a92610dfa6a","entry":"find_file","repo":"mcleish7/arithmetic","repo_kind":"official","path":"pretty_plotter.py","file_url":"https://github.com/mcleish7/arithmetic/blob/HEAD/pretty_plotter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cff91a92610dfa6a"}},{"code_sha256_prefix":"7cdfbdb5267c36af","entry":"generate_dataset","repo":"mcleish7/arithmetic","repo_kind":"official","path":"create_data_split.py","file_url":"https://github.com/mcleish7/arithmetic/blob/HEAD/create_data_split.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7cdfbdb5267c36af"}},{"code_sha256_prefix":"52b246cf6203173b","entry":"generate_no_carry_addition","repo":"mcleish7/arithmetic","repo_kind":"official","path":"create_data_split.py","file_url":"https://github.com/mcleish7/arithmetic/blob/HEAD/create_data_split.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52b246cf6203173b"}},{"code_sha256_prefix":"93ef1c29a04c0d05","entry":"get_time_elapsed","repo":"mcleish7/arithmetic","repo_kind":"official","path":"pretrain.py","file_url":"https://github.com/mcleish7/arithmetic/blob/HEAD/pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93ef1c29a04c0d05"}},{"code_sha256_prefix":"403c2dbbe04c7692","entry":"has_carry","repo":"mcleish7/arithmetic","repo_kind":"official","path":"create_data_split.py","file_url":"https://github.com/mcleish7/arithmetic/blob/HEAD/create_data_split.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"403c2dbbe04c7692"}},{"code_sha256_prefix":"b3bb8f995329f91d","entry":"index_hints_helper","repo":"mcleish7/arithmetic","repo_kind":"official","path":"arithmetic_eval_quicker.py","file_url":"https://github.com/mcleish7/arithmetic/blob/HEAD/arithmetic_eval_quicker.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3bb8f995329f91d"}},{"code_sha256_prefix":"86e5db13667ce384","entry":"one_hot_vector","repo":"mcleish7/arithmetic","repo_kind":"official","path":"create_pos_or_variants.py","file_url":"https://github.com/mcleish7/arithmetic/blob/HEAD/create_pos_or_variants.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"86e5db13667ce384"}},{"code_sha256_prefix":"3bff376763abe533","entry":"read_dataset","repo":"mcleish7/arithmetic","repo_kind":"official","path":"dataset_analysis.py","file_url":"https://github.com/mcleish7/arithmetic/blob/HEAD/dataset_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3bff376763abe533"}},{"code_sha256_prefix":"877bea5a25e5eb7a","entry":"remove_leading_zeros","repo":"mcleish7/arithmetic","repo_kind":"official","path":"dataset_analysis.py","file_url":"https://github.com/mcleish7/arithmetic/blob/HEAD/dataset_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"877bea5a25e5eb7a"}},{"code_sha256_prefix":"dc757096a531ea2b","entry":"zero_vector","repo":"mcleish7/arithmetic","repo_kind":"official","path":"create_pos_or_variants.py","file_url":"https://github.com/mcleish7/arithmetic/blob/HEAD/create_pos_or_variants.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc757096a531ea2b"}},{"code_sha256_prefix":"63fd2ecb6d9d9780","entry":"grid_logic","repo":"mcleish7/arithmetic","repo_kind":"official","path":"arithmetic_eval_quicker.py","file_url":"https://github.com/mcleish7/arithmetic/blob/HEAD/arithmetic_eval_quicker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63fd2ecb6d9d9780"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}