{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/counting-and-algorithmic-generalization-with","title":"Counting and Algorithmic Generalization with Transformers","arxiv_id":"2310.08661","date":"2023-10-12","proceeding":null,"authors":["Simon Ouellette","Rolf Pfister","Hansueli Jud"],"abstract":"Algorithmic generalization in machine learning refers to the ability to learn the underlying algorithm that generates data in a way that generalizes out-of-distribution. This is generally considered a difficult task for most machine learning algorithms. Here, we analyze algorithmic generalization when counting is required, either implicitly or explicitly. We show that standard Transformers are based on architectural decisions that hinder out-of-distribution performance for such tasks. In particular, we discuss the consequences of using layer normalization and of normalizing the attention weights via softmax. With ablation of the problematic operations, we demonstrate that a modified transformer can exhibit a good algorithmic generalization performance on counting while using a very lightweight architecture.","url_abs":"https://arxiv.org/abs/2310.08661v2","url_pdf":"https://arxiv.org/pdf/2310.08661v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"counting-and-algorithmic-generalization-with","repo_url":"https://github.com/simonouellette35/countingwithtransformers","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"layer-normalization","method_name":"Layer Normalization"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.08661","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.08661"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/simonouellette35/countingwithtransformers","reach":null}],"summary":{"ran_fixture":2,"ran_honours":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"a1d91b2d800d3bbb","entry":"one_hot_encode","repo":"simonouellette35/countingwithtransformers","repo_kind":"official","path":"trainStd_Transformer_Count.py","file_url":"https://github.com/simonouellette35/countingwithtransformers/blob/HEAD/trainStd_Transformer_Count.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a1d91b2d800d3bbb"}},{"code_sha256_prefix":"61ceef529c7ce104","entry":"preprocessTarget","repo":"simonouellette35/countingwithtransformers","repo_kind":"official","path":"trainStd_Transformer_Count.py","file_url":"https://github.com/simonouellette35/countingwithtransformers/blob/HEAD/trainStd_Transformer_Count.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61ceef529c7ce104"}},{"code_sha256_prefix":"b3077197a0cf2b1a","entry":"single_pred_accuracy","repo":"simonouellette35/countingwithtransformers","repo_kind":"official","path":"trainStd_Transformer_Count.py","file_url":"https://github.com/simonouellette35/countingwithtransformers/blob/HEAD/trainStd_Transformer_Count.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3077197a0cf2b1a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}