{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/induced-natural-language-rationales-and","title":"Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Models","arxiv_id":"2208.11445","date":"2022-08-24","proceeding":null,"authors":["Mirelle Bueno","Carlos Gemmell","Jeffrey Dalton","Roberto Lotufo","Rodrigo Nogueira"],"abstract":"The ability to extrapolate, i.e., to make predictions on sequences that are longer than those presented as training examples, is a challenging problem for current deep learning models. Recent work shows that this limitation persists in state-of-the-art Transformer-based models. Most solutions to this problem use specific architectures or training methods that do not generalize to other tasks. We demonstrate that large language models can succeed in extrapolation without modifying their architecture or training procedure. Our experimental results show that generating step-by-step rationales and introducing marker tokens are both required for effective extrapolation. First, we induce a language model to produce step-by-step rationales before outputting the answer to effectively communicate the task to the model. However, as sequences become longer, we find that current models struggle to keep track of token positions. To address this issue, we interleave output tokens with markup tokens that act as explicit positional and counting symbols. Our findings show how these two complementary approaches enable remarkable sequence extrapolation and highlight a limitation of current architectures to effectively generalize without explicit surface form guidance. Code available at https://github.com/MirelleB/induced-rationales-markup-tokens","url_abs":"https://arxiv.org/abs/2208.11445v3","url_pdf":"https://arxiv.org/pdf/2208.11445v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"induced-natural-language-rationales-and","repo_url":"https://github.com/mirelleb/induced-rationales-markup-tokens","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2208.11445","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.11445"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mirelleb/induced-rationales-markup-tokens","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"15e1a9b5ae261165","entry":"compute_exact_match","repo":"mirelleb/induced-rationales-markup-tokens","repo_kind":"official","path":"SCAN/few-shot-text-davinci-002/evaluate_model.py","file_url":"https://github.com/mirelleb/induced-rationales-markup-tokens/blob/HEAD/SCAN/few-shot-text-davinci-002/evaluate_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15e1a9b5ae261165"}},{"code_sha256_prefix":"5e2eea9004649619","entry":"operation_add","repo":"mirelleb/induced-rationales-markup-tokens","repo_kind":"official","path":"generate_dataset.py","file_url":"https://github.com/mirelleb/induced-rationales-markup-tokens/blob/HEAD/generate_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e2eea9004649619"}},{"code_sha256_prefix":"09054d4c92a616ed","entry":"posprocess","repo":"mirelleb/induced-rationales-markup-tokens","repo_kind":"official","path":"SCAN/few-shot-text-davinci-002/evaluate_model.py","file_url":"https://github.com/mirelleb/induced-rationales-markup-tokens/blob/HEAD/SCAN/few-shot-text-davinci-002/evaluate_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09054d4c92a616ed"}},{"code_sha256_prefix":"3ff404a924c6d556","entry":"sample_dataset","repo":"mirelleb/induced-rationales-markup-tokens","repo_kind":"official","path":"generate_dataset.py","file_url":"https://github.com/mirelleb/induced-rationales-markup-tokens/blob/HEAD/generate_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ff404a924c6d556"}},{"code_sha256_prefix":"e242b648fb05afcb","entry":"scan_preprocess","repo":"mirelleb/induced-rationales-markup-tokens","repo_kind":"official","path":"generate_dataset.py","file_url":"https://github.com/mirelleb/induced-rationales-markup-tokens/blob/HEAD/generate_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e242b648fb05afcb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}