{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unnatural-error-correction-gpt-4-can-almost","title":"Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Text","arxiv_id":"2311.18805","date":"2023-11-30","proceeding":null,"authors":["Qi Cao","Takeshi Kojima","Yutaka Matsuo","Yusuke Iwasawa"],"abstract":"While Large Language Models (LLMs) have achieved remarkable performance in many tasks, much about their inner workings remains unclear. In this study, we present novel experimental insights into the resilience of LLMs, particularly GPT-4, when subjected to extensive character-level permutations. To investigate this, we first propose the Scrambled Bench, a suite designed to measure the capacity of LLMs to handle scrambled input, in terms of both recovering scrambled sentences and answering questions given scrambled context. The experimental results indicate that most powerful LLMs demonstrate the capability akin to typoglycemia, a phenomenon where humans can understand the meaning of words even when the letters within those words are scrambled, as long as the first and last letters remain in place. More surprisingly, we found that only GPT-4 nearly flawlessly processes inputs with unnatural errors, even under the extreme condition, a task that poses significant challenges for other LLMs and often even for humans. Specifically, GPT-4 can almost perfectly reconstruct the original sentences from scrambled ones, decreasing the edit distance by 95%, even when all letters within each word are entirely scrambled. It is counter-intuitive that LLMs can exhibit such resilience despite severe disruption to input tokenization caused by scrambled text.","url_abs":"https://arxiv.org/abs/2311.18805v1","url_pdf":"https://arxiv.org/pdf/2311.18805v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unnatural-error-correction-gpt-4-can-almost","repo_url":"https://github.com/ccqq77/unnatural-error-correction","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2311.18805","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.18805"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ccqq77/unnatural-error-correction","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"b412f82e496dca53","entry":"scramble_word","repo":"ccqq77/unnatural-error-correction","repo_kind":"official","path":"process.py","file_url":"https://github.com/ccqq77/unnatural-error-correction/blob/HEAD/process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b412f82e496dca53"}},{"code_sha256_prefix":"e16a2fbacdccd797","entry":"scramble_word_keepfirst","repo":"ccqq77/unnatural-error-correction","repo_kind":"official","path":"process.py","file_url":"https://github.com/ccqq77/unnatural-error-correction/blob/HEAD/process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e16a2fbacdccd797"}},{"code_sha256_prefix":"d2691274473431c2","entry":"strip_tags","repo":"ccqq77/unnatural-error-correction","repo_kind":"official","path":"process.py","file_url":"https://github.com/ccqq77/unnatural-error-correction/blob/HEAD/process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2691274473431c2"}},{"code_sha256_prefix":"71f4ef76b0f78b75","entry":"decoder_for_hf","repo":"ccqq77/unnatural-error-correction","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ccqq77/unnatural-error-correction/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71f4ef76b0f78b75"}},{"code_sha256_prefix":"87d2435ca3afa304","entry":"decoder_for_openai","repo":"ccqq77/unnatural-error-correction","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ccqq77/unnatural-error-correction/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87d2435ca3afa304"}},{"code_sha256_prefix":"28145bbcf1d4f9ea","entry":"print_now","repo":"ccqq77/unnatural-error-correction","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ccqq77/unnatural-error-correction/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28145bbcf1d4f9ea"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}