{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-reversal-curse-llms-trained-on-a-is-b","title":"The Reversal Curse: LLMs trained on \"A is B\" fail to learn \"B is A\"","arxiv_id":"2309.12288","date":"2023-09-21","proceeding":null,"authors":["Lukas Berglund","Meg Tong","Max Kaufmann","Mikita Balesni","Asa Cooper Stickland","Tomasz Korbak","Owain Evans"],"abstract":"We expose a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form \"A is B\", it will not automatically generalize to the reverse direction \"B is A\". This is the Reversal Curse. For instance, if a model is trained on \"Valentina Tereshkova was the first woman to travel to space\", it will not automatically be able to answer the question, \"Who was the first woman to travel to space?\". Moreover, the likelihood of the correct answer (\"Valentina Tershkova\") will not be higher than for a random name. Thus, models do not generalize a prevalent pattern in their training set: if \"A is B\" occurs, \"B is A\" is more likely to occur. It is worth noting, however, that if \"A is B\" appears in-context, models can deduce the reverse relationship. We provide evidence for the Reversal Curse by finetuning GPT-3 and Llama-1 on fictitious statements such as \"Uriah Hawthorne is the composer of Abyssal Melodies\" and showing that they fail to correctly answer \"Who composed Abyssal Melodies?\". The Reversal Curse is robust across model sizes and model families and is not alleviated by data augmentation. We also evaluate ChatGPT (GPT-3.5 and GPT-4) on questions about real-world celebrities, such as \"Who is Tom Cruise's mother? [A: Mary Lee Pfeiffer]\" and the reverse \"Who is Mary Lee Pfeiffer's son?\". GPT-4 correctly answers questions like the former 79% of the time, compared to 33% for the latter. Code available at: https://github.com/lukasberglund/reversal_curse.","url_abs":"https://arxiv.org/abs/2309.12288v4","url_pdf":"https://arxiv.org/pdf/2309.12288v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-reversal-curse-llms-trained-on-a-is-b","repo_url":"https://github.com/lukasberglund/reversal_curse","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"the-reversal-curse-llms-trained-on-a-is-b","repo_url":"https://github.com/metr/public-tasks","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":null,"method_name":"Travel"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"fail","method_name":"fail"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2309.12288","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.12288"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/metr/public-tasks","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lukasberglund/reversal_curse","reach":{"status":"ok"}}],"summary":{"ran":9,"unverified":2},"by_repo_kind":{"official":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"2748d6bfde1f400a","entry":"convert_runs_to_df","repo":"lukasberglund/reversal_curse","repo_kind":"official","path":"src/wandb_utils.py","file_url":"https://github.com/lukasberglund/reversal_curse/blob/HEAD/src/wandb_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2748d6bfde1f400a"}},{"code_sha256_prefix":"8893b67058cf5e56","entry":"generate_wandb_substring_filter","repo":"lukasberglund/reversal_curse","repo_kind":"official","path":"src/wandb_utils.py","file_url":"https://github.com/lukasberglund/reversal_curse/blob/HEAD/src/wandb_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8893b67058cf5e56"}},{"code_sha256_prefix":"b34d526a98178a10","entry":"get_cost_per_1k_tokens","repo":"lukasberglund/reversal_curse","repo_kind":"official","path":"src/models/openai_complete.py","file_url":"https://github.com/lukasberglund/reversal_curse/blob/HEAD/src/models/openai_complete.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b34d526a98178a10"}},{"code_sha256_prefix":"aa42dd70be709dc2","entry":"get_openwebtext_path","repo":"lukasberglund/reversal_curse","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/lukasberglund/reversal_curse/blob/HEAD/src/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa42dd70be709dc2"}},{"code_sha256_prefix":"f94438c1b75c62d2","entry":"get_preprocess_function","repo":"lukasberglund/reversal_curse","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/lukasberglund/reversal_curse/blob/HEAD/src/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f94438c1b75c62d2"}},{"code_sha256_prefix":"e01bde09eb15910f","entry":"load_from_json","repo":"lukasberglund/reversal_curse","repo_kind":"official","path":"src/common.py","file_url":"https://github.com/lukasberglund/reversal_curse/blob/HEAD/src/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e01bde09eb15910f"}},{"code_sha256_prefix":"9ebcde58a78649e8","entry":"load_from_jsonl","repo":"lukasberglund/reversal_curse","repo_kind":"official","path":"src/common.py","file_url":"https://github.com/lukasberglund/reversal_curse/blob/HEAD/src/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ebcde58a78649e8"}},{"code_sha256_prefix":"6fe4b4ebee97037c","entry":"load_from_yaml","repo":"lukasberglund/reversal_curse","repo_kind":"official","path":"src/common.py","file_url":"https://github.com/lukasberglund/reversal_curse/blob/HEAD/src/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6fe4b4ebee97037c"}},{"code_sha256_prefix":"e35d51ba8013184f","entry":"log_after_retry","repo":"lukasberglund/reversal_curse","repo_kind":"official","path":"src/models/openai_complete.py","file_url":"https://github.com/lukasberglund/reversal_curse/blob/HEAD/src/models/openai_complete.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e35d51ba8013184f"}},{"code_sha256_prefix":"c9de75a7c39c5012","entry":"load_model","repo":"lukasberglund/reversal_curse","repo_kind":"official","path":"src/models/common.py","file_url":"https://github.com/lukasberglund/reversal_curse/blob/HEAD/src/models/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c9de75a7c39c5012"}},{"code_sha256_prefix":"e18553bc097d701c","entry":"load_tokenizer","repo":"lukasberglund/reversal_curse","repo_kind":"official","path":"src/models/common.py","file_url":"https://github.com/lukasberglund/reversal_curse/blob/HEAD/src/models/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e18553bc097d701c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}