{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/are-we-falling-in-a-middle-intelligence-trap","title":"An Analysis and Mitigation of the Reversal Curse","arxiv_id":"2311.07468","date":"2023-11-13","proceeding":null,"authors":["Ang Lv","Kaiyi Zhang","Shufang Xie","Quan Tu","Yuhan Chen","Ji-Rong Wen","Rui Yan"],"abstract":"Recent research observed a noteworthy phenomenon in large language models (LLMs), referred to as the ``reversal curse.'' The reversal curse is that when dealing with two entities, denoted as $a$ and $b$, connected by their relation $R$ and its inverse $R^{-1}$, LLMs excel in handling sequences in the form of ``$aRb$,'' but encounter challenges when processing ``$bR^{-1}a$,'' whether in generation or comprehension. For instance, GPT-4 can accurately respond to the query ``Tom Cruise's mother is?'' with ``Mary Lee Pfeiffer,'' but it struggles to provide a satisfactory answer when asked ``Mary Lee Pfeiffer's son is?'' In this paper, we undertake the first-ever study of how the reversal curse happens in LLMs. Our investigations reveal that the reversal curse can stem from the specific training objectives, which become particularly evident in the widespread use of next-token prediction within most causal language models. We hope this initial investigation can draw more attention to the reversal curse, as well as other underlying limitations in current LLMs.","url_abs":"https://arxiv.org/abs/2311.07468v3","url_pdf":"https://arxiv.org/pdf/2311.07468v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"are-we-falling-in-a-middle-intelligence-trap","repo_url":"https://github.com/trestad/mitigating-reversal-curse","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"glm","method_name":"GLM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.07468","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.07468"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/trestad/mitigating-reversal-curse","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":10},"by_repo_kind":{"official":{"samples":14,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"f0d20ebd827390d6","entry":"compute_bleu","repo":"trestad/mitigating-reversal-curse","repo_kind":"official","path":"generate.py","file_url":"https://github.com/trestad/mitigating-reversal-curse/blob/HEAD/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f0d20ebd827390d6"}},{"code_sha256_prefix":"33553b56020f3138","entry":"compute_metric","repo":"trestad/mitigating-reversal-curse","repo_kind":"official","path":"generate.py","file_url":"https://github.com/trestad/mitigating-reversal-curse/blob/HEAD/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"33553b56020f3138"}},{"code_sha256_prefix":"0fe702a77b67a5de","entry":"evaluate_completion","repo":"trestad/mitigating-reversal-curse","repo_kind":"official","path":"generate.py","file_url":"https://github.com/trestad/mitigating-reversal-curse/blob/HEAD/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0fe702a77b67a5de"}},{"code_sha256_prefix":"8a3c4f29d18d7783","entry":"new_torch_call","repo":"trestad/mitigating-reversal-curse","repo_kind":"official","path":"run_mlm.py","file_url":"https://github.com/trestad/mitigating-reversal-curse/blob/HEAD/run_mlm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a3c4f29d18d7783"}},{"code_sha256_prefix":"37a5eed2dbd663ca","entry":"gelu_fast","repo":"trestad/mitigating-reversal-curse","repo_kind":"official","path":"transformers/src/transformers/activations_tf.py","file_url":"https://github.com/trestad/mitigating-reversal-curse/blob/HEAD/transformers/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"37a5eed2dbd663ca"}},{"code_sha256_prefix":"b176efb26f72e507","entry":"get_architectures_from_config_class","repo":"trestad/mitigating-reversal-curse","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/trestad/mitigating-reversal-curse/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b176efb26f72e507"}},{"code_sha256_prefix":"3d999b95009c7537","entry":"get_config_class_from_processor_class","repo":"trestad/mitigating-reversal-curse","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/trestad/mitigating-reversal-curse/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d999b95009c7537"}},{"code_sha256_prefix":"0c0b81b081919d8e","entry":"get_processor_types_from_config_class","repo":"trestad/mitigating-reversal-curse","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/trestad/mitigating-reversal-curse/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c0b81b081919d8e"}},{"code_sha256_prefix":"9a09cbf63a4ddff4","entry":"hertz_to_mel","repo":"trestad/mitigating-reversal-curse","repo_kind":"official","path":"transformers/src/transformers/audio_utils.py","file_url":"https://github.com/trestad/mitigating-reversal-curse/blob/HEAD/transformers/src/transformers/audio_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a09cbf63a4ddff4"}},{"code_sha256_prefix":"7e7c601df5b22499","entry":"mel_filter_bank","repo":"trestad/mitigating-reversal-curse","repo_kind":"official","path":"transformers/src/transformers/audio_utils.py","file_url":"https://github.com/trestad/mitigating-reversal-curse/blob/HEAD/transformers/src/transformers/audio_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e7c601df5b22499"}},{"code_sha256_prefix":"2650530d2f0dd9b2","entry":"mel_to_hertz","repo":"trestad/mitigating-reversal-curse","repo_kind":"official","path":"transformers/src/transformers/audio_utils.py","file_url":"https://github.com/trestad/mitigating-reversal-curse/blob/HEAD/transformers/src/transformers/audio_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2650530d2f0dd9b2"}},{"code_sha256_prefix":"cc8c8c3ebf0c343f","entry":"mish","repo":"trestad/mitigating-reversal-curse","repo_kind":"official","path":"transformers/src/transformers/activations_tf.py","file_url":"https://github.com/trestad/mitigating-reversal-curse/blob/HEAD/transformers/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc8c8c3ebf0c343f"}},{"code_sha256_prefix":"d30fdeea401b3cc5","entry":"prepare_model_for_training","repo":"trestad/mitigating-reversal-curse","repo_kind":"official","path":"run_mlm.py","file_url":"https://github.com/trestad/mitigating-reversal-curse/blob/HEAD/run_mlm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d30fdeea401b3cc5"}},{"code_sha256_prefix":"e2d56cb91f999bef","entry":"quick_gelu","repo":"trestad/mitigating-reversal-curse","repo_kind":"official","path":"transformers/src/transformers/activations_tf.py","file_url":"https://github.com/trestad/mitigating-reversal-curse/blob/HEAD/transformers/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2d56cb91f999bef"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}