{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/score-a-framework-for-self-contradictory","title":"Self-Contradictory Reasoning Evaluation and Detection","arxiv_id":"2311.09603","date":"2023-11-16","proceeding":null,"authors":["Ziyi Liu","Soumya Sanyal","Isabelle Lee","Yongkang Du","Rahul Gupta","Yang Liu","Jieyu Zhao"],"abstract":"In a plethora of recent work, large language models (LLMs) demonstrated impressive reasoning ability, but many proposed downstream reasoning tasks only focus on final answers. Two fundamental questions persist: 1) how consistent is the reasoning, and 2) can models detect unreliable reasoning? In this paper, we investigate self-contradictory (Self-Contra) reasoning, where the model reasoning does not support its answers. To answer 1), we define and assess the Self-Contra rate across three datasets and delve into finer-grained categories of Self-Contra reasoning. We find that LLMs often contradict themselves in reasoning tasks involving contextual information understanding or commonsense. The model may generate correct answers by taking shortcuts in reasoning or overlooking contextual evidence, leading to compromised reasoning. For 2), we task the state-of-the-art model GPT-4 with identifying Self-Contra reasoning and finer-grained fallacies. We find that finer-grained categories enhanced detection can improve GPT-4's ability to detect Self-Contra. However, it is only able to detect Self-Contra with a 52.2% F1 score, much lower compared to 66.7% for humans. Our results indicate that current LLMs lack the robustness necessary for reliable reasoning and we emphasize the urgent need for establishing best practices in comprehensive reasoning evaluations beyond pure performance-based metrics.","url_abs":"https://arxiv.org/abs/2311.09603v4","url_pdf":"https://arxiv.org/pdf/2311.09603v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"score-a-framework-for-self-contradictory","repo_url":"https://github.com/uscnlp-lime/Self-Contradictory","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.09603","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.09603"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uscnlp-lime/Self-Contradictory","reach":{"status":"ok"}}],"summary":{"ran":8,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"e31f2e11cd735813","entry":"compare_answers","repo":"uscnlp-lime/Self-Contradictory","repo_kind":"official","path":"auto_detection/get_metrics.py","file_url":"https://github.com/uscnlp-lime/Self-Contradictory/blob/HEAD/auto_detection/get_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e31f2e11cd735813"}},{"code_sha256_prefix":"da8a01c0f4495d2e","entry":"exclude_answer","repo":"uscnlp-lime/Self-Contradictory","repo_kind":"official","path":"auto_detection/evaluate_ir_o.py","file_url":"https://github.com/uscnlp-lime/Self-Contradictory/blob/HEAD/auto_detection/evaluate_ir_o.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da8a01c0f4495d2e"}},{"code_sha256_prefix":"ef660d2f13a9f587","entry":"extract_reasoning_elements","repo":"uscnlp-lime/Self-Contradictory","repo_kind":"official","path":"auto_detection/evaluate_fga.py","file_url":"https://github.com/uscnlp-lime/Self-Contradictory/blob/HEAD/auto_detection/evaluate_fga.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef660d2f13a9f587"}},{"code_sha256_prefix":"5ac1b64a85172751","entry":"generate_prompt","repo":"uscnlp-lime/Self-Contradictory","repo_kind":"official","path":"generate_reasoning_multiple.py","file_url":"https://github.com/uscnlp-lime/Self-Contradictory/blob/HEAD/generate_reasoning_multiple.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ac1b64a85172751"}},{"code_sha256_prefix":"43a651673691945e","entry":"get_result_binary","repo":"uscnlp-lime/Self-Contradictory","repo_kind":"official","path":"auto_detection/get_metrics.py","file_url":"https://github.com/uscnlp-lime/Self-Contradictory/blob/HEAD/auto_detection/get_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"43a651673691945e"}},{"code_sha256_prefix":"c20e4fdc0fac95eb","entry":"get_result_ensemble","repo":"uscnlp-lime/Self-Contradictory","repo_kind":"official","path":"auto_detection/get_metrics.py","file_url":"https://github.com/uscnlp-lime/Self-Contradictory/blob/HEAD/auto_detection/get_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c20e4fdc0fac95eb"}},{"code_sha256_prefix":"bce89358f2f2ebe6","entry":"separate_sentences","repo":"uscnlp-lime/Self-Contradictory","repo_kind":"official","path":"auto_detection/evaluate_ir_o.py","file_url":"https://github.com/uscnlp-lime/Self-Contradictory/blob/HEAD/auto_detection/evaluate_ir_o.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bce89358f2f2ebe6"}},{"code_sha256_prefix":"61a2f19859750b4a","entry":"split_reasoning_answer","repo":"uscnlp-lime/Self-Contradictory","repo_kind":"official","path":"auto_detection/evaluate_fga.py","file_url":"https://github.com/uscnlp-lime/Self-Contradictory/blob/HEAD/auto_detection/evaluate_fga.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61a2f19859750b4a"}},{"code_sha256_prefix":"b065d8863e78b2be","entry":"extract_evidence","repo":"uscnlp-lime/Self-Contradictory","repo_kind":"official","path":"auto_detection/evaluate_fga.py","file_url":"https://github.com/uscnlp-lime/Self-Contradictory/blob/HEAD/auto_detection/evaluate_fga.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b065d8863e78b2be"}},{"code_sha256_prefix":"387ea733adfb813a","entry":"process_df","repo":"uscnlp-lime/Self-Contradictory","repo_kind":"official","path":"auto_detection/evaluate_ir_o.py","file_url":"https://github.com/uscnlp-lime/Self-Contradictory/blob/HEAD/auto_detection/evaluate_ir_o.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"387ea733adfb813a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}