{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/order-matters-in-hallucination-reasoning","title":"Order Matters in Hallucination: Reasoning Order as Benchmark and Reflexive Prompting for Large-Language-Models","arxiv_id":"2408.05093","date":"2024-08-09","proceeding":null,"authors":["Zikai Xie"],"abstract":"Large language models (LLMs) have generated significant attention since their inception, finding applications across various academic and industrial domains. However, these models often suffer from the \"hallucination problem\", where outputs, though grammatically and logically coherent, lack factual accuracy or are entirely fabricated. A particularly troubling issue discovered and widely discussed recently is the numerical comparison error where multiple LLMs incorrectly infer that \"9.11$>$9.9\". We discovered that the order in which LLMs generate answers and reasoning impacts their consistency. Specifically, results vary significantly when an LLM generates an answer first and then provides the reasoning versus generating the reasoning process first and then the conclusion. Inspired by this, we propose a new benchmark method for assessing LLM consistency: comparing responses generated through these two different approaches. This benchmark effectively identifies instances where LLMs fabricate answers and subsequently generate justifications. Furthermore, we introduce a novel and straightforward prompt strategy designed to mitigate this issue. Experimental results demonstrate that this strategy improves performance across various LLMs compared to direct questioning. This work not only sheds light on a critical flaw in LLMs but also offers a practical solution to enhance their reliability.","url_abs":"https://arxiv.org/abs/2408.05093v4","url_pdf":"https://arxiv.org/pdf/2408.05093v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"order-matters-in-hallucination-reasoning","repo_url":"https://github.com/xiezikai/reflexiveprompting","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"hallucination","task_name":"Hallucination"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.05093","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.05093"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xiezikai/reflexiveprompting","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":11,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a52bcb839042fe4e","entry":"generate_prompt","repo":"xiezikai/reflexiveprompting","repo_kind":"official","path":"self_contradictory_comparison.py","file_url":"https://github.com/xiezikai/reflexiveprompting/blob/HEAD/self_contradictory_comparison.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a52bcb839042fe4e"}},{"code_sha256_prefix":"4458b91a29c0bff8","entry":"generate_prompt_bigbench","repo":"xiezikai/reflexiveprompting","repo_kind":"official","path":"utils_bigbench.py","file_url":"https://github.com/xiezikai/reflexiveprompting/blob/HEAD/utils_bigbench.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4458b91a29c0bff8"}},{"code_sha256_prefix":"93e63c76186a7749","entry":"generate_prompt_logiqa","repo":"xiezikai/reflexiveprompting","repo_kind":"official","path":"utils_logiqa.py","file_url":"https://github.com/xiezikai/reflexiveprompting/blob/HEAD/utils_logiqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93e63c76186a7749"}},{"code_sha256_prefix":"5fa2dca6449f4d11","entry":"generate_prompt_mmlu","repo":"xiezikai/reflexiveprompting","repo_kind":"official","path":"utils_mmlu.py","file_url":"https://github.com/xiezikai/reflexiveprompting/blob/HEAD/utils_mmlu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5fa2dca6449f4d11"}},{"code_sha256_prefix":"0164b085d9b5b165","entry":"generate_prompt_truthfulqa","repo":"xiezikai/reflexiveprompting","repo_kind":"official","path":"utils_truthfulqa.py","file_url":"https://github.com/xiezikai/reflexiveprompting/blob/HEAD/utils_truthfulqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0164b085d9b5b165"}},{"code_sha256_prefix":"404b38fae85eae6b","entry":"generate_review_prompt_bigbench","repo":"xiezikai/reflexiveprompting","repo_kind":"official","path":"utils_bigbench.py","file_url":"https://github.com/xiezikai/reflexiveprompting/blob/HEAD/utils_bigbench.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"404b38fae85eae6b"}},{"code_sha256_prefix":"e279c0cd745747e9","entry":"generate_review_prompt_logiqa","repo":"xiezikai/reflexiveprompting","repo_kind":"official","path":"utils_logiqa.py","file_url":"https://github.com/xiezikai/reflexiveprompting/blob/HEAD/utils_logiqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e279c0cd745747e9"}},{"code_sha256_prefix":"0a1438ac13b9c397","entry":"generate_review_prompt_truthfulqa","repo":"xiezikai/reflexiveprompting","repo_kind":"official","path":"utils_mmlu.py","file_url":"https://github.com/xiezikai/reflexiveprompting/blob/HEAD/utils_mmlu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a1438ac13b9c397"}},{"code_sha256_prefix":"c89f00b344150505","entry":"get_reasoning_data_bigbench","repo":"xiezikai/reflexiveprompting","repo_kind":"official","path":"utils_bigbench.py","file_url":"https://github.com/xiezikai/reflexiveprompting/blob/HEAD/utils_bigbench.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c89f00b344150505"}},{"code_sha256_prefix":"8baf7ca3db1e6bbd","entry":"read_txt_file","repo":"xiezikai/reflexiveprompting","repo_kind":"official","path":"self_contradictory_comparison.py","file_url":"https://github.com/xiezikai/reflexiveprompting/blob/HEAD/self_contradictory_comparison.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8baf7ca3db1e6bbd"}},{"code_sha256_prefix":"9a661f243d6318b2","entry":"result_correction","repo":"xiezikai/reflexiveprompting","repo_kind":"official","path":"utils.py","file_url":"https://github.com/xiezikai/reflexiveprompting/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a661f243d6318b2"}},{"code_sha256_prefix":"8661f4d51670c645","entry":"get_reasoning_data_logiqa","repo":"xiezikai/reflexiveprompting","repo_kind":"official","path":"utils_logiqa.py","file_url":"https://github.com/xiezikai/reflexiveprompting/blob/HEAD/utils_logiqa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8661f4d51670c645"}},{"code_sha256_prefix":"1e6dd951c6a8a447","entry":"get_reasoning_data_mmlu","repo":"xiezikai/reflexiveprompting","repo_kind":"official","path":"utils_mmlu.py","file_url":"https://github.com/xiezikai/reflexiveprompting/blob/HEAD/utils_mmlu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e6dd951c6a8a447"}},{"code_sha256_prefix":"41bbba1590f51850","entry":"get_reasoning_data_truthfulqa","repo":"xiezikai/reflexiveprompting","repo_kind":"official","path":"utils_truthfulqa.py","file_url":"https://github.com/xiezikai/reflexiveprompting/blob/HEAD/utils_truthfulqa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41bbba1590f51850"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}