{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-21867","title":"ForEx: A Formal Verification Framework for Explainable Reasoning in Logical Fallacy Detection and Annotation","arxiv_id":"2606.21867","date":"2026-06-20","proceeding":null,"authors":["Pei-Cing Huang","Chienyu Liu","Chan Hsu","Ci-Siang Chen","Pei-Ju Lee","Yihuang Kang"],"abstract":"Current evaluations of Large Language Models (LLMs) on logical fallacy detection focus on predicted labels, but do not establish whether those labels are supported by the reasoning the models provide. We propose ForEx (Formal Verification for Explainable Reasoning), a framework that translates LLM-generated explanations into Lean4 and verifies whether the translated rationale is derivable under encoded premises, not the logical validity of the original natural language argument. To distinguish prediction outcomes from the formal status of the supporting reasoning, we introduce the LLM Argument Verification Matrix, which separates label consistency from formal verification status. Experiments on LOGIC-Climate show that over 90% of LLM outputs can be translated into formal reasoning chains that pass verification, while agreement with human annotations remains around 20%. These results expose a systematic gap between formal derivability and label agreement, a distinction invisible to prediction-based metrics. ForEx moves LLM evaluation beyond label correctness toward machine-checkable analysis of formalized reasoning chains.","url_abs":"https://arxiv.org/abs/2606.21867","url_pdf":"https://arxiv.org/pdf/2606.21867","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2606.21867","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.21867"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/D3-Laboratory/ForEx","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"138c05764eb27520","entry":"verify_lean_code_via_api","repo":"D3-Laboratory/ForEx","repo_kind":"found_in_text","path":"src/experiment_processor.py","file_url":"https://github.com/D3-Laboratory/ForEx/blob/HEAD/src/experiment_processor.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"138c05764eb27520"}},{"code_sha256_prefix":"bfdc665f09df75ca","entry":"_process_single_option","repo":"D3-Laboratory/ForEx","repo_kind":"found_in_text","path":"src/experiment_processor.py","file_url":"https://github.com/D3-Laboratory/ForEx/blob/HEAD/src/experiment_processor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bfdc665f09df75ca"}},{"code_sha256_prefix":"65b26a0acc1b2673","entry":"get_llm_response","repo":"D3-Laboratory/ForEx","repo_kind":"found_in_text","path":"src/experiment_processor.py","file_url":"https://github.com/D3-Laboratory/ForEx/blob/HEAD/src/experiment_processor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"65b26a0acc1b2673"}},{"code_sha256_prefix":"ad5b86651ffa4d6f","entry":"process_question","repo":"D3-Laboratory/ForEx","repo_kind":"found_in_text","path":"src/experiment_processor.py","file_url":"https://github.com/D3-Laboratory/ForEx/blob/HEAD/src/experiment_processor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad5b86651ffa4d6f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}