{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/finereason-evaluating-and-improving-llms","title":"FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving","arxiv_id":"2502.20238","date":"2025-02-27","proceeding":null,"authors":["Guizhen Chen","Weiwen Xu","Hao Zhang","Hou Pong Chan","Chaoqun Liu","Lidong Bing","Deli Zhao","Anh Tuan Luu","Yu Rong"],"abstract":"Many challenging reasoning tasks require not just rapid, intuitive responses, but a more deliberate, multi-step approach. Recent progress in large language models (LLMs) highlights an important shift from the \"System 1\" way of quick reactions to the \"System 2\" style of reflection-and-correction problem solving. However, current benchmarks heavily rely on the final-answer accuracy, leaving much of a model's intermediate reasoning steps unexamined. This fails to assess the model's ability to reflect and rectify mistakes within the reasoning process. To bridge this gap, we introduce FINEREASON, a logic-puzzle benchmark for fine-grained evaluation of LLMs' reasoning capabilities. Each puzzle can be decomposed into atomic steps, making it ideal for rigorous validation of intermediate correctness. Building on this, we introduce two tasks: state checking, and state transition, for a comprehensive evaluation of how models assess the current situation and plan the next move. To support broader research, we also provide a puzzle training set aimed at enhancing performance on general mathematical tasks. We show that models trained on our state checking and transition data demonstrate gains in math reasoning by up to 5.1% on GSM8K.","url_abs":"https://arxiv.org/abs/2502.20238v1","url_pdf":"https://arxiv.org/pdf/2502.20238v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"finereason-evaluating-and-improving-llms","repo_url":"https://github.com/DAMO-NLP-SG/FineReason","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"math","task_name":"Math"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.20238","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.20238"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/DAMO-NLP-SG/FineReason","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"187f0ed34850b94d","entry":"get_column_index","repo":"DAMO-NLP-SG/FineReason","repo_kind":"official","path":"gridpuzzle_tree.py","file_url":"https://github.com/DAMO-NLP-SG/FineReason/blob/HEAD/gridpuzzle_tree.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"187f0ed34850b94d"}},{"code_sha256_prefix":"1cc67da174449279","entry":"get_row_index","repo":"DAMO-NLP-SG/FineReason","repo_kind":"official","path":"gridpuzzle_tree.py","file_url":"https://github.com/DAMO-NLP-SG/FineReason/blob/HEAD/gridpuzzle_tree.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"1cc67da174449279"}},{"code_sha256_prefix":"6d77004440e1ceac","entry":"select_data","repo":"DAMO-NLP-SG/FineReason","repo_kind":"official","path":"data_loading.py","file_url":"https://github.com/DAMO-NLP-SG/FineReason/blob/HEAD/data_loading.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"6d77004440e1ceac"}},{"code_sha256_prefix":"cdeb864dfd1da49f","entry":"transform_condition","repo":"DAMO-NLP-SG/FineReason","repo_kind":"official","path":"gridpuzzle_tree.py","file_url":"https://github.com/DAMO-NLP-SG/FineReason/blob/HEAD/gridpuzzle_tree.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"cdeb864dfd1da49f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}