{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/are-your-llms-capable-of-stable-reasoning","title":"Are Your LLMs Capable of Stable Reasoning?","arxiv_id":"2412.13147","date":"2024-12-17","proceeding":null,"authors":["Junnan Liu","Hongwei Liu","Linchen Xiao","Ziyi Wang","Kuikun Liu","Songyang Gao","Wenwei Zhang","Songyang Zhang","Kai Chen"],"abstract":"The rapid advancement of Large Language Models (LLMs) has demonstrated remarkable progress in complex reasoning tasks. However, a significant discrepancy persists between benchmark performances and real-world applications. We identify this gap as primarily stemming from current evaluation protocols and metrics, which inadequately capture the full spectrum of LLM capabilities, particularly in complex reasoning tasks where both accuracy and consistency are crucial. This work makes two key contributions. First, we introduce G-Pass@k, a novel evaluation metric that provides a continuous assessment of model performance across multiple sampling attempts, quantifying both the model's peak performance potential and its stability. Second, we present LiveMathBench, a dynamic benchmark comprising challenging, contemporary mathematical problems designed to minimize data leakage risks during evaluation. Through extensive experiments using G-Pass@k on state-of-the-art LLMs with LiveMathBench, we provide comprehensive insights into both their maximum capabilities and operational consistency. Our findings reveal substantial room for improvement in LLMs' \"realistic\" reasoning capabilities, highlighting the need for more robust evaluation methods. The benchmark and detailed results are available at: https://github.com/open-compass/GPassK.","url_abs":"https://arxiv.org/abs/2412.13147v3","url_pdf":"https://arxiv.org/pdf/2412.13147v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"are-your-llms-capable-of-stable-reasoning","repo_url":"https://github.com/open-compass/gpassk","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.13147","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.13147"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/open-compass/gpassk","reach":{"status":"ok"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"3f5a4145bbfd9c9d","entry":"compute_g_pass_at_k","repo":"open-compass/gpassk","repo_kind":"official","path":"lighteval/src/lighteval_metric.py","file_url":"https://github.com/open-compass/gpassk/blob/HEAD/lighteval/src/lighteval_metric.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f5a4145bbfd9c9d"}},{"code_sha256_prefix":"461e43583194c67f","entry":"compute_mg_pass_at_k","repo":"open-compass/gpassk","repo_kind":"official","path":"lighteval/src/lighteval_metric.py","file_url":"https://github.com/open-compass/gpassk/blob/HEAD/lighteval/src/lighteval_metric.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"461e43583194c67f"}},{"code_sha256_prefix":"d2a746e088a94b53","entry":"compute_pass_at_k","repo":"open-compass/gpassk","repo_kind":"official","path":"lighteval/src/lighteval_metric.py","file_url":"https://github.com/open-compass/gpassk/blob/HEAD/lighteval/src/lighteval_metric.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2a746e088a94b53"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}