{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/codemind-a-framework-to-challenge-large","title":"CodeMind: Evaluating Large Language Models for Code Reasoning","arxiv_id":"2402.09664","date":"2024-02-15","proceeding":null,"authors":["Changshu Liu","Yang Chen","Reyhaneh Jabbarvand"],"abstract":"Large Language Models (LLMs) have been widely used to automate programming tasks. Their capabilities have been evaluated by assessing the quality of generated code through tests or proofs. The extent to which they can reason about code is a critical question revealing important insights about their true capabilities. This paper introduces CodeMind, a framework designed to gauge the code reasoning abilities of LLMs through the following explicit and implicit code reasoning tasks: Independent Execution Reasoning (IER), Specification Reasoning (SR) and Dynamic Semantics Reasoning (DSR). The first evaluates the abilities of LLMs to simulate the execution of given inputs to a code and predict the output (IER). The second assesses the abilities of LLMs to incorporate the simulation of test data in the specification into code generation (SR). Finally, CodeMind evaluates LLMs' abilities to understand overall code semantics only given a specific input/output (DSR). Our extensive evaluation of ten LLMs across four widely used benchmarks using CodeMind shows that LLMs, depending on their size and training strategy, can reason about some dynamic aspects of code. However, their performance drops for code with higher complexity, non-trivial logical and arithmetic operators, non-primitive types, and API calls. We show that these reasoning tasks evaluate LLMs differently, and a comprehensive evaluation of code reasoning requires them all. Finally, we show that the performance of LLMs in bug repair is not correlated with any of the code reasoning tasks, and except for advanced frontier models, other LLMs do not incorporate code reasoning when performing bug repair.","url_abs":"https://arxiv.org/abs/2402.09664v5","url_pdf":"https://arxiv.org/pdf/2402.09664v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"codemind-a-framework-to-challenge-large","repo_url":"https://github.com/intelligent-cat-lab/codemind","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NCSA"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.09664","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.09664"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/intelligent-cat-lab/codemind","reach":{"status":"ok","spdx":"NCSA"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"fc0f16f3d89d8f5d","entry":"calculate_average","repo":"intelligent-cat-lab/codemind","repo_kind":"official","path":"dataset/classeval/ClassEval_4@AssessmentSystem.add_course_score/transformation.py","file_url":"https://github.com/intelligent-cat-lab/codemind/blob/HEAD/dataset/classeval/ClassEval_4%40AssessmentSystem.add_course_score/transformation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NCSA","inline_ok":false,"mcp_get_code":{"code_sha256":"fc0f16f3d89d8f5d"}},{"code_sha256_prefix":"7005b45caa50b550","entry":"my_decorator","repo":"intelligent-cat-lab/codemind","repo_kind":"official","path":"dataset/classeval/ClassEval_4@AssessmentSystem.add_course_score/transformation.py","file_url":"https://github.com/intelligent-cat-lab/codemind/blob/HEAD/dataset/classeval/ClassEval_4%40AssessmentSystem.add_course_score/transformation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NCSA","inline_ok":false,"mcp_get_code":{"code_sha256":"7005b45caa50b550"}},{"code_sha256_prefix":"8f511fb90293f74f","entry":"my_decorator","repo":"intelligent-cat-lab/codemind","repo_kind":"official","path":"dataset/classeval/ClassEval_56@MetricsCalculator.f1_score/transformation.py","file_url":"https://github.com/intelligent-cat-lab/codemind/blob/HEAD/dataset/classeval/ClassEval_56%40MetricsCalculator.f1_score/transformation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NCSA","inline_ok":false,"mcp_get_code":{"code_sha256":"8f511fb90293f74f"}},{"code_sha256_prefix":"f786939c6d16793f","entry":"calculate_total_predictions","repo":"intelligent-cat-lab/codemind","repo_kind":"official","path":"dataset/classeval/ClassEval_56@MetricsCalculator.f1_score/transformation.py","file_url":"https://github.com/intelligent-cat-lab/codemind/blob/HEAD/dataset/classeval/ClassEval_56%40MetricsCalculator.f1_score/transformation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NCSA","inline_ok":false,"mcp_get_code":{"code_sha256":"f786939c6d16793f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}