{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/semcoder-training-code-language-models-with","title":"SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning","arxiv_id":"2406.01006","date":"2024-06-03","proceeding":null,"authors":["Yangruibo Ding","Jinjun Peng","Marcus J. Min","Gail Kaiser","Junfeng Yang","Baishakhi Ray"],"abstract":"Code Large Language Models (Code LLMs) have excelled at tasks like code completion but often miss deeper semantics such as execution effects and dynamic states. This paper aims to bridge the gap between Code LLMs' reliance on static text data and the need for semantic understanding for complex tasks like debugging and program repair. We introduce a novel strategy, monologue reasoning, to train Code LLMs to reason comprehensive semantics, encompassing high-level functional descriptions, local execution effects of individual statements, and overall input/output behavior, thereby linking static code text with dynamic execution states. We begin by collecting PyX, a clean Python corpus of fully executable code samples with functional descriptions and test cases. We propose training Code LLMs not only to write code but also to understand code semantics by reasoning about key properties, constraints, and execution behaviors using natural language, mimicking human verbal debugging, i.e., rubber-duck debugging. This approach led to the development of SemCoder, a Code LLM with only 6.7B parameters, which shows competitive performance with GPT-3.5-turbo on code generation and execution reasoning tasks. SemCoder achieves 79.3% on HumanEval (GPT-3.5-turbo: 76.8%), 63.6% on CRUXEval-I (GPT-3.5-turbo: 50.3%), and 63.9% on CRUXEval-O (GPT-3.5-turbo: 59.0%). We also study the effectiveness of SemCoder's monologue-style execution reasoning compared to concrete scratchpad reasoning, showing that our approach integrates semantics from multiple dimensions more smoothly. Finally, we demonstrate the potential of applying learned semantics to improve Code LLMs' debugging and self-refining capabilities. Our data, code, and models are available at: https://github.com/ARiSE-Lab/SemCoder.","url_abs":"https://arxiv.org/abs/2406.01006v2","url_pdf":"https://arxiv.org/pdf/2406.01006v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"semcoder-training-code-language-models-with","repo_url":"https://github.com/arise-lab/semcoder","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"code-completion","task_name":"Code Completion"},{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"humaneval","task_name":"HumanEval"},{"task_slug":"program-repair","task_name":"Program Repair"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.01006","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.01006"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ARiSE-Lab/SemCoder","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/arise-lab/semcoder","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"ran_draft_wrong":2,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":15,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"23ef3078d45c966b","entry":"attach_execution_exception","repo":"ARiSE-Lab/SemCoder","repo_kind":"official","path":"experiments/build_self_refine_input.py","file_url":"https://github.com/ARiSE-Lab/SemCoder/blob/HEAD/experiments/build_self_refine_input.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"23ef3078d45c966b"}},{"code_sha256_prefix":"8fbbf1d9e53be9ec","entry":"chunked","repo":"ARiSE-Lab/SemCoder","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/ARiSE-Lab/SemCoder/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8fbbf1d9e53be9ec"}},{"code_sha256_prefix":"b9090aa3af341871","entry":"complete_code","repo":"ARiSE-Lab/SemCoder","repo_kind":"official","path":"experiments/cruxeval_utils.py","file_url":"https://github.com/ARiSE-Lab/SemCoder/blob/HEAD/experiments/cruxeval_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b9090aa3af341871"}},{"code_sha256_prefix":"7e638715779072af","entry":"concise_spad","repo":"arise-lab/semcoder","repo_kind":"official","path":"experiments/construct_trace.py","file_url":"https://github.com/arise-lab/semcoder/blob/HEAD/experiments/construct_trace.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e638715779072af"}},{"code_sha256_prefix":"5c7a36496bbe50b3","entry":"convertv2s","repo":"arise-lab/semcoder","repo_kind":"official","path":"experiments/construct_trace.py","file_url":"https://github.com/arise-lab/semcoder/blob/HEAD/experiments/construct_trace.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c7a36496bbe50b3"}},{"code_sha256_prefix":"660d74f09a475daa","entry":"extract_failed_test","repo":"ARiSE-Lab/SemCoder","repo_kind":"official","path":"experiments/build_self_refine_input.py","file_url":"https://github.com/ARiSE-Lab/SemCoder/blob/HEAD/experiments/build_self_refine_input.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"660d74f09a475daa"}},{"code_sha256_prefix":"7e07a365b5619a81","entry":"extract_trace","repo":"ARiSE-Lab/SemCoder","repo_kind":"official","path":"experiments/build_self_refine_input.py","file_url":"https://github.com/ARiSE-Lab/SemCoder/blob/HEAD/experiments/build_self_refine_input.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e07a365b5619a81"}},{"code_sha256_prefix":"6834377c553301df","entry":"handle_value","repo":"arise-lab/semcoder","repo_kind":"official","path":"experiments/construct_trace.py","file_url":"https://github.com/arise-lab/semcoder/blob/HEAD/experiments/construct_trace.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6834377c553301df"}},{"code_sha256_prefix":"b4a6d7a477e69976","entry":"make_cot_output_prompt","repo":"ARiSE-Lab/SemCoder","repo_kind":"official","path":"experiments/cruxeval_prompts.py","file_url":"https://github.com/ARiSE-Lab/SemCoder/blob/HEAD/experiments/cruxeval_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b4a6d7a477e69976"}},{"code_sha256_prefix":"7ff4e308c6863412","entry":"make_direct_output_prompt","repo":"ARiSE-Lab/SemCoder","repo_kind":"official","path":"experiments/cruxeval_prompts.py","file_url":"https://github.com/ARiSE-Lab/SemCoder/blob/HEAD/experiments/cruxeval_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ff4e308c6863412"}},{"code_sha256_prefix":"1c29bebc3b2830c3","entry":"make_forward_monologue_output_prompt","repo":"ARiSE-Lab/SemCoder","repo_kind":"official","path":"experiments/cruxeval_prompts.py","file_url":"https://github.com/ARiSE-Lab/SemCoder/blob/HEAD/experiments/cruxeval_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c29bebc3b2830c3"}},{"code_sha256_prefix":"926bd7fef63dd64e","entry":"attach_trace_to_code","repo":"arise-lab/semcoder","repo_kind":"official","path":"experiments/construct_trace.py","file_url":"https://github.com/arise-lab/semcoder/blob/HEAD/experiments/construct_trace.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"926bd7fef63dd64e"}},{"code_sha256_prefix":"da99ae3e6a8eb89e","entry":"get_model_context","repo":"ARiSE-Lab/SemCoder","repo_kind":"official","path":"src/wrapper.py","file_url":"https://github.com/ARiSE-Lab/SemCoder/blob/HEAD/src/wrapper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da99ae3e6a8eb89e"}},{"code_sha256_prefix":"0ea1d7ce8d9d3e2a","entry":"get_task","repo":"ARiSE-Lab/SemCoder","repo_kind":"official","path":"experiments/cruxeval_utils.py","file_url":"https://github.com/ARiSE-Lab/SemCoder/blob/HEAD/experiments/cruxeval_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ea1d7ce8d9d3e2a"}},{"code_sha256_prefix":"7b178c461482e3f6","entry":"pad_sequences","repo":"ARiSE-Lab/SemCoder","repo_kind":"official","path":"src/wrapper.py","file_url":"https://github.com/ARiSE-Lab/SemCoder/blob/HEAD/src/wrapper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b178c461482e3f6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}