{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/debugbench-evaluating-debugging-capability-of","title":"DebugBench: Evaluating Debugging Capability of Large Language Models","arxiv_id":"2401.04621","date":"2024-01-09","proceeding":null,"authors":["Runchu Tian","Yining Ye","Yujia Qin","Xin Cong","Yankai Lin","Yinxu Pan","Yesai Wu","Haotian Hui","Weichuan Liu","Zhiyuan Liu","Maosong Sun"],"abstract":"Large Language Models (LLMs) have demonstrated exceptional coding capability. However, as another critical component of programming proficiency, the debugging capability of LLMs remains relatively unexplored. Previous evaluations of LLMs' debugging ability are significantly limited by the risk of data leakage, the scale of the dataset, and the variety of tested bugs. To overcome these deficiencies, we introduce `DebugBench', an LLM debugging benchmark consisting of 4,253 instances. It covers four major bug categories and 18 minor types in C++, Java, and Python. To construct DebugBench, we collect code snippets from the LeetCode community, implant bugs into source data with GPT-4, and assure rigorous quality checks. We evaluate two commercial and four open-source models in a zero-shot scenario. We find that (1) while closed-source models exhibit inferior debugging performance compared to humans, open-source models relatively lower pass rate scores; (2) the complexity of debugging notably fluctuates depending on the bug category; (3) incorporating runtime feedback has a clear impact on debugging performance which is not always helpful. As an extension, we also compare LLM debugging and code generation, revealing a strong correlation between them for closed-source models. These findings will benefit the development of LLMs in debugging.","url_abs":"https://arxiv.org/abs/2401.04621v3","url_pdf":"https://arxiv.org/pdf/2401.04621v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"debugbench-evaluating-debugging-capability-of","repo_url":"https://github.com/thunlp/debugbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.04621","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.04621"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thunlp/debugbench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"unverified":4},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"84fd42bee7fa9feb","entry":"clean_code","repo":"thunlp/debugbench","repo_kind":"official","path":"dataset_construction/clean_comments.py","file_url":"https://github.com/thunlp/debugbench/blob/HEAD/dataset_construction/clean_comments.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"84fd42bee7fa9feb"}},{"code_sha256_prefix":"39624643acd840a9","entry":"clean_data","repo":"thunlp/debugbench","repo_kind":"official","path":"dataset_construction/clean_comments.py","file_url":"https://github.com/thunlp/debugbench/blob/HEAD/dataset_construction/clean_comments.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"39624643acd840a9"}},{"code_sha256_prefix":"4adff2a4043ada95","entry":"clean_datum","repo":"thunlp/debugbench","repo_kind":"official","path":"dataset_construction/clean_comments.py","file_url":"https://github.com/thunlp/debugbench/blob/HEAD/dataset_construction/clean_comments.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4adff2a4043ada95"}},{"code_sha256_prefix":"aeccb4b8e6ad5100","entry":"merge","repo":"thunlp/debugbench","repo_kind":"official","path":"dataset_construction/generate_multiples.py","file_url":"https://github.com/thunlp/debugbench/blob/HEAD/dataset_construction/generate_multiples.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aeccb4b8e6ad5100"}},{"code_sha256_prefix":"5306e87da44a3e47","entry":"load_bug_data","repo":"thunlp/debugbench","repo_kind":"official","path":"evaluation/debug_with_traceback.py","file_url":"https://github.com/thunlp/debugbench/blob/HEAD/evaluation/debug_with_traceback.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5306e87da44a3e47"}},{"code_sha256_prefix":"f55f7f3e0fc56c6c","entry":"load_bugs","repo":"thunlp/debugbench","repo_kind":"official","path":"dataset_construction/generate_multiples.py","file_url":"https://github.com/thunlp/debugbench/blob/HEAD/dataset_construction/generate_multiples.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f55f7f3e0fc56c6c"}},{"code_sha256_prefix":"140a272ea2caef78","entry":"load_cases","repo":"thunlp/debugbench","repo_kind":"official","path":"dataset_construction/generate_multiples.py","file_url":"https://github.com/thunlp/debugbench/blob/HEAD/dataset_construction/generate_multiples.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"140a272ea2caef78"}},{"code_sha256_prefix":"8a7b54cf614e7e35","entry":"load_cases","repo":"thunlp/debugbench","repo_kind":"official","path":"dataset_construction/generate_singles.py","file_url":"https://github.com/thunlp/debugbench/blob/HEAD/dataset_construction/generate_singles.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a7b54cf614e7e35"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}