{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-08455","title":"CUDABeaver: Benchmarking LLM-Based Automated CUDA Debugging","arxiv_id":"2605.08455","date":"2026-05-08","proceeding":null,"authors":["Shiyang Li","Haoyang Chen","Mattia Fazzini","Caiwen Ding"],"abstract":"Debugging CUDA programs has long been challenging because failures often arise from subtle interactions among hardware behavior, compiler decisions, memory hierarchy, and asynchronous execution. More importantly, with the rapid expansion of GPU usage across scientific computing, machine learning, graphics, and systems workloads, CUDA debugging has become more challenging than ever. Current evaluations of LLM-based CUDA programming largely miss this setting: a model can pass correctness tests with repair by degeneration, simplifying the CUDA code into a safer but slower program that abandons the original optimization structure. We introduce CUDABEAVER, a benchmark for CUDA debugging from real failing workspaces produced during LLM-based CUDA generation. Each task provides the broken candidate, native build/test commands, raw error evidence, and a single editable file. CUDABEAVER evaluates whether a fixer truly repairs the failing CUDA code or merely finds a slower test-passing replacement, reporting results by failure category, debugging trajectory, stagnation mode, and performance preservation. We further propose pass@k(M,C,A), a protocol-conditional CUDA debugging metric by making the fixer M, corpus C, and protocol axes Aexplicit. Using this metric across 213 tasks and seven frontier LLMs, we show that protocol-aware evaluation gives a more faithful view of CUDA debugging ability: when performance-loss tolerance is high, fixers appear much stronger, but even a minor stricter performance requirement can sharply reduce measured success, shifting scores by up to 40 percentage points.","url_abs":"https://arxiv.org/abs/2605.08455","url_pdf":"https://arxiv.org/pdf/2605.08455","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.08455","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.08455"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/HaoyangChen23/CUDABeaver","reach":null}],"summary":{"ran":13,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":15,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"d2b695569531f7f1","entry":"applied_kernels_build","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"harness/applied_kernels_eval.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/harness/applied_kernels_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d2b695569531f7f1"}},{"code_sha256_prefix":"d36caf49c46ee297","entry":"classify_result","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"harness/classifier.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/harness/classifier.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d36caf49c46ee297"}},{"code_sha256_prefix":"b6a7fd5c26eb65ed","entry":"compare_baselines","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"harness/analyze_pipeline.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/harness/analyze_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b6a7fd5c26eb65ed"}},{"code_sha256_prefix":"7d801489557079f3","entry":"compute_pooled_metrics","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"analysis/pool.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/analysis/pool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7d801489557079f3"}},{"code_sha256_prefix":"9eb2a8de6eddd631","entry":"extract_per_task_status","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"analysis/pool.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/analysis/pool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9eb2a8de6eddd631"}},{"code_sha256_prefix":"4e306a5173ace8b9","entry":"is_applied_kernels_task","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"harness/applied_kernels_eval.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/harness/applied_kernels_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4e306a5173ace8b9"}},{"code_sha256_prefix":"fa567bfe1fe23a63","entry":"load_committee_data","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"analysis/p_sweep.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/analysis/p_sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fa567bfe1fe23a63"}},{"code_sha256_prefix":"f265eefafc9c09e8","entry":"normalize_axis","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"analysis/axis_normalize.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/analysis/axis_normalize.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f265eefafc9c09e8"}},{"code_sha256_prefix":"87866fd72c58a184","entry":"parse_experiment","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"analysis/axis_normalize.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/analysis/axis_normalize.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"87866fd72c58a184"}},{"code_sha256_prefix":"c4d4ee0968f50a68","entry":"pass_at_5","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"analysis/p_sweep.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/analysis/p_sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c4d4ee0968f50a68"}},{"code_sha256_prefix":"63ec501f4ea97483","entry":"render_table","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"analysis/render_latex.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/analysis/render_latex.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"63ec501f4ea97483"}},{"code_sha256_prefix":"be0421e44019a9f1","entry":"scan_manifests","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"analysis/scanner.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/analysis/scanner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"be0421e44019a9f1"}},{"code_sha256_prefix":"82fb732bddd9d41a","entry":"task_id_to_stem","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"harness/benchmark.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/harness/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"82fb732bddd9d41a"}},{"code_sha256_prefix":"af652593943d9d07","entry":"applied_kernels_test","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"harness/applied_kernels_eval.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/harness/applied_kernels_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"af652593943d9d07"}},{"code_sha256_prefix":"7ac0050f42ef99ee","entry":"classify_failures","repo":"HaoyangChen23/CUDABeaver","repo_kind":"found_in_text","path":"harness/analyze_pipeline.py","file_url":"https://github.com/HaoyangChen23/CUDABeaver/blob/HEAD/harness/analyze_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7ac0050f42ef99ee"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}