{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/codeassistbench-cab-dataset-benchmarking-for","title":"CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistance","arxiv_id":"2507.10646","date":"2025-07-14","proceeding":null,"authors":["Myeongsoo Kim","Shweta Garg","Baishakhi Ray","Varun Kumar","Anoop Deoras"],"abstract":"Programming assistants powered by large language models have transformed software development, yet most benchmarks focus narrowly on code generation tasks. Recent efforts like InfiBench and StackEval attempt to address this gap using Stack Overflow data but remain limited to single-turn interactions in isolated contexts, require significant manual curation, and fail to represent complete project environments. We introduce CodeAssistBench (CAB), the first benchmark framework for evaluating multi-turn programming assistance in realistic settings that address real-world questions about actual codebases. Unlike existing programming Q&A benchmarks, CAB automatically generates scalable datasets from question-related GitHub issues using configurable parameters (e.g., repository creation date, star count, programming languages), and includes automatic containerization of codebases for evaluation. It then evaluates models through simulated users in these containerized environments with full codebase access. Using this framework, we constructed a test set of 3,286 real-world programming questions across 231 repositories, spanning seven programming languages and diverse problem domains. Our evaluation of leading LLMs reveals a substantial capability gap: while models perform well on Stack Overflow questions with success rates of 70-83%, they resolve only up to 16.49% of CAB's recent issues. This discrepancy highlights the challenges of providing assistance in complex, project-specific contexts versus answering standalone questions.","url_abs":"https://arxiv.org/abs/2507.10646v2","url_pdf":"https://arxiv.org/pdf/2507.10646v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"code-generation","task_name":"Code Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2507.10646","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2507.10646"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/amazon-science/CodeAssistBench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":24},"by_repo_kind":{"found_in_text":{"samples":24,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c765c446ca7e2dac","entry":"add_task_ids_to_dataset","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/add_task_id.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/add_task_id.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c765c446ca7e2dac"}},{"code_sha256_prefix":"dc66001eb75b6b17","entry":"call_llm_with_adaptive_context","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/generate_dockerfile.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/generate_dockerfile.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dc66001eb75b6b17"}},{"code_sha256_prefix":"e39cf6d4f8748818","entry":"classify_issue","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/docker_filter.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/docker_filter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e39cf6d4f8748818"}},{"code_sha256_prefix":"d86db1a58b7c185c","entry":"contains_media","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/collect_1000_issues.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/collect_1000_issues.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d86db1a58b7c185c"}},{"code_sha256_prefix":"7804bcc5b694e016","entry":"contains_media","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/get_github_issue.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/get_github_issue.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7804bcc5b694e016"}},{"code_sha256_prefix":"404dd6e9ec43f905","entry":"convert_to_jsonl","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/convert_to_jsonl.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/convert_to_jsonl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"404dd6e9ec43f905"}},{"code_sha256_prefix":"47526b086dc31647","entry":"evaluate_issue","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/conv_filter.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/conv_filter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"47526b086dc31647"}},{"code_sha256_prefix":"dbf897955d48ae15","entry":"extract_repo_name_from_url","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/generate_dockerfile_with_strands.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/generate_dockerfile_with_strands.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dbf897955d48ae15"}},{"code_sha256_prefix":"890c6f2df7f0bc62","entry":"generate_failure_explanation","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/generate_dockerfile.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/generate_dockerfile.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"890c6f2df7f0bc62"}},{"code_sha256_prefix":"3ab69c08296acaf6","entry":"generate_response","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/docker_filter.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/docker_filter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ab69c08296acaf6"}},{"code_sha256_prefix":"aae2d43cdb39625c","entry":"get_commit_before_date","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/merge_commit_ids.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/merge_commit_ids.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aae2d43cdb39625c"}},{"code_sha256_prefix":"af0f9bf1386e13d8","entry":"get_first_commit_date","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/filter_by_commit_date.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/filter_by_commit_date.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"af0f9bf1386e13d8"}},{"code_sha256_prefix":"27882b8a9c5e9151","entry":"get_issues","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/get_github_issue.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/get_github_issue.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"27882b8a9c5e9151"}},{"code_sha256_prefix":"749e16b560b7b6f0","entry":"get_repo_license","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/collect_1000_issues.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/collect_1000_issues.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"749e16b560b7b6f0"}},{"code_sha256_prefix":"9009ab6e87dcd0f1","entry":"load_commits_for_repo","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/merge_commit_ids.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/merge_commit_ids.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9009ab6e87dcd0f1"}},{"code_sha256_prefix":"8f0e8db8999c8578","entry":"parse_github_url","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/get_github_issue.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/get_github_issue.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8f0e8db8999c8578"}},{"code_sha256_prefix":"5c1d0e3f4624c9bd","entry":"process_issue_file","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/merge_commit_ids.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/merge_commit_ids.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c1d0e3f4624c9bd"}},{"code_sha256_prefix":"4394bd84adf0886c","entry":"save_llm_response","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/docker_filter.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/docker_filter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4394bd84adf0886c"}},{"code_sha256_prefix":"9f6003f231d81758","entry":"search_repos_with_issues","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/collect_1000_issues.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/collect_1000_issues.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9f6003f231d81758"}},{"code_sha256_prefix":"841dffee01585864","entry":"setup_logging","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/generate_dockerfile.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/generate_dockerfile.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"841dffee01585864"}},{"code_sha256_prefix":"4aa375e6a657c179","entry":"setup_logging","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/generate_dockerfile_with_strands.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/generate_dockerfile_with_strands.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4aa375e6a657c179"}},{"code_sha256_prefix":"9aa073e29d69e5c5","entry":"test_build_dockerfile","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/generate_dockerfile_with_strands.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/generate_dockerfile_with_strands.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9aa073e29d69e5c5"}},{"code_sha256_prefix":"e4b4ca867e839824","entry":"validate_date_format","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/get_github_repo.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/get_github_repo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e4b4ca867e839824"}},{"code_sha256_prefix":"499605db5913293e","entry":"verify_jsonl","repo":"amazon-science/CodeAssistBench","repo_kind":"found_in_text","path":"script/convert_to_jsonl.py","file_url":"https://github.com/amazon-science/CodeAssistBench/blob/HEAD/script/convert_to_jsonl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"499605db5913293e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}