{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/class-level-code-generation-from-natural","title":"Class-Level Code Generation from Natural Language Using Iterative, Tool-Enhanced Reasoning over Repository","arxiv_id":"2405.01573","date":"2024-04-22","proceeding":null,"authors":["Ajinkya Deshpande","Anmol Agarwal","Shashank Shet","Arun Iyer","Aditya Kanade","Ramakrishna Bairi","Suresh Parthasarathy"],"abstract":"LLMs have demonstrated significant potential in code generation tasks, achieving promising results at the function or statement level across various benchmarks. However, the complexities associated with creating code artifacts like classes, particularly within the context of real-world software repositories, remain underexplored. Prior research treats class-level generation as an isolated task, neglecting the intricate dependencies & interactions that characterize real-world software environments. To address this gap, we introduce RepoClassBench, a comprehensive benchmark designed to rigorously evaluate LLMs in generating complex, class-level code within real-world repositories. RepoClassBench includes \"Natural Language to Class generation\" tasks across Java, Python & C# from a selection of repositories. We ensure that each class in our dataset not only has cross-file dependencies within the repository but also includes corresponding test cases to verify its functionality. We find that current models struggle with the realistic challenges posed by our benchmark, primarily due to their limited exposure to relevant repository contexts. To address this shortcoming, we introduce Retrieve-Repotools-Reflect (RRR), a novel approach that equips LLMs with static analysis tools to iteratively navigate & reason about repository-level context in an agent-based framework. Our experiments demonstrate that RRR significantly outperforms existing baselines on RepoClassBench, showcasing its effectiveness across programming languages & under various settings. Our findings emphasize the critical need for code-generation benchmarks to incorporate repo-level dependencies to more accurately reflect the complexities of software development. Our work shows the benefits of leveraging specialized tools to enhance LLMs' understanding of repository context. We plan to make our dataset & evaluation harness public.","url_abs":"https://arxiv.org/abs/2405.01573v2","url_pdf":"https://arxiv.org/pdf/2405.01573v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"class-level-code-generation-from-natural","repo_url":"https://github.com/microsoft/repoclassbench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"class-level-code-generation","task_name":"Class-level Code Generation"},{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"navigate","task_name":"Navigate"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.01573","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.01573"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/repoclassbench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7c44da58af120b8c","entry":"extract_method_info","repo":"microsoft/repoclassbench","repo_kind":"listed","path":"repoclassbench/evaluator/java_evaluator.py","file_url":"https://github.com/microsoft/repoclassbench/blob/HEAD/repoclassbench/evaluator/java_evaluator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c44da58af120b8c"}},{"code_sha256_prefix":"4b3e5e9f9e7d2f72","entry":"filter_nodes","repo":"microsoft/repoclassbench","repo_kind":"listed","path":"repoclassbench/evaluator/java_evaluator.py","file_url":"https://github.com/microsoft/repoclassbench/blob/HEAD/repoclassbench/evaluator/java_evaluator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b3e5e9f9e7d2f72"}},{"code_sha256_prefix":"881ab4691b2595de","entry":"get_method_body","repo":"microsoft/repoclassbench","repo_kind":"listed","path":"repoclassbench/evaluator/java_evaluator.py","file_url":"https://github.com/microsoft/repoclassbench/blob/HEAD/repoclassbench/evaluator/java_evaluator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"881ab4691b2595de"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}