{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/long-code-arena-a-set-of-benchmarks-for-long","title":"Long Code Arena: a Set of Benchmarks for Long-Context Code Models","arxiv_id":"2406.11612","date":"2024-06-17","proceeding":null,"authors":["Egor Bogomolov","Aleksandra Eliseeva","Timur Galimzyanov","Evgeniy Glukhov","Anton Shapkin","Maria Tigina","Yaroslav Golubev","Alexander Kovrigin","Arie van Deursen","Maliheh Izadi","Timofey Bryksin"],"abstract":"Nowadays, the fields of code and natural language processing are evolving rapidly. In particular, models become better at processing long context windows - supported context sizes have increased by orders of magnitude over the last few years. However, there is a shortage of benchmarks for code processing that go beyond a single file of context, while the most popular ones are limited to a single method. With this work, we aim to close this gap by introducing Long Code Arena, a suite of six benchmarks for code processing tasks that require project-wide context. These tasks cover different aspects of code processing: library-based code generation, CI builds repair, project-level code completion, commit message generation, bug localization, and module summarization. For each task, we provide a manually verified dataset for testing, an evaluation suite, and open-source baseline solutions based on popular LLMs to showcase the usage of the dataset and to simplify adoption by other researchers. We publish the benchmark page on HuggingFace Spaces with the leaderboard, links to HuggingFace Hub for all the datasets, and link to the GitHub repository with baselines: https://huggingface.co/spaces/JetBrains-Research/long-code-arena.","url_abs":"https://arxiv.org/abs/2406.11612v1","url_pdf":"https://arxiv.org/pdf/2406.11612v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"long-code-arena-a-set-of-benchmarks-for-long","repo_url":"https://github.com/jetbrains-research/lca-baselines","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"code-completion","task_name":"Code Completion"},{"task_slug":"code-generation","task_name":"Code Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.11612","atlas_url":"https://app.syntology.ai/?focus=2406.11612","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.11612"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jetbrains-research/lca-baselines","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ba5ac24a77be2ea0","entry":"extract_json_from_output","repo":"jetbrains-research/lca-baselines","repo_kind":"official","path":"bug_localization/src/baselines/backbone/utils.py","file_url":"https://github.com/jetbrains-research/lca-baselines/blob/HEAD/bug_localization/src/baselines/backbone/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba5ac24a77be2ea0"}},{"code_sha256_prefix":"e1f0a26bc2cc3531","entry":"f1_score","repo":"jetbrains-research/lca-baselines","repo_kind":"official","path":"bug_localization/src/baselines/metrics/quality_metrics.py","file_url":"https://github.com/jetbrains-research/lca-baselines/blob/HEAD/bug_localization/src/baselines/metrics/quality_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1f0a26bc2cc3531"}},{"code_sha256_prefix":"2026474821ea5965","entry":"precision","repo":"jetbrains-research/lca-baselines","repo_kind":"official","path":"bug_localization/src/baselines/metrics/quality_metrics.py","file_url":"https://github.com/jetbrains-research/lca-baselines/blob/HEAD/bug_localization/src/baselines/metrics/quality_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2026474821ea5965"}},{"code_sha256_prefix":"55ad0889b7e67762","entry":"recall","repo":"jetbrains-research/lca-baselines","repo_kind":"official","path":"bug_localization/src/baselines/metrics/quality_metrics.py","file_url":"https://github.com/jetbrains-research/lca-baselines/blob/HEAD/bug_localization/src/baselines/metrics/quality_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55ad0889b7e67762"}},{"code_sha256_prefix":"4f4cada568ab31f2","entry":"sort_files_by_relevance","repo":"jetbrains-research/lca-baselines","repo_kind":"official","path":"bug_localization/src/baselines/context_composers/utils.py","file_url":"https://github.com/jetbrains-research/lca-baselines/blob/HEAD/bug_localization/src/baselines/context_composers/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f4cada568ab31f2"}},{"code_sha256_prefix":"0af2c471041b8b65","entry":"tokenize_with_regex","repo":"jetbrains-research/lca-baselines","repo_kind":"official","path":"bug_localization/src/baselines/context_composers/utils.py","file_url":"https://github.com/jetbrains-research/lca-baselines/blob/HEAD/bug_localization/src/baselines/context_composers/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0af2c471041b8b65"}},{"code_sha256_prefix":"514143a5817b30e9","entry":"tokenize_with_tiktoken","repo":"jetbrains-research/lca-baselines","repo_kind":"official","path":"bug_localization/src/baselines/context_composers/utils.py","file_url":"https://github.com/jetbrains-research/lca-baselines/blob/HEAD/bug_localization/src/baselines/context_composers/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"514143a5817b30e9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}