{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/coir-a-comprehensive-benchmark-for-code","title":"CoIR: A Comprehensive Benchmark for Code Information Retrieval Models","arxiv_id":"2407.02883","date":"2024-07-03","proceeding":null,"authors":["Xiangyang Li","Kuicai Dong","Yi Quan Lee","Wei Xia","Hao Zhang","Xinyi Dai","Yasheng Wang","Ruiming Tang"],"abstract":"Despite the substantial success of Information Retrieval (IR) in various NLP tasks, most IR systems predominantly handle queries and corpora in natural language, neglecting the domain of code retrieval. Code retrieval is critically important yet remains under-explored, with existing methods and benchmarks inadequately representing the diversity of code in various domains and tasks. Addressing this gap, we present COIR (Code Information Retrieval Benchmark), a robust and comprehensive benchmark specifically designed to assess code retrieval capabilities. COIR comprises ten meticulously curated code datasets, spanning eight distinctive retrieval tasks across seven diverse domains. We first discuss the construction of COIR and its diverse dataset composition. Further, we evaluate nine widely used retrieval models using COIR, uncovering significant difficulties in performing code retrieval tasks even with state-of-the-art systems. To facilitate easy adoption and integration within existing research workflows, COIR has been developed as a user-friendly Python framework, readily installable via pip. It shares same data schema as other popular benchmarks like MTEB and BEIR, enabling seamless cross-benchmark evaluations. Through COIR, we aim to invigorate research in the code retrieval domain, providing a versatile benchmarking tool that encourages further development and exploration of code retrieval systems https://github.com/CoIR-team/coir.","url_abs":"https://arxiv.org/abs/2407.02883v2","url_pdf":"https://arxiv.org/pdf/2407.02883v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"coir-a-comprehensive-benchmark-for-code","repo_url":"https://github.com/coir-team/coir","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"code-search","task_name":"Code Search"},{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[{"slug":"coir","name":"CoIR","full_name":"Code Information Retrieval Benchmark"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/code-search-on","task":"Code Search","dataset":"","model":"Voyage-code-002","rank_in_archive_order":1,"of":1,"metrics":{"nDCG@10":"56.26"},"uses_additional_data":false},{"leaderboard":"/sota/code-search-on-coir","task":"Code Search","dataset":"CoIR","model":"Voyage-code-002","rank_in_archive_order":1,"of":1,"metrics":{"nDCG@10":"56.26"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.02883","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.02883"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/coir-team/coir","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8ca90d5388372be5","entry":"cos_sim","repo":"coir-team/coir","repo_kind":"official","path":"coir/beir/util.py","file_url":"https://github.com/coir-team/coir/blob/HEAD/coir/beir/util.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8ca90d5388372be5"}},{"code_sha256_prefix":"afa18317f44e63ad","entry":"dot_score","repo":"coir-team/coir","repo_kind":"official","path":"coir/beir/util.py","file_url":"https://github.com/coir-team/coir/blob/HEAD/coir/beir/util.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"afa18317f44e63ad"}},{"code_sha256_prefix":"5851a19162cf6895","entry":"download_and_unzip","repo":"coir-team/coir","repo_kind":"official","path":"coir/beir/util.py","file_url":"https://github.com/coir-team/coir/blob/HEAD/coir/beir/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5851a19162cf6895"}},{"code_sha256_prefix":"0adc655290cb9e75","entry":"get_tasks","repo":"coir-team/coir","repo_kind":"official","path":"coir/data_loader.py","file_url":"https://github.com/coir-team/coir/blob/HEAD/coir/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0adc655290cb9e75"}},{"code_sha256_prefix":"1599b1504459ed71","entry":"load_data_from_hf","repo":"coir-team/coir","repo_kind":"official","path":"coir/data_loader.py","file_url":"https://github.com/coir-team/coir/blob/HEAD/coir/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1599b1504459ed71"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}