{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-graph-based-anns-algorithms-to","title":"ParlayANN: Scalable and Deterministic Parallel Graph-Based Approximate Nearest Neighbor Search Algorithms","arxiv_id":"2305.04359","date":"2023-05-07","proceeding":null,"authors":["Magdalen Dobson Manohar","Zheqi Shen","Guy E. Blelloch","Laxman Dhulipala","Yan Gu","Harsha Vardhan Simhadri","Yihan Sun"],"abstract":"Approximate nearest-neighbor search (ANNS) algorithms are a key part of the modern deep learning stack due to enabling efficient similarity search over high-dimensional vector space representations (i.e., embeddings) of data. Among various ANNS algorithms, graph-based algorithms are known to achieve the best throughput-recall tradeoffs. Despite the large scale of modern ANNS datasets, existing parallel graph based implementations suffer from significant challenges to scale to large datasets due to heavy use of locks and other sequential bottlenecks, which 1) prevents them from efficiently scaling to a large number of processors, and 2) results in nondeterminism that is undesirable in certain applications. In this paper, we introduce ParlayANN, a library of deterministic and parallel graph-based approximate nearest neighbor search algorithms, along with a set of useful tools for developing such algorithms. In this library, we develop novel parallel implementations for four state-of-the-art graph-based ANNS algorithms that scale to billion-scale datasets. Our algorithms are deterministic and achieve high scalability across a diverse set of challenging datasets. In addition to the new algorithmic ideas, we also conduct a detailed experimental study of our new algorithms as well as two existing non-graph approaches. Our experimental results both validate the effectiveness of our new techniques, and lead to a comprehensive comparison among ANNS algorithms on large scale datasets with a list of interesting findings.","url_abs":"https://arxiv.org/abs/2305.04359v2","url_pdf":"https://arxiv.org/pdf/2305.04359v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-graph-based-anns-algorithms-to","repo_url":"https://github.com/cmuparlay/parlayann","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":null,"method_name":"Library"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.04359","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.04359"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cmuparlay/parlayann","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2f0bf6ae0c7bb35c","entry":"build_vamana_index","repo":"cmuparlay/parlayann","repo_kind":"official","path":"python/wrapper.py","file_url":"https://github.com/cmuparlay/parlayann/blob/HEAD/python/wrapper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f0bf6ae0c7bb35c"}},{"code_sha256_prefix":"e2cd8476d25fa918","entry":"load_index","repo":"cmuparlay/parlayann","repo_kind":"official","path":"python/wrapper.py","file_url":"https://github.com/cmuparlay/parlayann/blob/HEAD/python/wrapper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2cd8476d25fa918"}},{"code_sha256_prefix":"4f92b2fd2cd50f8a","entry":"onPprocessors","repo":"cmuparlay/parlayann","repo_kind":"official","path":"algorithms/bench/common/runTests.py","file_url":"https://github.com/cmuparlay/parlayann/blob/HEAD/algorithms/bench/common/runTests.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f92b2fd2cd50f8a"}},{"code_sha256_prefix":"ea07b85c2dd8e0eb","entry":"onPprocessors","repo":"cmuparlay/parlayann","repo_kind":"official","path":"algorithms/bench/common/runTestsANN.py","file_url":"https://github.com/cmuparlay/parlayann/blob/HEAD/algorithms/bench/common/runTestsANN.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea07b85c2dd8e0eb"}},{"code_sha256_prefix":"6cf4f80abf133b0d","entry":"shellGetOutput","repo":"cmuparlay/parlayann","repo_kind":"official","path":"algorithms/bench/common/runTests.py","file_url":"https://github.com/cmuparlay/parlayann/blob/HEAD/algorithms/bench/common/runTests.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6cf4f80abf133b0d"}},{"code_sha256_prefix":"cd5fe41954096f92","entry":"shellGetOutput","repo":"cmuparlay/parlayann","repo_kind":"official","path":"algorithms/bench/common/runTestsANN.py","file_url":"https://github.com/cmuparlay/parlayann/blob/HEAD/algorithms/bench/common/runTestsANN.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd5fe41954096f92"}},{"code_sha256_prefix":"ed0638321bb2eedc","entry":"stripFloat","repo":"cmuparlay/parlayann","repo_kind":"official","path":"algorithms/bench/common/runTests.py","file_url":"https://github.com/cmuparlay/parlayann/blob/HEAD/algorithms/bench/common/runTests.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed0638321bb2eedc"}},{"code_sha256_prefix":"00992e21f672eb50","entry":"vectors_metadata_from_file","repo":"cmuparlay/parlayann","repo_kind":"official","path":"python/_files.py","file_url":"https://github.com/cmuparlay/parlayann/blob/HEAD/python/_files.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00992e21f672eb50"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}