{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ann-benchmarks-a-benchmarking-tool-for","title":"ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms","arxiv_id":"1807.05614","date":"2018-07-15","proceeding":null,"authors":["Martin Aumüller","Erik Bernhardsson","Alexander Faithfull"],"abstract":"This paper describes ANN-Benchmarks, a tool for evaluating the performance of in-memory approximate nearest neighbor algorithms. It provides a standard interface for measuring the performance and quality achieved by nearest neighbor algorithms on different standard data sets. It supports several different ways of integrating $k$-NN algorithms, and its configuration system automatically tests a range of parameter settings for each algorithm. Algorithms are compared with respect to many different (approximate) quality measures, and adding more is easy and fast; the included plotting front-ends can visualise these as images, $\\LaTeX$ plots, and websites with interactive plots. ANN-Benchmarks aims to provide a constantly updated overview of the current state of the art of $k$-NN algorithms. In the short term, this overview allows users to choose the correct $k$-NN algorithm and parameters for their similarity search task; in the longer term, algorithm designers will be able to use this overview to test and refine automatic parameter tuning. The paper gives an overview of the system, evaluates the results of the benchmark, and points out directions for future work. Interestingly, very different approaches to $k$-NN search yield comparable quality-performance trade-offs. The system is available at http://ann-benchmarks.com .","url_abs":"https://arxiv.org/abs/1807.05614v2","url_pdf":"https://arxiv.org/pdf/1807.05614v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ann-benchmarks-a-benchmarking-tool-for","repo_url":"https://github.com/erikbern/ann-benchmarks","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"ann-benchmarks-a-benchmarking-tool-for","repo_url":"https://github.com/maumueller/ann-benchmarks-reproducibility","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1807.05614","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1807.05614"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/erikbern/ann-benchmarks","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/maumueller/ann-benchmarks-reproducibility","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":13},"by_repo_kind":{"listed":{"samples":16,"ran":3,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b78ed1fd13d41cb3","entry":"algorithm_status","repo":"maumueller/ann-benchmarks-reproducibility","repo_kind":"listed","path":"ann_benchmarks/algorithms/definitions.py","file_url":"https://github.com/maumueller/ann-benchmarks-reproducibility/blob/HEAD/ann_benchmarks/algorithms/definitions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b78ed1fd13d41cb3"}},{"code_sha256_prefix":"15a8353f5fe218be","entry":"get_unique_algorithms","repo":"maumueller/ann-benchmarks-reproducibility","repo_kind":"listed","path":"ann_benchmarks/algorithms/definitions.py","file_url":"https://github.com/maumueller/ann-benchmarks-reproducibility/blob/HEAD/ann_benchmarks/algorithms/definitions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15a8353f5fe218be"}},{"code_sha256_prefix":"a8a893f658ddb978","entry":"instantiate_algorithm","repo":"maumueller/ann-benchmarks-reproducibility","repo_kind":"listed","path":"ann_benchmarks/algorithms/definitions.py","file_url":"https://github.com/maumueller/ann-benchmarks-reproducibility/blob/HEAD/ann_benchmarks/algorithms/definitions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8a893f658ddb978"}},{"code_sha256_prefix":"b33d2a8b86372d58","entry":"algorithm_status","repo":"erikbern/ann-benchmarks","repo_kind":"listed","path":"ann_benchmarks/definitions.py","file_url":"https://github.com/erikbern/ann-benchmarks/blob/HEAD/ann_benchmarks/definitions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b33d2a8b86372d58"}},{"code_sha256_prefix":"ad492dc47d8560b5","entry":"euclidean","repo":"erikbern/ann-benchmarks","repo_kind":"listed","path":"ann_benchmarks/distance.py","file_url":"https://github.com/erikbern/ann-benchmarks/blob/HEAD/ann_benchmarks/distance.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad492dc47d8560b5"}},{"code_sha256_prefix":"44fa38f4dba6c740","entry":"float_parse_entry","repo":"erikbern/ann-benchmarks","repo_kind":"listed","path":"ann_benchmarks/data.py","file_url":"https://github.com/erikbern/ann-benchmarks/blob/HEAD/ann_benchmarks/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44fa38f4dba6c740"}},{"code_sha256_prefix":"43546c638e833781","entry":"float_unparse_entry","repo":"erikbern/ann-benchmarks","repo_kind":"listed","path":"ann_benchmarks/data.py","file_url":"https://github.com/erikbern/ann-benchmarks/blob/HEAD/ann_benchmarks/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43546c638e833781"}},{"code_sha256_prefix":"b6d98bf31780fdf9","entry":"get_bool_env_var","repo":"erikbern/ann-benchmarks","repo_kind":"listed","path":"ann_benchmarks/util.py","file_url":"https://github.com/erikbern/ann-benchmarks/blob/HEAD/ann_benchmarks/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6d98bf31780fdf9"}},{"code_sha256_prefix":"328aec72f4ecb66d","entry":"get_config_files","repo":"erikbern/ann-benchmarks","repo_kind":"listed","path":"ann_benchmarks/definitions.py","file_url":"https://github.com/erikbern/ann-benchmarks/blob/HEAD/ann_benchmarks/definitions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"328aec72f4ecb66d"}},{"code_sha256_prefix":"311bb1de5567138a","entry":"get_dataset","repo":"erikbern/ann-benchmarks","repo_kind":"listed","path":"ann_benchmarks/datasets.py","file_url":"https://github.com/erikbern/ann-benchmarks/blob/HEAD/ann_benchmarks/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"311bb1de5567138a"}},{"code_sha256_prefix":"08c7a13d549d2592","entry":"get_dataset_fn","repo":"erikbern/ann-benchmarks","repo_kind":"listed","path":"ann_benchmarks/datasets.py","file_url":"https://github.com/erikbern/ann-benchmarks/blob/HEAD/ann_benchmarks/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08c7a13d549d2592"}},{"code_sha256_prefix":"378ed3f6e88f78c2","entry":"int_parse_entry","repo":"erikbern/ann-benchmarks","repo_kind":"listed","path":"ann_benchmarks/data.py","file_url":"https://github.com/erikbern/ann-benchmarks/blob/HEAD/ann_benchmarks/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"378ed3f6e88f78c2"}},{"code_sha256_prefix":"df5a5ca06135cc3f","entry":"jaccard","repo":"erikbern/ann-benchmarks","repo_kind":"listed","path":"ann_benchmarks/distance.py","file_url":"https://github.com/erikbern/ann-benchmarks/blob/HEAD/ann_benchmarks/distance.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df5a5ca06135cc3f"}},{"code_sha256_prefix":"2e5446b131505e9a","entry":"load_and_transform_dataset","repo":"erikbern/ann-benchmarks","repo_kind":"listed","path":"ann_benchmarks/runner.py","file_url":"https://github.com/erikbern/ann-benchmarks/blob/HEAD/ann_benchmarks/runner.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e5446b131505e9a"}},{"code_sha256_prefix":"68b87eb6bd321966","entry":"norm","repo":"erikbern/ann-benchmarks","repo_kind":"listed","path":"ann_benchmarks/distance.py","file_url":"https://github.com/erikbern/ann-benchmarks/blob/HEAD/ann_benchmarks/distance.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68b87eb6bd321966"}},{"code_sha256_prefix":"762ddcae80faee0b","entry":"train_test_split","repo":"erikbern/ann-benchmarks","repo_kind":"listed","path":"ann_benchmarks/datasets.py","file_url":"https://github.com/erikbern/ann-benchmarks/blob/HEAD/ann_benchmarks/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"762ddcae80faee0b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}