{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lotusfilter-fast-diverse-nearest-neighbor-1","title":"LotusFilter: Fast Diverse Nearest Neighbor Search via a Learned Cutoff Table","arxiv_id":"2506.04790","date":"2025-06-05","proceeding":"CVPR 2025 1","authors":["Yusuke Matsui"],"abstract":"Approximate nearest neighbor search (ANNS) is an essential building block for applications like RAG but can sometimes yield results that are overly similar to each other. In certain scenarios, search results should be similar to the query and yet diverse. We propose LotusFilter, a post-processing module to diversify ANNS results. We precompute a cutoff table summarizing vectors that are close to each other. During the filtering, LotusFilter greedily looks up the table to delete redundant vectors from the candidates. We demonstrated that the LotusFilter operates fast (0.02 [ms/query]) in settings resembling real-world RAG applications, utilizing features such as OpenAI embeddings. Our code is publicly available at https://github.com/matsui528/lotf.","url_abs":"https://arxiv.org/abs/2506.04790v1","url_pdf":"https://arxiv.org/pdf/2506.04790v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lotusfilter-fast-diverse-nearest-neighbor-1","repo_url":"https://github.com/matsui528/lotf","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"rag","task_name":"RAG"}],"methods":[{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"rag","method_name":"RAG"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2506.04790","atlas_url":"https://app.syntology.ai/?focus=2506.04790","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.04790"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/matsui528/lotf","reach":null}],"summary":{"ran_draft_wrong":2,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"02850216018fe759","entry":"batch_range_search","repo":"matsui528/lotf","repo_kind":"official","path":"src/lotf/core.py","file_url":"https://github.com/matsui528/lotf/blob/HEAD/src/lotf/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"02850216018fe759"}},{"code_sha256_prefix":"3a09eaed6d37475c","entry":"build_neighbor_lists","repo":"matsui528/lotf","repo_kind":"official","path":"src/lotf/core.py","file_url":"https://github.com/matsui528/lotf/blob/HEAD/src/lotf/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a09eaed6d37475c"}},{"code_sha256_prefix":"af54b8be915373f5","entry":"flatten_with_offsets","repo":"matsui528/lotf","repo_kind":"official","path":"src/lotf/core.py","file_url":"https://github.com/matsui528/lotf/blob/HEAD/src/lotf/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af54b8be915373f5"}},{"code_sha256_prefix":"eb1f77774104bb33","entry":"CutoffTable","repo":"matsui528/lotf","repo_kind":"official","path":"src/lotf/core.py","file_url":"https://github.com/matsui528/lotf/blob/HEAD/src/lotf/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb1f77774104bb33"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}