{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/joint-learning-of-deep-retrieval-model-and","title":"Joint Learning of Deep Retrieval Model and Product Quantization based Embedding Index","arxiv_id":"2105.03933","date":"2021-05-09","proceeding":null,"authors":["Han Zhang","Hongwei Shen","Yiming Qiu","Yunjiang Jiang","Songlin Wang","Sulong Xu","Yun Xiao","Bo Long","Wen-Yun Yang"],"abstract":"Embedding index that enables fast approximate nearest neighbor(ANN) search, serves as an indispensable component for state-of-the-art deep retrieval systems. Traditional approaches, often separating the two steps of embedding learning and index building, incur additional indexing time and decayed retrieval accuracy. In this paper, we propose a novel method called Poeem, which stands for product quantization based embedding index jointly trained with deep retrieval model, to unify the two separate steps within an end-to-end training, by utilizing a few techniques including the gradient straight-through estimator, warm start strategy, optimal space decomposition and Givens rotation. Extensive experimental results show that the proposed method not only improves retrieval accuracy significantly but also reduces the indexing time to almost none. We have open sourced our approach for the sake of comparison and reproducibility.","url_abs":"https://arxiv.org/abs/2105.03933v3","url_pdf":"https://arxiv.org/pdf/2105.03933v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"joint-learning-of-deep-retrieval-model-and","repo_url":"https://github.com/jdcomsearch/poeem","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2105.03933","atlas_url":"https://app.syntology.ai/?focus=2105.03933","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.03933"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jdcomsearch/poeem","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"06339b9bd192f3e1","entry":"compute_distortion","repo":"jdcomsearch/poeem","repo_kind":"official","path":"src/python/embedding.py","file_url":"https://github.com/jdcomsearch/poeem/blob/HEAD/src/python/embedding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"06339b9bd192f3e1"}},{"code_sha256_prefix":"6d8bad237377c500","entry":"compute_rotation","repo":"jdcomsearch/poeem","repo_kind":"official","path":"src/python/embedding.py","file_url":"https://github.com/jdcomsearch/poeem/blob/HEAD/src/python/embedding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d8bad237377c500"}},{"code_sha256_prefix":"d3edc68ba2422435","entry":"encode","repo":"jdcomsearch/poeem","repo_kind":"official","path":"src/ops/python/encode.py","file_url":"https://github.com/jdcomsearch/poeem/blob/HEAD/src/ops/python/encode.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3edc68ba2422435"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}