{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-discrete-representations-via-1","title":"Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrieval","arxiv_id":"2110.05789","date":"2021-10-12","proceeding":null,"authors":["Jingtao Zhan","Jiaxin Mao","Yiqun Liu","Jiafeng Guo","Min Zhang","Shaoping Ma"],"abstract":"Dense Retrieval (DR) has achieved state-of-the-art first-stage ranking effectiveness. However, the efficiency of most existing DR models is limited by the large memory cost of storing dense vectors and the time-consuming nearest neighbor search (NNS) in vector space. Therefore, we present RepCONC, a novel retrieval model that learns discrete Representations via CONstrained Clustering. RepCONC jointly trains dual-encoders and the Product Quantization (PQ) method to learn discrete document representations and enables fast approximate NNS with compact indexes. It models quantization as a constrained clustering process, which requires the document embeddings to be uniformly clustered around the quantization centroids and supports end-to-end optimization of the quantization method and dual-encoders. We theoretically demonstrate the importance of the uniform clustering constraint in RepCONC and derive an efficient approximate solution for constrained clustering by reducing it to an instance of the optimal transport problem. Besides constrained clustering, RepCONC further adopts a vector-based inverted file system (IVF) to support highly efficient vector search on CPUs. Extensive experiments on two popular ad-hoc retrieval benchmarks show that RepCONC achieves better ranking effectiveness than competitive vector quantization baselines under different compression ratio settings. It also substantially outperforms a wide range of existing retrieval models in terms of retrieval effectiveness, memory efficiency, and time efficiency.","url_abs":"https://arxiv.org/abs/2110.05789v1","url_pdf":"https://arxiv.org/pdf/2110.05789v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-discrete-representations-via-1","repo_url":"https://github.com/jingtaozhan/repconc","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"learning-discrete-representations-via-1","repo_url":"https://github.com/jingtaozhan/DRhard","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"learning-discrete-representations-via-1","repo_url":"https://github.com/jingtaozhan/JPQ","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learning-discrete-representations-via-1","repo_url":"https://github.com/jingtaozhan/LearningToRetrieve","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"constrained-clustering","task_name":"Constrained Clustering"},{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"open-domain-question-answering","task_name":"Open-Domain Question Answering"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2110.05789","atlas_url":"https://app.syntology.ai/?focus=2110.05789","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.05789"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jingtaozhan/DRhard","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jingtaozhan/JPQ","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jingtaozhan/LearningToRetrieve","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jingtaozhan/repconc","reach":null}],"summary":{"ran":13,"ran_draft_wrong":1,"unverified":8},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":21,"ran":13,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d644ea00a07b6437","entry":"PassagePreprocessingFn","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"preprocess.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/preprocess.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d644ea00a07b6437"}},{"code_sha256_prefix":"784e8a98f055bde0","entry":"batch_to_device","repo":"jingtaozhan/JPQ","repo_kind":"listed","path":"jpq/model.py","file_url":"https://github.com/jingtaozhan/JPQ/blob/HEAD/jpq/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"784e8a98f055bde0"}},{"code_sha256_prefix":"5149666c4e47a3f9","entry":"construct_flatindex_from_embeddings","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"retrieve_utils.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/retrieve_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5149666c4e47a3f9"}},{"code_sha256_prefix":"a9a03792761f628c","entry":"get_collate_function","repo":"jingtaozhan/JPQ","repo_kind":"listed","path":"jpq/dataset.py","file_url":"https://github.com/jingtaozhan/JPQ/blob/HEAD/jpq/dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9a03792761f628c"}},{"code_sha256_prefix":"9e1ebe9b0455aa43","entry":"get_doc_embeds","repo":"jingtaozhan/JPQ","repo_kind":"listed","path":"jpq/run_train.py","file_url":"https://github.com/jingtaozhan/JPQ/blob/HEAD/jpq/run_train.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e1ebe9b0455aa43"}},{"code_sha256_prefix":"62fc247c87911474","entry":"index_retrieve","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"retrieve_utils.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/retrieve_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"62fc247c87911474"}},{"code_sha256_prefix":"1a34a07957ed4ee3","entry":"load_candidate_from_stream","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"msmarco_eval.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/msmarco_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1a34a07957ed4ee3"}},{"code_sha256_prefix":"9bbe63268d471884","entry":"load_rank","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"dataset.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9bbe63268d471884"}},{"code_sha256_prefix":"42005ba9f770d57d","entry":"load_reference_from_stream","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"msmarco_eval.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/msmarco_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"42005ba9f770d57d"}},{"code_sha256_prefix":"3a4b386b64960337","entry":"load_rel","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"dataset.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"3a4b386b64960337"}},{"code_sha256_prefix":"07b0e4cab6c2d738","entry":"mean_pooling","repo":"jingtaozhan/repconc","repo_kind":"official","path":"src/repconc/models/dense/modeling_dense.py","file_url":"https://github.com/jingtaozhan/repconc/blob/HEAD/src/repconc/models/dense/modeling_dense.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07b0e4cab6c2d738"}},{"code_sha256_prefix":"7f738a55dde7b691","entry":"pack_tensor_2D","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"dataset.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"7f738a55dde7b691"}},{"code_sha256_prefix":"b201d8d93b242b70","entry":"pad_input_ids","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"preprocess.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/preprocess.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b201d8d93b242b70"}},{"code_sha256_prefix":"e827d94caab33f59","entry":"url_to_filename","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"star_tokenizer.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/star_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e827d94caab33f59"}},{"code_sha256_prefix":"b2c86c4e199eeb01","entry":"compute_loss","repo":"jingtaozhan/JPQ","repo_kind":"listed","path":"jpq/run_train.py","file_url":"https://github.com/jingtaozhan/JPQ/blob/HEAD/jpq/run_train.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2c86c4e199eeb01"}},{"code_sha256_prefix":"66e8bf589981ec7f","entry":"convert_index_to_gpu","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"retrieve_utils.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/retrieve_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"66e8bf589981ec7f"}},{"code_sha256_prefix":"b2be6c932ce41084","entry":"hf_bucket_url","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"star_tokenizer.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/star_tokenizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b2be6c932ce41084"}},{"code_sha256_prefix":"9004f78cc68a7b5c","entry":"inbatch_train","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"model.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9004f78cc68a7b5c"}},{"code_sha256_prefix":"4381d1f588a7f42a","entry":"is_remote_url","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"star_tokenizer.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/star_tokenizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4381d1f588a7f42a"}},{"code_sha256_prefix":"b32da4856889747c","entry":"load_index","repo":"jingtaozhan/JPQ","repo_kind":"listed","path":"jpq/run_retrieval.py","file_url":"https://github.com/jingtaozhan/JPQ/blob/HEAD/jpq/run_retrieval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b32da4856889747c"}},{"code_sha256_prefix":"4020d64dcc4b6c53","entry":"load_reference","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"msmarco_eval.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/msmarco_eval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4020d64dcc4b6c53"}},{"code_sha256_prefix":"650169b0049d3567","entry":"randneg_train","repo":"jingtaozhan/DRhard","repo_kind":"listed","path":"model.py","file_url":"https://github.com/jingtaozhan/DRhard/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"650169b0049d3567"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}