{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-laws-for-dense-retrieval","title":"Scaling Laws For Dense Retrieval","arxiv_id":"2403.18684","date":"2024-03-27","proceeding":null,"authors":["Yan Fang","Jingtao Zhan","Qingyao Ai","Jiaxin Mao","Weihang Su","Jia Chen","Yiqun Liu"],"abstract":"Scaling up neural models has yielded significant advancements in a wide array of tasks, particularly in language generation. Previous studies have found that the performance of neural models frequently adheres to predictable scaling laws, correlated with factors such as training set size and model size. This insight is invaluable, especially as large-scale experiments grow increasingly resource-intensive. Yet, such scaling law has not been fully explored in dense retrieval due to the discrete nature of retrieval metrics and complex relationships between training data and model sizes in retrieval tasks. In this study, we investigate whether the performance of dense retrieval models follows the scaling law as other neural models. We propose to use contrastive log-likelihood as the evaluation metric and conduct extensive experiments with dense retrieval models implemented with different numbers of parameters and trained with different amounts of annotated data. Results indicate that, under our settings, the performance of dense retrieval models follows a precise power-law scaling related to the model size and the number of annotations. Additionally, we examine scaling with prevalent data augmentation methods to assess the impact of annotation quality, and apply the scaling law to find the best resource allocation strategy under a budget constraint. We believe that these insights will significantly contribute to understanding the scaling effect of dense retrieval models and offer meaningful guidance for future research endeavors.","url_abs":"https://arxiv.org/abs/2403.18684v2","url_pdf":"https://arxiv.org/pdf/2403.18684v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-laws-for-dense-retrieval","repo_url":"https://github.com/jingtaozhan/drscale","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2403.18684","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.18684"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jingtaozhan/drscale","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"784e8a98f055bde0","entry":"batch_to_device","repo":"jingtaozhan/drscale","repo_kind":"official","path":"modeling.py","file_url":"https://github.com/jingtaozhan/drscale/blob/HEAD/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"784e8a98f055bde0"}},{"code_sha256_prefix":"008a1957ed0f9fdc","entry":"get_collate_function","repo":"jingtaozhan/drscale","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/jingtaozhan/drscale/blob/HEAD/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"008a1957ed0f9fdc"}},{"code_sha256_prefix":"7cbdad4c2ba7d4ec","entry":"is_main_process","repo":"jingtaozhan/drscale","repo_kind":"official","path":"training_arguments.py","file_url":"https://github.com/jingtaozhan/drscale/blob/HEAD/training_arguments.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7cbdad4c2ba7d4ec"}},{"code_sha256_prefix":"c8685cb7b15dff2b","entry":"load_msmarco_posneg","repo":"jingtaozhan/drscale","repo_kind":"official","path":"utils.py","file_url":"https://github.com/jingtaozhan/drscale/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8685cb7b15dff2b"}},{"code_sha256_prefix":"27a9b1aa386a7635","entry":"load_msmarco_queries","repo":"jingtaozhan/drscale","repo_kind":"official","path":"utils.py","file_url":"https://github.com/jingtaozhan/drscale/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27a9b1aa386a7635"}},{"code_sha256_prefix":"c8927edacebe2572","entry":"tokenize","repo":"jingtaozhan/drscale","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/jingtaozhan/drscale/blob/HEAD/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8927edacebe2572"}},{"code_sha256_prefix":"7715840d7525041b","entry":"truncate_run","repo":"jingtaozhan/drscale","repo_kind":"official","path":"retrieval.py","file_url":"https://github.com/jingtaozhan/drscale/blob/HEAD/retrieval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7715840d7525041b"}},{"code_sha256_prefix":"7a79a6ea04d70b62","entry":"load_msmarco_qrels","repo":"jingtaozhan/drscale","repo_kind":"official","path":"utils.py","file_url":"https://github.com/jingtaozhan/drscale/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a79a6ea04d70b62"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}