{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-deep-contrastive-learning-batch-size","title":"Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup","arxiv_id":"2101.06983","date":"2021-01-18","proceeding":"ACL (RepL4NLP) 2021 8","authors":["Luyu Gao","Yunyi Zhang","Jiawei Han","Jamie Callan"],"abstract":"Contrastive learning has been applied successfully to learn vector representations of text. Previous research demonstrated that learning high-quality representations benefits from batch-wise contrastive loss with a large number of negatives. In practice, the technique of in-batch negative is used, where for each example in a batch, other batch examples' positives will be taken as its negatives, avoiding encoding extra negatives. This, however, still conditions each example's loss on all batch examples and requires fitting the entire large batch into GPU memory. This paper introduces a gradient caching technique that decouples backpropagation between contrastive loss and the encoder, removing encoder backward pass data dependency along the batch dimension. As a result, gradients can be computed for one subset of the batch at a time, leading to almost constant memory usage.","url_abs":"https://arxiv.org/abs/2101.06983v2","url_pdf":"https://arxiv.org/pdf/2101.06983v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-deep-contrastive-learning-batch-size","repo_url":"https://github.com/luyug/GC-DPR","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"scaling-deep-contrastive-learning-batch-size","repo_url":"https://github.com/luyug/GradCache","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"unanswered"}},{"paper_slug":"scaling-deep-contrastive-learning-batch-size","repo_url":"https://github.com/delyan-boychev/grad-cache-con-learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"scaling-deep-contrastive-learning-batch-size","repo_url":"https://github.com/lightonai/pylate","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"scaling-deep-contrastive-learning-batch-size","repo_url":"https://github.com/luyug/Condenser","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"scaling-deep-contrastive-learning-batch-size","repo_url":"https://github.com/luyug/Reranker","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"scaling-deep-contrastive-learning-batch-size","repo_url":"https://github.com/openmatch/ance-tele","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":null,"task_name":"GPU"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2101.06983","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2101.06983"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/delyan-boychev/grad-cache-con-learning","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lightonai/pylate","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luyug/GradCache","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luyug/Reranker","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openmatch/ance-tele","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luyug/GC-DPR","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luyug/Condenser","reach":null}],"summary":{"ran_draft_wrong":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"3e65fdbefb42a546","entry":"load_passages","repo":"luyug/GC-DPR","repo_kind":"official","path":"dense_retriever.py","file_url":"https://github.com/luyug/GC-DPR/blob/HEAD/dense_retriever.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3e65fdbefb42a546"}},{"code_sha256_prefix":"4d5516637b50f1ad","entry":"no_op_collate","repo":"luyug/GC-DPR","repo_kind":"official","path":"generate_dense_embeddings.py","file_url":"https://github.com/luyug/GC-DPR/blob/HEAD/generate_dense_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4d5516637b50f1ad"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}