{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/disco-clip-a-distributed-contrastive-loss-for","title":"DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Training","arxiv_id":"2304.08480","date":"2023-04-17","proceeding":"CVPR 2023 1","authors":["Yihao Chen","Xianbiao Qi","Jianan Wang","Lei Zhang"],"abstract":"We propose DisCo-CLIP, a distributed memory-efficient CLIP training approach, to reduce the memory consumption of contrastive loss when training contrastive learning models. Our approach decomposes the contrastive loss and its gradient computation into two parts, one to calculate the intra-GPU gradients and the other to compute the inter-GPU gradients. According to our decomposition, only the intra-GPU gradients are computed on the current GPU, while the inter-GPU gradients are collected via all_reduce from other GPUs instead of being repeatedly computed on every GPU. In this way, we can reduce the GPU memory consumption of contrastive loss computation from $\\bigO(B^2)$ to $\\bigO(\\frac{B^2}{N})$, where $B$ and $N$ are the batch size and the number of GPUs used for training. Such a distributed solution is mathematically equivalent to the original non-distributed contrastive loss computation, without sacrificing any computation accuracy. It is particularly efficient for large-batch CLIP training. For instance, DisCo-CLIP can enable contrastive training of a ViT-B/32 model with a batch size of 32K or 196K using 8 or 64 A100 40GB GPUs, compared with the original CLIP solution which requires 128 A100 40GB GPUs to train a ViT-B/32 model with a batch size of 32K. The code will be released at https://github.com/IDEA-Research/DisCo-CLIP","url_abs":"https://arxiv.org/abs/2304.08480v1","url_pdf":"https://arxiv.org/pdf/2304.08480v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"disco-clip-a-distributed-contrastive-loss-for","repo_url":"https://github.com/idea-research/disco-clip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":null,"task_name":"GPU"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2304.08480","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.08480"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/openai/CLIP","reach":{"status":"unanswered"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/mlfoundations/open_clip","reach":null}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"9bb117639aa6c209","entry":"ClipLoss","repo":"mlfoundations/open_clip","repo_kind":"found_in_text","path":"src/open_clip/loss.py","file_url":"https://github.com/mlfoundations/open_clip/blob/HEAD/src/open_clip/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9bb117639aa6c209"}},{"code_sha256_prefix":"9875e379a3b0cd8a","entry":"_all_gather_with_grad","repo":"mlfoundations/open_clip","repo_kind":"found_in_text","path":"src/open_clip/loss.py","file_url":"https://github.com/mlfoundations/open_clip/blob/HEAD/src/open_clip/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9875e379a3b0cd8a"}},{"code_sha256_prefix":"e105065a1a8c0e5a","entry":"gather_features","repo":"mlfoundations/open_clip","repo_kind":"found_in_text","path":"src/open_clip/loss.py","file_url":"https://github.com/mlfoundations/open_clip/blob/HEAD/src/open_clip/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e105065a1a8c0e5a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}