{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/simpler-faster-stronger-breaking-the-log-k","title":"Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE","arxiv_id":"2107.01152","date":"2021-07-02","proceeding":null,"authors":["Junya Chen","Zhe Gan","Xuan Li","Qing Guo","Liqun Chen","Shuyang Gao","Tagyoung Chung","Yi Xu","Belinda Zeng","Wenlian Lu","Fan Li","Lawrence Carin","Chenyang Tao"],"abstract":"InfoNCE-based contrastive representation learners, such as SimCLR, have been tremendously successful in recent years. However, these contrastive schemes are notoriously resource demanding, as their effectiveness breaks down with small-batch training (i.e., the log-K curse, whereas K is the batch-size). In this work, we reveal mathematically why contrastive learners fail in the small-batch-size regime, and present a novel simple, non-trivial contrastive objective named FlatNCE, which fixes this issue. Unlike InfoNCE, our FlatNCE no longer explicitly appeals to a discriminative classification goal for contrastive learning. Theoretically, we show FlatNCE is the mathematical dual formulation of InfoNCE, thus bridging the classical literature on energy modeling; and empirically, we demonstrate that, with minimal modification of code, FlatNCE enables immediate performance boost independent of the subject-matter engineering efforts. The significance of this work is furthered by the powerful generalization of contrastive learning techniques, and the introduction of new tools to monitor and diagnose contrastive training. We substantiate our claims with empirical evidence on CIFAR10, ImageNet, and other datasets, where FlatNCE consistently outperforms InfoNCE.","url_abs":"https://arxiv.org/abs/2107.01152v1","url_pdf":"https://arxiv.org/pdf/2107.01152v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"simpler-faster-stronger-breaking-the-log-k","repo_url":"https://github.com/Junya-Chen/FlatCLR","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"colorjitter","method_name":"ColorJitter"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"infonce","method_name":"InfoNCE"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"nt-xent","method_name":"NT-Xent"},{"method_slug":"random-gaussian-blur","method_name":"Random Gaussian Blur"},{"method_slug":"random-resized-crop","method_name":"Random Resized Crop"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"simclr","method_name":"SimCLR"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2107.01152","atlas_url":"https://app.syntology.ai/?focus=2107.01152","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}