{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cloob-modern-hopfield-networks-with-infoloob-1","title":"CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP","arxiv_id":"2110.11316","date":"2021-10-21","proceeding":null,"authors":["Andreas Fürst","Elisabeth Rumetshofer","Johannes Lehner","Viet Tran","Fei Tang","Hubert Ramsauer","David Kreil","Michael Kopp","Günter Klambauer","Angela Bitto-Nemling","Sepp Hochreiter"],"abstract":"CLIP yielded impressive results on zero-shot transfer learning tasks and is considered as a foundation model like BERT or GPT3. CLIP vision models that have a rich representation are pre-trained using the InfoNCE objective and natural language supervision before they are fine-tuned on particular tasks. Though CLIP excels at zero-shot transfer learning, it suffers from an explaining away problem, that is, it focuses on one or few features, while neglecting other relevant features. This problem is caused by insufficiently extracting the covariance structure in the original multi-modal data. We suggest to use modern Hopfield networks to tackle the problem of explaining away. Their retrieved embeddings have an enriched covariance structure derived from co-occurrences of features in the stored embeddings. However, modern Hopfield networks increase the saturation effect of the InfoNCE objective which hampers learning. We propose to use the InfoLOOB objective to mitigate this saturation effect. We introduce the novel \"Contrastive Leave One Out Boost\" (CLOOB), which uses modern Hopfield networks for covariance enrichment together with the InfoLOOB objective. In experiments we compare CLOOB to CLIP after pre-training on the Conceptual Captions and the YFCC dataset with respect to their zero-shot transfer learning performance on other datasets. CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets.","url_abs":"https://arxiv.org/abs/2110.11316v4","url_pdf":"https://arxiv.org/pdf/2110.11316v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cloob-modern-hopfield-networks-with-infoloob-1","repo_url":"https://github.com/ml-jku/cloob","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"infonce","method_name":"InfoNCE"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2110.11316","atlas_url":"https://app.syntology.ai/?focus=2110.11316","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.11316"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml-jku/cloob","reach":null}],"summary":{"ran_draft_wrong":6,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1},"community":{"samples":7,"ran":3,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"6dd41de81496f6ba","entry":"cloob","repo":"ml-jku/cloob","repo_kind":"official","path":"src/training/methods.py","file_url":"https://github.com/ml-jku/cloob/blob/HEAD/src/training/methods.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6dd41de81496f6ba"}},{"code_sha256_prefix":"1d4710b26e6ea9f5","entry":"eval_mode","repo":"JD-P/cloob-latent-diffusion","repo_kind":"community","path":"train_latent_diffusion.py","file_url":"https://github.com/JD-P/cloob-latent-diffusion/blob/HEAD/train_latent_diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d4710b26e6ea9f5"}},{"code_sha256_prefix":"7543087fb92417f3","entry":"get_alphas_sigmas","repo":"JD-P/cloob-latent-diffusion","repo_kind":"community","path":"train_latent_diffusion.py","file_url":"https://github.com/JD-P/cloob-latent-diffusion/blob/HEAD/train_latent_diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7543087fb92417f3"}},{"code_sha256_prefix":"e0792a694e05deeb","entry":"hopfield","repo":"ml-jku/cloob","repo_kind":"official","path":"src/training/methods.py","file_url":"https://github.com/ml-jku/cloob/blob/HEAD/src/training/methods.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e0792a694e05deeb"}},{"code_sha256_prefix":"4d5e096e98580645","entry":"hopfield_retrieval","repo":"ml-jku/cloob","repo_kind":"official","path":"src/training/methods.py","file_url":"https://github.com/ml-jku/cloob/blob/HEAD/src/training/methods.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4d5e096e98580645"}},{"code_sha256_prefix":"f14e4cd457038dcb","entry":"infoLOOB_loss","repo":"ml-jku/cloob","repo_kind":"official","path":"src/training/methods.py","file_url":"https://github.com/ml-jku/cloob/blob/HEAD/src/training/methods.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f14e4cd457038dcb"}},{"code_sha256_prefix":"1fdcc5450a1266f3","entry":"parse_prompt","repo":"JD-P/cloob-latent-diffusion","repo_kind":"community","path":"cfg_sample.py","file_url":"https://github.com/JD-P/cloob-latent-diffusion/blob/HEAD/cfg_sample.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1fdcc5450a1266f3"}},{"code_sha256_prefix":"07006f9eb4731a60","entry":"download_checkpoint","repo":"crowsonkb/cloob-training","repo_kind":"community","path":"cloob_training/pretrained.py","file_url":"https://github.com/crowsonkb/cloob-training/blob/HEAD/cloob_training/pretrained.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07006f9eb4731a60"}},{"code_sha256_prefix":"1b433f466bd550fe","entry":"get_and_init_model","repo":"crowsonkb/cloob-training","repo_kind":"community","path":"cloob_training/model.py","file_url":"https://github.com/crowsonkb/cloob-training/blob/HEAD/cloob_training/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b433f466bd550fe"}},{"code_sha256_prefix":"704cbcc71dd35d3d","entry":"get_config","repo":"crowsonkb/cloob-training","repo_kind":"community","path":"cloob_training/pretrained.py","file_url":"https://github.com/crowsonkb/cloob-training/blob/HEAD/cloob_training/pretrained.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"704cbcc71dd35d3d"}},{"code_sha256_prefix":"1e3e01f4209f0a95","entry":"get_hk_model","repo":"crowsonkb/cloob-training","repo_kind":"community","path":"cloob_training/model.py","file_url":"https://github.com/crowsonkb/cloob-training/blob/HEAD/cloob_training/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e3e01f4209f0a95"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}