Papers › No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without...

No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models

26 Mar 2026arXiv:2603.25722added by Syntology

Hai X. Pham, David T. Hoffmann, Ricardo Guerrero, Brais Martinez

Title, abstract, authors and date from arXiv's metadata (CC0); this paper is not in the Papers with Code archive (frozen 2025-07-28).

Contrastive vision-language (V&L) models remain a popular choice for various applications. However, several limitations have emerged, most notably the limited ability of V&L models to learn compositional representations. Prior methods often addressed this limitation by generating custom training data to obtain hard negative samples. Hard negatives have been shown to improve performance on compositionality tasks, but are often specific to a single benchmark, do not generalize, and can cause substantial degradation of basic V&L capabilities such as zero-shot or retrieval performance, rendering them impractical. In this work we follow a different approach. We identify two root causes that limit compositionality performance of V&Ls: 1) Long training captions do not require a compositional representation; and 2) The final global pooling in the text and image encoders lead to a complete loss of the necessary information to learn binding in the first place. As a remedy, we propose two simple solutions: 1) We obtain short concept centric caption parts using standard NLP software and align those with the image; and 2) We introduce a parameter-free cross-modal attention-pooling to obtain concept centric visual embeddings from the image encoder. With these two changes and simple auxiliary contrastive losses, we obtain SOTA performance on standard compositionality benchmarks, while maintaining or improving strong zero-shot and retrieval capabilities. This is achieved without increasing inference cost. We release the code for this work at https://github.com/saic-fi/concept_centric_clip.

PaperPDF

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

saicfi/concept_centric_clip found in paper text by Syntologynot reachable when probed 2026-09-16 — repositories for recent papers often appear after camera-readySyntology: not harvested report

Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Syntology holds the repository link but has not harvested or run code from it.

Results from the paper

The Papers with Code archive ends with its 2025-07-28 snapshot. This paper's arXiv identifier, 2603.25722, was issued in March 2026, after that date, so the archive has no leaderboard rows for it.

Placed on leaderboards by Syntology Syntology

Syntology's extractor, the model Claude Sonnet 4.5, judged this paper's tables to report on 9 leaderboards (a model's judgement, not a result) and has not placed the paper on any of them: Fine-Grained Image Classification · Caltech-101 (refused by a rule: the model gave no value to check against the cell); Fine-Grained Image Classification · Food-101 (refused by a rule: the table's column header does not name the leaderboard's metric); Fine-Grained Image Classification · Oxford 102 Flowers (refused by a rule: the table's column header does not name the leaderboard's metric); Image Classification · CIFAR-10 (refused by a rule: the table's column header does not name the leaderboard's metric); Image Classification · CIFAR-100 (refused by a rule: the table's column header does not name the leaderboard's metric); Image Classification · DTD (refused by a rule: the table's column header does not name the leaderboard's metric); Image Classification · Flowers-102 (refused by a rule: the table's column header does not name the leaderboard's metric); Image Classification · Food-101 (refused by a rule: the table's column header does not name the leaderboard's metric); Image-to-Text Retrieval · Flickr30k (rejected by the independent check). What is not shown.

Syntology has checked 8,886 of the 18,637 papers on this site that are newer than the archive (for 3,717 of them no archive leaderboard matched the paper's tables, so there was nothing further to check); 733 were read and have nothing to place (arXiv has no HTML version of the paper, or that version has no tables), 41 could not be read (the extractor's reply could not be parsed), and results from the other 8,977 appear after they are checked.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections