Methods › Computer Vision › Vision and Language Pre-Trained Models › AltCLIP

AltCLIP

5 papers tagged archive 2025-07-28

Introduced by Zhongzhi Chen et al. in AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities

archive 2025-07-28 Description, source and code snippet are the archive's method entry.

In this work, we present a conceptually simple and effective method to train a strong bilingual multimodal representation model. Starting from the pretrained multimodal representation model CLIP released by OpenAI, we switched its text encoder with a pretrained multilingual text encoder XLM-R, and aligned both languages and image representations by a two-stage training schema consisting of teacher learning and contrastive learning. We validate our method through evaluations of a wide range of tasks. We set new state-of-the-art performances on a bunch of tasks including ImageNet-CN, Flicker30k- CN, and COCO-CN. Further, we obtain very close performances with CLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding. Our models and code are available at https://github.com/FlagAI-Open/FlagAI.

Paper Source

Papers archive 2025-07-28

5 shown of 5, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.

Sparse vs Contiguous Adversarial Pixel Perturbations in Multimodal Models: An Empirical Analysis
25 Jul 2024 · 1 repository · arXiv:2407.18251
Assessing Brittleness of Image-Text Retrieval Benchmarks from Vision-Language Models Perspective
21 Jul 2024 · 0 repositories · arXiv:2407.15239
A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene
17 Apr 2024 · 0 repositories · arXiv:2404.11249
AltDiffusion: A Multilingual Text-to-Image Diffusion Model
19 Aug 2023 · 1 repository · arXiv:2308.09991

Syntology ran 9 of 11 samples · 2 unverified · 11 pointer-only (licence)
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities
12 Nov 2022 · 2 repositories · arXiv:2211.06679

Syntology ran 2 of 11 samples · 9 unverified

Tasks archive 2025-07-28

20 shown of 23 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.

Task	Papers
Image Classification	2
Knowledge Distillation	2
Zero-Shot Image Classification	2
Blocking	1
Concept Alignment	1
Contrastive Learning	1
Cross-Modal Retrieval	1
Image Retrieval	1
Image-text Retrieval	1
Image-to-Text Retrieval	1
Information Retrieval	1
Language Modelling	1
Retrieval	1
Text Retrieval	1
Text-to-Image Generation	1
XLM-R	1
Zero-Shot Cross-Modal Retrieval	1
Zero-Shot Transfer Image Classification	1
Zero-Shot Transfer Image Classification (CN)	1
Zero-shot Image Retrieval	1

Usage over time archive 2025-07-28

Papers per year the archive tags with this method, by the paper's archive date (5 dated). Bars are counts, not a trend claim.

Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).

Categories archive 2025-07-28

Vision and Language Pre-Trained Models

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections