Papers › Toward Building General Foundation Models for Language, Vision, and Vision-Language...

Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks

12 Jan 2023arXiv:2301.05065archive 2025-07-28

Xinsong Zhang, Yan Zeng, Jipeng Zhang, Hang Li

Foundation models or pre-trained models have substantially improved the performance of various language, vision, and vision-language understanding tasks. However, existing foundation models can only perform the best in one type of tasks, namely language, vision, or vision-language. It is still an open question whether it is possible to construct a foundation model performing the best for all the understanding tasks, which we call a general foundation model. In this paper, we propose a new general foundation model, X-FM (the X-Foundation Model). X-FM has one language encoder, one vision encoder, and one fusion encoder, as well as a new training method. The training method includes two new techniques for learning X-FM from text, image, and image-text pair data. One is to stop gradients from the vision-language training when learning the language encoder. The other is to leverage the vision-language training to guide the learning of the vision encoder. Extensive experiments on benchmark datasets show that X-FM can significantly outperform existing general foundation models and perform better than or comparable to existing foundation models specifically for language, vision, or vision-language understanding. Code and pre-trained models are released at https://github.com/zhangxinsong-nlp/XFM.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

zhangxinsong-nlp/XFM officialmentioned in papermentioned on GitHubpytorchSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Cross-Modal RetrievalOpen-Ended Question AnsweringVisual GroundingVisual Question Answering (VQA)Visual Reasoning

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Cross-Modal Retrieval COCO 2014 XFM (base) Image-to-text R@1 84.2 #4 of 36 Archive leaderboard report
Cross-Modal Retrieval COCO 2014 XFM (base) Image-to-text R@10 98.4 #4 of 36 Archive leaderboard report
Cross-Modal Retrieval COCO 2014 XFM (base) Image-to-text R@5 96.4 #4 of 36 Archive leaderboard report
Cross-Modal Retrieval COCO 2014 XFM (base) Text-to-image R@1 67.0 #4 of 36 Archive leaderboard report
Cross-Modal Retrieval COCO 2014 XFM (base) Text-to-image R@10 92.4 #4 of 36 Archive leaderboard report
Cross-Modal Retrieval COCO 2014 XFM (base) Text-to-image R@5 87.2 #4 of 36 Archive leaderboard report
Visual Grounding RefCOCO+ test B XFM (base) Accuracy (%) 79.8 #4 of 6 Archive leaderboard report
Visual Grounding RefCOCO+ testA XFM (base) Accuracy (%) 90.4 #4 of 7 Archive leaderboard report
Visual Grounding RefCOCO+ val XFM (base) Accuracy (%) 86.1 #4 of 6 Archive leaderboard report
Visual Question Answering (VQA) VQA v2 test-dev XFM (base) Accuracy 80.4 #12 of 56 Archive leaderboard report
Visual Reasoning NLVR2 Dev XFM (base) Accuracy 87.6 #3 of 15 Archive leaderboard report
Visual Reasoning NLVR2 Test XFM (base) Accuracy 88.4 #3 of 14 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections