{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multi-grained-vision-language-pre-training","title":"Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts","arxiv_id":"2111.08276","date":"2021-11-16","proceeding":null,"authors":["Yan Zeng","Xinsong Zhang","Hang Li"],"abstract":"Most existing methods in vision language pre-training rely on object-centric features extracted through object detection and make fine-grained alignments between the extracted features and texts. It is challenging for these methods to learn relations among multiple objects. To this end, we propose a new method called X-VLM to perform `multi-grained vision language pre-training.' The key to learning multi-grained alignments is to locate visual concepts in the image given the associated texts, and in the meantime align the texts with the visual concepts, where the alignments are in multi-granularity. Experimental results show that X-VLM effectively leverages the learned multi-grained alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.","url_abs":"https://arxiv.org/abs/2111.08276v3","url_pdf":"https://arxiv.org/pdf/2111.08276v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multi-grained-vision-language-pre-training","repo_url":"https://github.com/zengyan-97/x-vlm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"open-vocabulary-attribute-detection","task_name":"Open Vocabulary Attribute Detection"},{"task_slug":"referring-expression-segmentation","task_name":"Referring Expression Segmentation"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/cross-modal-retrieval-on-coco-2014","task":"Cross-Modal Retrieval","dataset":"COCO 2014","model":"X-VLM (base)","rank_in_archive_order":9,"of":36,"metrics":{"Image-to-text R@1":"81.2","Image-to-text R@10":"98.2","Image-to-text R@5":"95.6","Text-to-image R@1":"63.4","Text-to-image R@10":"91.5","Text-to-image R@5":"85.8"},"uses_additional_data":true},{"leaderboard":"/sota/cross-modal-retrieval-on-flickr30k","task":"Cross-Modal Retrieval","dataset":"Flickr30k","model":"X-VLM (base)","rank_in_archive_order":7,"of":27,"metrics":{"Image-to-text R@1":"97.1","Image-to-text R@10":"100.0","Image-to-text R@5":"100.0","Text-to-image R@1":"86.9","Text-to-image R@10":"98.7","Text-to-image R@5":"97.3"},"uses_additional_data":true},{"leaderboard":"/sota/image-captioning-on-coco-captions","task":"Image Captioning","dataset":"COCO Captions","model":"X-VLM (base)","rank_in_archive_order":14,"of":41,"metrics":{"BLEU-4":"41.3","CIDER":"140.8"},"uses_additional_data":false},{"leaderboard":"/sota/image-retrieval-on-flickr30k-1k-test","task":"Image Retrieval","dataset":"Flickr30K 1K test","model":"X-VLM (base)","rank_in_archive_order":1,"of":18,"metrics":{"R@1":"86.9","R@10":"98.7","R@5":"97.3"},"uses_additional_data":true},{"leaderboard":"/sota/open-vocabulary-attribute-detection-on-ovad-1","task":"Open Vocabulary Attribute Detection","dataset":"OVAD-Box benchmark","model":"X-VLM","rank_in_archive_order":1,"of":7,"metrics":{"mean average precision":"28.0"},"uses_additional_data":true},{"leaderboard":"/sota/visual-grounding-on-refcoco-test-b","task":"Visual Grounding","dataset":"RefCOCO+ test B","model":"X-VLM (base)","rank_in_archive_order":6,"of":6,"metrics":{"Accuracy (%)":"76.91"},"uses_additional_data":false},{"leaderboard":"/sota/visual-grounding-on-refcoco-testa","task":"Visual Grounding","dataset":"RefCOCO+ testA","model":"X-VLM (base)","rank_in_archive_order":6,"of":7,"metrics":{"Accuracy (%)":"89.00"},"uses_additional_data":false},{"leaderboard":"/sota/visual-grounding-on-refcoco-val","task":"Visual Grounding","dataset":"RefCOCO+ val","model":"X-VLM (base)","rank_in_archive_order":6,"of":6,"metrics":{"Accuracy (%)":"84.51"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-v2-test-dev","task":"Visual Question Answering (VQA)","dataset":"VQA v2 test-dev","model":"X-VLM (base)","rank_in_archive_order":17,"of":56,"metrics":{"Accuracy":"78.22"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-nlvr2-dev","task":"Visual Reasoning","dataset":"NLVR2 Dev","model":"X-VLM (base)","rank_in_archive_order":9,"of":15,"metrics":{"Accuracy":"84.41"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-nlvr2-test","task":"Visual Reasoning","dataset":"NLVR2 Test","model":"X-VLM (base)","rank_in_archive_order":8,"of":14,"metrics":{"Accuracy":"84.76"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2111.08276","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.08276"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/zengyan-97/X-VLM","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zengyan-97/x-vlm","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"51e7514fd736cca3","entry":"build_mlp","repo":"zengyan-97/X-VLM","repo_kind":"official","path":"models/xvlm.py","file_url":"https://github.com/zengyan-97/X-VLM/blob/HEAD/models/xvlm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"51e7514fd736cca3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}