{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cross-view-language-modeling-towards-unified","title":"Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training","arxiv_id":"2206.00621","date":"2022-06-01","proceeding":null,"authors":["Yan Zeng","Wangchunshu Zhou","Ao Luo","Ziming Cheng","Xinsong Zhang"],"abstract":"In this paper, we introduce Cross-View Language Modeling, a simple and effective pre-training framework that unifies cross-lingual and cross-modal pre-training with shared architectures and objectives. Our approach is motivated by a key observation that cross-lingual and cross-modal pre-training share the same goal of aligning two different views of the same object into a common semantic space. To this end, the cross-view language modeling framework considers both multi-modal data (i.e., image-caption pairs) and multi-lingual data (i.e., parallel sentence pairs) as two different views of the same object, and trains the model to align the two views by maximizing the mutual information between them with conditional masked language modeling and contrastive learning. We pre-train CCLM, a Cross-lingual Cross-modal Language Model, with the cross-view language modeling framework. Empirical results on IGLUE, a multi-lingual multi-modal benchmark, and two multi-lingual image-text retrieval datasets show that while conceptually simpler, CCLM significantly outperforms the prior state-of-the-art with an average absolute improvement of over 10%. Moreover, CCLM is the first multi-lingual multi-modal pre-trained model that surpasses the translate-test performance of representative English vision-language models by zero-shot cross-lingual transfer.","url_abs":"https://arxiv.org/abs/2206.00621v2","url_pdf":"https://arxiv.org/pdf/2206.00621v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cross-view-language-modeling-towards-unified","repo_url":"https://github.com/zengyan-97/cclm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"cross-lingual-transfer","task_name":"Cross-Lingual Transfer"},{"task_slug":"image-text-retrieval","task_name":"Image-text Retrieval"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"masked-language-modeling","task_name":"Masked Language Modeling"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"zero-shot-cross-lingual-image-to-text","task_name":"Zero-Shot Cross-Lingual Image-to-Text Retrieval"},{"task_slug":"zero-shot-cross-lingual-text-to-image","task_name":"Zero-Shot Cross-Lingual Text-to-Image Retrieval"},{"task_slug":"zero-shot-cross-lingual-visual-natural","task_name":"Zero-Shot Cross-Lingual Visual Natural Language Inference"},{"task_slug":"zero-shot-cross-lingual-visual-question","task_name":"Zero-Shot Cross-Lingual Visual Question Answering"},{"task_slug":"zero-shot-cross-lingual-visual-reasoning","task_name":"Zero-Shot Cross-Lingual Visual Reasoning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2206.00621","atlas_url":"https://app.syntology.ai/?focus=2206.00621","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.00621"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zengyan-97/cclm","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran_honours":3,"ran_draft_wrong":2,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"e0a06ded5d4f6c3c","entry":"box_cxcywh_to_xyxy","repo":"zengyan-97/cclm","repo_kind":"official","path":"models/box_ops.py","file_url":"https://github.com/zengyan-97/cclm/blob/HEAD/models/box_ops.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"e0a06ded5d4f6c3c"}},{"code_sha256_prefix":"ce749f837424dd8c","entry":"box_iou","repo":"zengyan-97/cclm","repo_kind":"official","path":"models/box_ops.py","file_url":"https://github.com/zengyan-97/cclm/blob/HEAD/models/box_ops.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"ce749f837424dd8c"}},{"code_sha256_prefix":"031b02fb990284ba","entry":"box_xyxy_to_cxcywh","repo":"zengyan-97/cclm","repo_kind":"official","path":"models/box_ops.py","file_url":"https://github.com/zengyan-97/cclm/blob/HEAD/models/box_ops.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"031b02fb990284ba"}},{"code_sha256_prefix":"e5b89b1f72c319ed","entry":"clip_loss","repo":"zengyan-97/cclm","repo_kind":"official","path":"models/clip_vit.py","file_url":"https://github.com/zengyan-97/cclm/blob/HEAD/models/clip_vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"e5b89b1f72c319ed"}},{"code_sha256_prefix":"4675779aec92d6c0","entry":"contrastive_loss","repo":"zengyan-97/cclm","repo_kind":"official","path":"models/clip_vit.py","file_url":"https://github.com/zengyan-97/cclm/blob/HEAD/models/clip_vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"4675779aec92d6c0"}},{"code_sha256_prefix":"c7f85279d772ea19","entry":"itm_eval","repo":"zengyan-97/cclm","repo_kind":"official","path":"Retrieval.py","file_url":"https://github.com/zengyan-97/cclm/blob/HEAD/Retrieval.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"c7f85279d772ea19"}},{"code_sha256_prefix":"2cc882e3c611cc37","entry":"get_acc","repo":"zengyan-97/cclm","repo_kind":"official","path":"VQA.py","file_url":"https://github.com/zengyan-97/cclm/blob/HEAD/VQA.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"2cc882e3c611cc37"}},{"code_sha256_prefix":"e3d6fbf640a143ee","entry":"reinit_scheduler_properties_mysched","repo":"zengyan-97/cclm","repo_kind":"official","path":"NLVR_pretrain.py","file_url":"https://github.com/zengyan-97/cclm/blob/HEAD/NLVR_pretrain.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"e3d6fbf640a143ee"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}