{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pali-a-jointly-scaled-multilingual-language","title":"PaLI: A Jointly-Scaled Multilingual Language-Image Model","arxiv_id":"2209.06794","date":"2022-09-14","proceeding":null,"authors":["Xi Chen","Xiao Wang","Soravit Changpinyo","AJ Piergiovanni","Piotr Padlewski","Daniel Salz","Sebastian Goodman","Adam Grycner","Basil Mustafa","Lucas Beyer","Alexander Kolesnikov","Joan Puigcerver","Nan Ding","Keran Rong","Hassan Akbari","Gaurav Mishra","Linting Xue","Ashish Thapliyal","James Bradbury","Weicheng Kuo","Mojtaba Seyedhosseini","Chao Jia","Burcu Karagol Ayan","Carlos Riquelme","Andreas Steiner","Anelia Angelova","Xiaohua Zhai","Neil Houlsby","Radu Soricut"],"abstract":"Effective scaling and a flexible task interface enable large language models to excel at many tasks. We present PaLI (Pathways Language and Image model), a model that extends this approach to the joint modeling of language and vision. PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages. To train PaLI, we make use of large pre-trained encoder-decoder language models and Vision Transformers (ViTs). This allows us to capitalize on their existing capabilities and leverage the substantial cost of training them. We find that joint scaling of the vision and language components is important. Since existing Transformers for language are much larger than their vision counterparts, we train a large, 4-billion parameter ViT (ViT-e) to quantify the benefits from even larger-capacity vision models. To train PaLI, we create a large multilingual mix of pretraining tasks, based on a new image-text training set containing 10B images and texts in over 100 languages. PaLI achieves state-of-the-art in multiple vision and language tasks (such as captioning, visual question-answering, scene-text understanding), while retaining a simple, modular, and scalable design.","url_abs":"https://arxiv.org/abs/2209.06794v4","url_pdf":"https://arxiv.org/pdf/2209.06794v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pali-a-jointly-scaled-multilingual-language","repo_url":"https://github.com/google-research/big_vision","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"few-shot-image-classification","task_name":"Few-Shot Image Classification"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"},{"task_slug":"zero-shot-image-classification","task_name":"Zero-Shot Image Classification"},{"task_slug":"zero-shot-transfer-image-classification","task_name":"Zero-Shot Transfer Image Classification"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-captioning-on-nocaps-in-domain","task":"Image Captioning","dataset":"nocaps in-domain","model":"PaLI","rank_in_archive_order":1,"of":41,"metrics":{"CIDEr":"149.1"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-in-domain","task":"Image Captioning","dataset":"nocaps in-domain","model":"PaLI","rank_in_archive_order":4,"of":41,"metrics":{"B1":"88.02","B2":"75.21","B3":"59.38","B4":"41.16","CIDEr":"121.09","METEOR":"34.22","ROUGE-L":"64.39","SPICE":"15.69"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-near-domain","task":"Image Captioning","dataset":"nocaps near-domain","model":"PaLI","rank_in_archive_order":2,"of":40,"metrics":{"B1":"88.57","B2":"75.56","B3":"58.99","B4":"39.98","CIDEr":"124.35","METEOR":"33.47","ROUGE-L":"63.99","SPICE":"15.75"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-near-domain","task":"Image Captioning","dataset":"nocaps near-domain","model":"PaLI","rank_in_archive_order":40,"of":40,"metrics":{"SPICE":"15.75"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-out-of-domain","task":"Image Captioning","dataset":"nocaps out-of-domain","model":"PaLI","rank_in_archive_order":1,"of":40,"metrics":{"B1":"86.28","B2":"71.19","B3":"52.63","B4":"32.0","CIDEr":"126.67","METEOR":"30.99","ROUGE-L":"61.35","SPICE":"15.49"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet-v2","task":"Image Classification","dataset":"ImageNet V2","model":"ViT-e","rank_in_archive_order":2,"of":33,"metrics":{"Top 1 Accuracy":"84.3"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-objectnet","task":"Image Classification","dataset":"ObjectNet","model":"ViT-e","rank_in_archive_order":13,"of":106,"metrics":{"Top-1 Accuracy":"72.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-ok-vqa","task":"Visual Question Answering (VQA)","dataset":"OK-VQA","model":"PaLI 17B","rank_in_archive_order":4,"of":37,"metrics":{"Accuracy":"64.5"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-textvqa-test-1","task":"Visual Question Answering (VQA)","dataset":"TextVQA test-standard","model":"PaLI","rank_in_archive_order":1,"of":12,"metrics":{"overall":"73.1"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-v2-test-dev","task":"Visual Question Answering (VQA)","dataset":"VQA v2 test-dev","model":"PaLI","rank_in_archive_order":1,"of":56,"metrics":{"Accuracy":"84.3"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vizwiz-2020-vqa","task":"Visual Question Answering (VQA)","dataset":"VizWiz 2020 VQA","model":"PaLI","rank_in_archive_order":1,"of":16,"metrics":{"overall":"73.3"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-1","task":"Zero-Shot Transfer Image Classification","dataset":"ImageNet","model":"LiT ViT-e","rank_in_archive_order":6,"of":23,"metrics":{"Accuracy (Private)":"85.4"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-1","task":"Zero-Shot Transfer Image Classification","dataset":"ImageNet","model":"PaLI","rank_in_archive_order":19,"of":23,"metrics":{"Accuracy (Private)":"72.11"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-3","task":"Zero-Shot Transfer Image Classification","dataset":"ImageNet V2","model":"LiT ViT-e","rank_in_archive_order":5,"of":13,"metrics":{"Accuracy (Private)":"80.6"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-3","task":"Zero-Shot Transfer Image Classification","dataset":"ImageNet V2","model":"PaLI","rank_in_archive_order":13,"of":13,"metrics":{"Accuracy (Private)":"64.46"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-5","task":"Zero-Shot Transfer Image Classification","dataset":"ImageNet-A","model":"LiT ViT-e","rank_in_archive_order":3,"of":13,"metrics":{"Accuracy (Private)":"88.0"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-5","task":"Zero-Shot Transfer Image Classification","dataset":"ImageNet-A","model":"PaLI","rank_in_archive_order":13,"of":13,"metrics":{"Accuracy (Private)":"44.7"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-4","task":"Zero-Shot Transfer Image Classification","dataset":"ImageNet-R","model":"LiT ViT-e","rank_in_archive_order":3,"of":12,"metrics":{"Accuracy":"96.1"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-4","task":"Zero-Shot Transfer Image Classification","dataset":"ImageNet-R","model":"PaLI","rank_in_archive_order":12,"of":12,"metrics":{"Accuracy":"81.97"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-9","task":"Zero-Shot Transfer Image Classification","dataset":"ImageNet-S","model":"PaLI","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy (Private)":"63.83","Top 5 Accuracy":"79.3"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-6","task":"Zero-Shot Transfer Image Classification","dataset":"ObjectNet","model":"LiT ViT-e","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy (Private)":"84.9"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-6","task":"Zero-Shot Transfer Image Classification","dataset":"ObjectNet","model":"PaLI","rank_in_archive_order":9,"of":9,"metrics":{"Accuracy (Private)":"42.62","Top 5 Accuracy":"58.35"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2209.06794","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.06794"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-research/big_vision","reach":null}],"summary":{"ran":1,"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"264ee304ace700f9","entry":"Model","repo":"google-research/big_vision","repo_kind":"official","path":"big_vision/models/proj/paligemma/paligemma.py","file_url":"https://github.com/google-research/big_vision/blob/HEAD/big_vision/models/proj/paligemma/paligemma.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"264ee304ace700f9"}},{"code_sha256_prefix":"8841a31daa3af774","entry":"make_attn_mask","repo":"google-research/big_vision","repo_kind":"official","path":"big_vision/models/proj/paligemma/paligemma.py","file_url":"https://github.com/google-research/big_vision/blob/HEAD/big_vision/models/proj/paligemma/paligemma.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8841a31daa3af774"}},{"code_sha256_prefix":"774022ae9ec0b97e","entry":"combine_and_keep_eval","repo":"google-research/big_vision","repo_kind":"official","path":"big_vision/configs/proj/paligemma/transfers/common.py","file_url":"https://github.com/google-research/big_vision/blob/HEAD/big_vision/configs/proj/paligemma/transfers/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"OPAQUE_PARAMS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"774022ae9ec0b97e"}},{"code_sha256_prefix":"e56c7b7495cab683","entry":"combine_and_keep_train","repo":"google-research/big_vision","repo_kind":"official","path":"big_vision/configs/proj/paligemma/transfers/common.py","file_url":"https://github.com/google-research/big_vision/blob/HEAD/big_vision/configs/proj/paligemma/transfers/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"OPAQUE_PARAMS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e56c7b7495cab683"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}