{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pali-x-on-scaling-up-a-multilingual-vision","title":"PaLI-X: On Scaling up a Multilingual Vision and Language Model","arxiv_id":"2305.18565","date":"2023-05-29","proceeding":null,"authors":["Xi Chen","Josip Djolonga","Piotr Padlewski","Basil Mustafa","Soravit Changpinyo","Jialin Wu","Carlos Riquelme Ruiz","Sebastian Goodman","Xiao Wang","Yi Tay","Siamak Shakeri","Mostafa Dehghani","Daniel Salz","Mario Lucic","Michael Tschannen","Arsha Nagrani","Hexiang Hu","Mandar Joshi","Bo Pang","Ceslee Montgomery","Paulina Pietrzyk","Marvin Ritter","AJ Piergiovanni","Matthias Minderer","Filip Pavetic","Austin Waters","Gang Li","Ibrahim Alabdulmohsin","Lucas Beyer","Julien Amelot","Kenton Lee","Andreas Peter Steiner","Yang Li","Daniel Keysers","Anurag Arnab","Yuanzhong Xu","Keran Rong","Alexander Kolesnikov","Mojtaba Seyedhosseini","Anelia Angelova","Xiaohua Zhai","Neil Houlsby","Radu Soricut"],"abstract":"We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-shot (in-context) learning, as well as object detection, video question answering, and video captioning. PaLI-X advances the state-of-the-art on most vision-and-language benchmarks considered (25+ of them). Finally, we observe emerging capabilities, such as complex counting and multilingual object detection, tasks that are not explicitly in the training mix.","url_abs":"https://arxiv.org/abs/2305.18565v1","url_pdf":"https://arxiv.org/pdf/2305.18565v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pali-x-on-scaling-up-a-multilingual-vision","repo_url":"https://github.com/doc-doc/NExT-OE","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"pali-x-on-scaling-up-a-multilingual-vision","repo_url":"https://github.com/kyegomez/PALI","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"chart-question-answering","task_name":"Chart Question Answering"},{"task_slug":"fine-grained-image-recognition","task_name":"Fine-Grained Image Recognition"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"temporal-casual-qa","task_name":"Temporal/Casual QA"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"document-understanding","task_name":"document understanding"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/chart-question-answering-on-chartqa","task":"Chart Question Answering","dataset":"ChartQA","model":"PaLI-X (Single-task FT w/ OCR)","rank_in_archive_order":10,"of":27,"metrics":{"1:1 Accuracy":"72.3"},"uses_additional_data":true},{"leaderboard":"/sota/chart-question-answering-on-chartqa","task":"Chart Question Answering","dataset":"ChartQA","model":"PaLI-X (Single-task FT)","rank_in_archive_order":11,"of":27,"metrics":{"1:1 Accuracy":"70.9"},"uses_additional_data":true},{"leaderboard":"/sota/chart-question-answering-on-chartqa","task":"Chart Question Answering","dataset":"ChartQA","model":"PaLI-X (Multi-task FT)","rank_in_archive_order":12,"of":27,"metrics":{"1:1 Accuracy":"70.6"},"uses_additional_data":true},{"leaderboard":"/sota/fine-grained-image-recognition-on-oven","task":"Fine-Grained Image Recognition","dataset":"OVEN","model":"PaLI-X","rank_in_archive_order":1,"of":5,"metrics":{"Accuracy":"23.1"},"uses_additional_data":false},{"leaderboard":"/sota/temporal-casual-qa-on-next-qa","task":"Temporal/Casual QA","dataset":"NExT-QA","model":"PaLI-X","rank_in_archive_order":1,"of":8,"metrics":{"WUPS":"38.3"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-docvqa-test","task":"Visual Question Answering (VQA)","dataset":"DocVQA test","model":"PaLI-X (Single-task FT w/ OCR)","rank_in_archive_order":14,"of":33,"metrics":{"ANLS":"0.868"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-docvqa-test","task":"Visual Question Answering (VQA)","dataset":"DocVQA test","model":"PaLI-X (Multi-task FT)","rank_in_archive_order":21,"of":33,"metrics":{"ANLS":"0.809"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-docvqa-test","task":"Visual Question Answering (VQA)","dataset":"DocVQA test","model":"PaLI-X (Single-task FT)","rank_in_archive_order":23,"of":33,"metrics":{"ANLS":"0.80"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-vqa-on-infoseek","task":"Visual Question Answering (VQA)","dataset":"InfoSeek","model":"PaLI-X","rank_in_archive_order":2,"of":7,"metrics":{"Accuracy":"24"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-vqa-on","task":"Visual Question Answering (VQA)","dataset":"InfographicVQA","model":"PaLI-X (Single-task FT w/ OCR)","rank_in_archive_order":10,"of":21,"metrics":{"ANLS":"54.8"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-vqa-on","task":"Visual Question Answering (VQA)","dataset":"InfographicVQA","model":"PaLI-X (Multi-task FT)","rank_in_archive_order":12,"of":21,"metrics":{"ANLS":"50.7"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-vqa-on","task":"Visual Question Answering (VQA)","dataset":"InfographicVQA","model":"PaLI-X (Single-task FT)","rank_in_archive_order":13,"of":21,"metrics":{"ANLS":"49.2"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-ok-vqa","task":"Visual Question Answering (VQA)","dataset":"OK-VQA","model":"PaLI-X (Single-task FT)","rank_in_archive_order":3,"of":37,"metrics":{"Accuracy":"66.1"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.18565","atlas_url":"https://app.syntology.ai/?focus=2305.18565","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.18565"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/doc-doc/NExT-OE","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kyegomez/PALI","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":4,"ran_honours":1,"ran_draft_wrong":1,"ran":1},"by_repo_kind":{"listed":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1f7c550e65142ef8","entry":"cast_tuple","repo":"kyegomez/PALI","repo_kind":"listed","path":"pali/transformer.py","file_url":"https://github.com/kyegomez/PALI/blob/HEAD/pali/transformer.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1f7c550e65142ef8"}},{"code_sha256_prefix":"4a06bf1ac86f840a","entry":"compact","repo":"kyegomez/PALI","repo_kind":"listed","path":"pali/attend.py","file_url":"https://github.com/kyegomez/PALI/blob/HEAD/pali/attend.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a06bf1ac86f840a"}},{"code_sha256_prefix":"60fff7c3c400d7ff","entry":"default","repo":"kyegomez/PALI","repo_kind":"listed","path":"pali/attend.py","file_url":"https://github.com/kyegomez/PALI/blob/HEAD/pali/attend.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60fff7c3c400d7ff"}},{"code_sha256_prefix":"424012cb37b31172","entry":"default","repo":"kyegomez/PALI","repo_kind":"listed","path":"pali/transformer.py","file_url":"https://github.com/kyegomez/PALI/blob/HEAD/pali/transformer.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"424012cb37b31172"}},{"code_sha256_prefix":"5444a7d75d38878d","entry":"eval_decorator","repo":"kyegomez/PALI","repo_kind":"listed","path":"pali/autoregressive_wrapper.py","file_url":"https://github.com/kyegomez/PALI/blob/HEAD/pali/autoregressive_wrapper.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5444a7d75d38878d"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"kyegomez/PALI","repo_kind":"listed","path":"pali/attend.py","file_url":"https://github.com/kyegomez/PALI/blob/HEAD/pali/attend.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"864f3eecd1ea0a36","entry":"top_p","repo":"kyegomez/PALI","repo_kind":"listed","path":"pali/autoregressive_wrapper.py","file_url":"https://github.com/kyegomez/PALI/blob/HEAD/pali/autoregressive_wrapper.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"864f3eecd1ea0a36"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}