{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pix2struct-screenshot-parsing-as-pretraining","title":"Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding","arxiv_id":"2210.03347","date":"2022-10-07","proceeding":null,"authors":["Kenton Lee","Mandar Joshi","Iulia Turc","Hexiang Hu","Fangyu Liu","Julian Eisenschlos","Urvashi Khandelwal","Peter Shaw","Ming-Wei Chang","Kristina Toutanova"],"abstract":"Visually-situated language is ubiquitous -- sources range from textbooks with diagrams to web pages with images and tables, to mobile apps with buttons and forms. Perhaps due to this diversity, previous work has typically relied on domain-specific recipes with limited sharing of the underlying data, model architectures, and objectives. We present Pix2Struct, a pretrained image-to-text model for purely visual language understanding, which can be finetuned on tasks containing visually-situated language. Pix2Struct is pretrained by learning to parse masked screenshots of web pages into simplified HTML. The web, with its richness of visual elements cleanly reflected in the HTML structure, provides a large source of pretraining data well suited to the diversity of downstream tasks. Intuitively, this objective subsumes common pretraining signals such as OCR, language modeling, image captioning. In addition to the novel pretraining strategy, we introduce a variable-resolution input representation and a more flexible integration of language and vision inputs, where language prompts such as questions are rendered directly on top of the input image. For the first time, we show that a single pretrained model can achieve state-of-the-art results in six out of nine tasks across four domains: documents, illustrations, user interfaces, and natural images.","url_abs":"https://arxiv.org/abs/2210.03347v2","url_pdf":"https://arxiv.org/pdf/2210.03347v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pix2struct-screenshot-parsing-as-pretraining","repo_url":"https://github.com/google-research/pix2struct","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"pix2struct-screenshot-parsing-as-pretraining","repo_url":"https://github.com/huggingface/transformers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"pix2struct-screenshot-parsing-as-pretraining","repo_url":"https://github.com/MindCode-4/code-2/tree/main/pix2struct","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"pix2struct-screenshot-parsing-as-pretraining","repo_url":"https://github.com/yangyucheng000/University/tree/main/model-1/pix2struct","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"chart-question-answering","task_name":"Chart Question Answering"},{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-to-text","task_name":"Image to text"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"optical-character-recognition","task_name":"Optical Character Recognition (OCR)"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/chart-question-answering-on-chartqa","task":"Chart Question Answering","dataset":"ChartQA","model":"Pix2Struct-large","rank_in_archive_order":23,"of":27,"metrics":{"1:1 Accuracy":"58.6"},"uses_additional_data":false},{"leaderboard":"/sota/chart-question-answering-on-chartqa","task":"Chart Question Answering","dataset":"ChartQA","model":"Pix2Struct-base","rank_in_archive_order":24,"of":27,"metrics":{"1:1 Accuracy":"56.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-docvqa-test","task":"Visual Question Answering (VQA)","dataset":"DocVQA test","model":"Pix2Struct-large","rank_in_archive_order":26,"of":33,"metrics":{"ANLS":"0.766"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-docvqa-test","task":"Visual Question Answering (VQA)","dataset":"DocVQA test","model":"Pix2Struct-base","rank_in_archive_order":28,"of":33,"metrics":{"ANLS":"0.721"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-vqa-on","task":"Visual Question Answering (VQA)","dataset":"InfographicVQA","model":"Pix2Struct-large","rank_in_archive_order":18,"of":21,"metrics":{"ANLS":"40"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-vqa-on","task":"Visual Question Answering (VQA)","dataset":"InfographicVQA","model":"Pix2Struct-base","rank_in_archive_order":19,"of":21,"metrics":{"ANLS":"38.2"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2210.03347","atlas_url":"https://app.syntology.ai/?focus=2210.03347","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.03347"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huggingface/transformers","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangyucheng000/University/tree/main/model-1/pix2struct","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindCode-4/code-2/tree/main/pix2struct","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-research/pix2struct","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b4928537d2a4c2e0","entry":"aggregate_metrics","repo":"google-research/pix2struct","repo_kind":"official","path":"pix2struct/metrics.py","file_url":"https://github.com/google-research/pix2struct/blob/HEAD/pix2struct/metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b4928537d2a4c2e0"}},{"code_sha256_prefix":"e80ac1aa6513e2cd","entry":"apply_single_inference","repo":"google-research/pix2struct","repo_kind":"official","path":"pix2struct/demo_utils.py","file_url":"https://github.com/google-research/pix2struct/blob/HEAD/pix2struct/demo_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e80ac1aa6513e2cd"}},{"code_sha256_prefix":"04dcc2a4874ace61","entry":"group_target","repo":"google-research/pix2struct","repo_kind":"official","path":"pix2struct/postprocessors.py","file_url":"https://github.com/google-research/pix2struct/blob/HEAD/pix2struct/postprocessors.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"04dcc2a4874ace61"}},{"code_sha256_prefix":"07dae26b6a90c397","entry":"multi_target","repo":"google-research/pix2struct","repo_kind":"official","path":"pix2struct/postprocessors.py","file_url":"https://github.com/google-research/pix2struct/blob/HEAD/pix2struct/postprocessors.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"07dae26b6a90c397"}},{"code_sha256_prefix":"b398544f82bc9630","entry":"transfer_warmup_cosine_decay_schedule","repo":"google-research/pix2struct","repo_kind":"official","path":"pix2struct/transfer_utils.py","file_url":"https://github.com/google-research/pix2struct/blob/HEAD/pix2struct/transfer_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b398544f82bc9630"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}