{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/layout-and-task-aware-instruction-prompt-for","title":"Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answering","arxiv_id":"2306.00526","date":"2023-06-01","proceeding":null,"authors":["Wenjin Wang","Yunhao Li","Yixin Ou","Yin Zhang"],"abstract":"Layout-aware pre-trained models has achieved significant progress on document image question answering. They introduce extra learnable modules into existing language models to capture layout information within document images from text bounding box coordinates obtained by OCR tools. However, extra modules necessitate pre-training on extensive document images. This prevents these methods from directly utilizing off-the-shelf instruction-tuning language foundation models, which have recently shown promising potential in zero-shot learning. Instead, in this paper, we find that instruction-tuning language models like Claude and ChatGPT can understand layout by spaces and line breaks. Based on this observation, we propose the LAyout and Task aware Instruction Prompt (LATIN-Prompt), which consists of layout-aware document content and task-aware instruction. Specifically, the former uses appropriate spaces and line breaks to recover the layout information among text segments obtained by OCR tools, and the latter ensures that generated answers adhere to formatting requirements. Moreover, we propose the LAyout and Task aware Instruction Tuning (LATIN-Tuning) to improve the performance of small instruction-tuning models like Alpaca. Experimental results show that LATIN-Prompt enables zero-shot performance of Claude and ChatGPT to be comparable to the fine-tuning performance of SOTAs on document image question answering, and LATIN-Tuning enhances the zero-shot performance of Alpaca significantly. For example, LATIN-Prompt improves the performance of Claude and ChatGPT on DocVQA by 263% and 20% respectively. LATIN-Tuning improves the performance of Alpaca on DocVQA by 87.7%. Quantitative and qualitative analyses demonstrate the effectiveness of LATIN-Prompt and LATIN-Tuning. We provide the code in supplementary and will release it to facilitate future research.","url_abs":"https://arxiv.org/abs/2306.00526v4","url_pdf":"https://arxiv.org/pdf/2306.00526v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"layout-and-task-aware-instruction-prompt-for","repo_url":"https://github.com/wenjinw/latin-prompt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"layout-and-task-aware-instruction-prompt-for","repo_url":"https://github.com/deepopinion/anls-star-metric","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"layout-and-task-aware-instruction-prompt-for","repo_url":"https://github.com/deepopinion/anls_star_metric","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"optical-character-recognition","task_name":"Optical Character Recognition (OCR)"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"aware","method_name":"AWARE"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-docvqa-test","task":"Visual Question Answering (VQA)","dataset":"DocVQA test","model":"GPT-4","rank_in_archive_order":9,"of":33,"metrics":{"ANLS":"0.884"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-docvqa-test","task":"Visual Question Answering (VQA)","dataset":"DocVQA test","model":"Claude + LATIN-Prompt","rank_in_archive_order":19,"of":33,"metrics":{"ANLS":"0.8336"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-docvqa-test","task":"Visual Question Answering (VQA)","dataset":"DocVQA test","model":"GPT-3.5 + LATIN-Prompt","rank_in_archive_order":20,"of":33,"metrics":{"ANLS":"0.8255"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-vqa-on","task":"Visual Question Answering (VQA)","dataset":"InfographicVQA","model":"Claude + LATIN-Prompt","rank_in_archive_order":11,"of":21,"metrics":{"ANLS":"54.51"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-vqa-on","task":"Visual Question Answering (VQA)","dataset":"InfographicVQA","model":"GPT-3.5 + LATIN-Prompt","rank_in_archive_order":14,"of":21,"metrics":{"ANLS":"48.98"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.00526","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.00526"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepopinion/anls-star-metric","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wenjinw/latin-prompt","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepopinion/anls_star_metric","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":9},"by_repo_kind":{"official":{"samples":8,"ran":1,"repositories":1},"listed":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c23469a99394f25f","entry":"normalize_bbox","repo":"wenjinw/latin-prompt","repo_kind":"official","path":"utils/docvqa.py","file_url":"https://github.com/wenjinw/latin-prompt/blob/HEAD/utils/docvqa.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c23469a99394f25f"}},{"code_sha256_prefix":"9741f65bce116c5a","entry":"compute_l2_dist","repo":"wenjinw/latin-prompt","repo_kind":"official","path":"utils/util.py","file_url":"https://github.com/wenjinw/latin-prompt/blob/HEAD/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9741f65bce116c5a"}},{"code_sha256_prefix":"e5d4f184819c330c","entry":"construct_nested_dict","repo":"deepopinion/anls-star-metric","repo_kind":"listed","path":"src/anls_star/key_scores_utils.py","file_url":"https://github.com/deepopinion/anls-star-metric/blob/HEAD/src/anls_star/key_scores_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e5d4f184819c330c"}},{"code_sha256_prefix":"c480c7bac03cd36f","entry":"cos_sim","repo":"wenjinw/latin-prompt","repo_kind":"official","path":"utils/util.py","file_url":"https://github.com/wenjinw/latin-prompt/blob/HEAD/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c480c7bac03cd36f"}},{"code_sha256_prefix":"87fed178011a98e9","entry":"document_sim","repo":"wenjinw/latin-prompt","repo_kind":"official","path":"utils/util.py","file_url":"https://github.com/wenjinw/latin-prompt/blob/HEAD/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87fed178011a98e9"}},{"code_sha256_prefix":"db4bf2624e9f59ae","entry":"load_image","repo":"wenjinw/latin-prompt","repo_kind":"official","path":"utils/docvqa.py","file_url":"https://github.com/wenjinw/latin-prompt/blob/HEAD/utils/docvqa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db4bf2624e9f59ae"}},{"code_sha256_prefix":"a2e70e7ad97878ef","entry":"merge_and_calculate_mean","repo":"deepopinion/anls-star-metric","repo_kind":"listed","path":"src/anls_star/key_scores_utils.py","file_url":"https://github.com/deepopinion/anls-star-metric/blob/HEAD/src/anls_star/key_scores_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a2e70e7ad97878ef"}},{"code_sha256_prefix":"f03fda2a07bb8465","entry":"openai_chat_completion","repo":"wenjinw/latin-prompt","repo_kind":"official","path":"utils/openai_api.py","file_url":"https://github.com/wenjinw/latin-prompt/blob/HEAD/utils/openai_api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f03fda2a07bb8465"}},{"code_sha256_prefix":"430f1617115f7078","entry":"openai_completion","repo":"wenjinw/latin-prompt","repo_kind":"official","path":"utils/openai_api.py","file_url":"https://github.com/wenjinw/latin-prompt/blob/HEAD/utils/openai_api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"430f1617115f7078"}},{"code_sha256_prefix":"ec25301365f6d685","entry":"quad_to_box","repo":"wenjinw/latin-prompt","repo_kind":"official","path":"utils/docvqa.py","file_url":"https://github.com/wenjinw/latin-prompt/blob/HEAD/utils/docvqa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec25301365f6d685"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}