{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ureader-universal-ocr-free-visually-situated","title":"UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model","arxiv_id":"2310.05126","date":"2023-10-08","proceeding":null,"authors":["Jiabo Ye","Anwen Hu","Haiyang Xu","Qinghao Ye","Ming Yan","Guohai Xu","Chenliang Li","Junfeng Tian","Qi Qian","Ji Zhang","Qin Jin","Liang He","Xin Alex Lin","Fei Huang"],"abstract":"Text is ubiquitous in our visual world, conveying crucial information, such as in documents, websites, and everyday photographs. In this work, we propose UReader, a first exploration of universal OCR-free visually-situated language understanding based on the Multimodal Large Language Model (MLLM). By leveraging the shallow text recognition ability of the MLLM, we only finetuned 1.2% parameters and the training cost is much lower than previous work following domain-specific pretraining and finetuning paradigms. Concretely, UReader is jointly finetuned on a wide range of Visually-situated Language Understanding tasks via a unified instruction format. To enhance the visual text and semantic understanding, we further apply two auxiliary tasks with the same format, namely text reading and key points generation tasks. We design a shape-adaptive cropping module before the encoder-decoder architecture of MLLM to leverage the frozen low-resolution vision encoder for processing high-resolution images. Without downstream finetuning, our single model achieves state-of-the-art ocr-free performance in 8 out of 10 visually-situated language understanding tasks, across 5 domains: documents, tables, charts, natural images, and webpage screenshots. Codes and instruction-tuning datasets will be released.","url_abs":"https://arxiv.org/abs/2310.05126v1","url_pdf":"https://arxiv.org/pdf/2310.05126v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ureader-universal-ocr-free-visually-situated","repo_url":"https://github.com/lukeforeveryoung/ureader","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"multimodal-large-language-model","task_name":"Multimodal Large Language Model"},{"task_slug":"optical-character-recognition","task_name":"Optical Character Recognition (OCR)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.05126","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.05126"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lukeforeveryoung/ureader","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":9,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":11,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d05ffe7e5265888d","entry":"batchify","repo":"lukeforeveryoung/ureader","repo_kind":"official","path":"pipeline/utils.py","file_url":"https://github.com/lukeforeveryoung/ureader/blob/HEAD/pipeline/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d05ffe7e5265888d"}},{"code_sha256_prefix":"649f50025297c2b9","entry":"collate_fn","repo":"lukeforeveryoung/ureader","repo_kind":"official","path":"pipeline/evaluation.py","file_url":"https://github.com/lukeforeveryoung/ureader/blob/HEAD/pipeline/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"649f50025297c2b9"}},{"code_sha256_prefix":"45711ccd8932d38a","entry":"detokenize_generations","repo":"lukeforeveryoung/ureader","repo_kind":"official","path":"mplug_owl/processing_mplug_owl.py","file_url":"https://github.com/lukeforeveryoung/ureader/blob/HEAD/mplug_owl/processing_mplug_owl.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"45711ccd8932d38a"}},{"code_sha256_prefix":"104beed5dd489b98","entry":"do_generate","repo":"lukeforeveryoung/ureader","repo_kind":"official","path":"pipeline/interface.py","file_url":"https://github.com/lukeforeveryoung/ureader/blob/HEAD/pipeline/interface.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"104beed5dd489b98"}},{"code_sha256_prefix":"97b29adf427aa64e","entry":"get_cosine_schedule_with_warmup","repo":"lukeforeveryoung/ureader","repo_kind":"official","path":"pipeline/utils.py","file_url":"https://github.com/lukeforeveryoung/ureader/blob/HEAD/pipeline/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"97b29adf427aa64e"}},{"code_sha256_prefix":"45567e61fe09093a","entry":"get_ltor_masks_and_position_ids_from_embeddings","repo":"lukeforeveryoung/ureader","repo_kind":"official","path":"mplug_owl/modeling_mplug_owl.py","file_url":"https://github.com/lukeforeveryoung/ureader/blob/HEAD/mplug_owl/modeling_mplug_owl.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"45567e61fe09093a"}},{"code_sha256_prefix":"53b3ed9d6fed2b5f","entry":"get_param_groups","repo":"lukeforeveryoung/ureader","repo_kind":"official","path":"pipeline/utils.py","file_url":"https://github.com/lukeforeveryoung/ureader/blob/HEAD/pipeline/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"53b3ed9d6fed2b5f"}},{"code_sha256_prefix":"15e54d5d53ab90ac","entry":"post_process_code","repo":"lukeforeveryoung/ureader","repo_kind":"official","path":"serve/model_utils.py","file_url":"https://github.com/lukeforeveryoung/ureader/blob/HEAD/serve/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"15e54d5d53ab90ac"}},{"code_sha256_prefix":"607a861da4671ce5","entry":"post_process_output","repo":"lukeforeveryoung/ureader","repo_kind":"official","path":"serve/model_utils.py","file_url":"https://github.com/lukeforeveryoung/ureader/blob/HEAD/serve/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"607a861da4671ce5"}},{"code_sha256_prefix":"a314d930b7b0cfbb","entry":"tokenize_prompts","repo":"lukeforeveryoung/ureader","repo_kind":"official","path":"mplug_owl/processing_mplug_owl.py","file_url":"https://github.com/lukeforeveryoung/ureader/blob/HEAD/mplug_owl/processing_mplug_owl.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a314d930b7b0cfbb"}},{"code_sha256_prefix":"96dde8a246fae2ac","entry":"get_media_indices","repo":"lukeforeveryoung/ureader","repo_kind":"official","path":"mplug_owl/modeling_mplug_owl.py","file_url":"https://github.com/lukeforeveryoung/ureader/blob/HEAD/mplug_owl/modeling_mplug_owl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"96dde8a246fae2ac"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}