{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/monkey-image-resolution-and-text-label-are","title":"Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models","arxiv_id":"2311.06607","date":"2023-11-11","proceeding":"CVPR 2024 1","authors":["Zhang Li","Biao Yang","Qiang Liu","Zhiyin Ma","Shuo Zhang","Jingxu Yang","Yabo Sun","Yuliang Liu","Xiang Bai"],"abstract":"Large Multimodal Models (LMMs) have shown promise in vision-language tasks but struggle with high-resolution input and detailed scene understanding. Addressing these challenges, we introduce Monkey to enhance LMM capabilities. Firstly, Monkey processes input images by dividing them into uniform patches, each matching the size (e.g., 448x448) used in the original training of the well-trained vision encoder. Equipped with individual adapter for each patch, Monkey can handle higher resolutions up to 1344x896 pixels, enabling the detailed capture of complex visual information. Secondly, it employs a multi-level description generation method, enriching the context for scene-object associations. This two-part strategy ensures more effective learning from generated data: the higher resolution allows for a more detailed capture of visuals, which in turn enhances the effectiveness of comprehensive descriptions. Extensive ablative results validate the effectiveness of our designs. Additionally, experiments on 18 datasets further demonstrate that Monkey surpasses existing LMMs in many tasks like Image Captioning and various Visual Question Answering formats. Specially, in qualitative tests focused on dense text question answering, Monkey has exhibited encouraging results compared with GPT4V. Code is available at https://github.com/Yuliang-Liu/Monkey.","url_abs":"https://arxiv.org/abs/2311.06607v4","url_pdf":"https://arxiv.org/pdf/2311.06607v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"monkey-image-resolution-and-text-label-are","repo_url":"https://github.com/yuliang-liu/monkey","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"mmr-total","task_name":"MMR total"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"high-resolution-input","method_name":"High-resolution input"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/mmr-total-on-mrr-benchmark","task":"MMR total","dataset":"MRR-Benchmark","model":"Monkey-Chat-7B","rank_in_archive_order":13,"of":14,"metrics":{"Total Column Score":"214"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.06607","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.06607"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuliang-liu/monkey","reach":null}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"17d2b38da4eaebf2","entry":"get_feature","repo":"yuliang-liu/monkey","repo_kind":"official","path":"data_generation/grit/grit/modeling/meta_arch/grit.py","file_url":"https://github.com/yuliang-liu/monkey/blob/HEAD/data_generation/grit/grit/modeling/meta_arch/grit.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17d2b38da4eaebf2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}