{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/otterhd-a-high-resolution-multi-modality","title":"OtterHD: A High-Resolution Multi-modality Model","arxiv_id":"2311.04219","date":"2023-11-07","proceeding":null,"authors":["Bo Li","Peiyuan Zhang","Jingkang Yang","Yuanhan Zhang","Fanyi Pu","Ziwei Liu"],"abstract":"In this paper, we present OtterHD-8B, an innovative multimodal model evolved from Fuyu-8B, specifically engineered to interpret high-resolution visual inputs with granular precision. Unlike conventional models that are constrained by fixed-size vision encoders, OtterHD-8B boasts the ability to handle flexible input dimensions, ensuring its versatility across various inference requirements. Alongside this model, we introduce MagnifierBench, an evaluation framework designed to scrutinize models' ability to discern minute details and spatial relationships of small objects. Our comparative analysis reveals that while current leading models falter on this benchmark, OtterHD-8B, particularly when directly processing high-resolution inputs, outperforms its counterparts by a substantial margin. The findings illuminate the structural variances in visual information processing among different models and the influence that the vision encoders' pre-training resolution disparities have on model effectiveness within such benchmarks. Our study highlights the critical role of flexibility and high-resolution input capabilities in large multimodal models and also exemplifies the potential inherent in the Fuyu architecture's simplicity for handling complex visual data.","url_abs":"https://arxiv.org/abs/2311.04219v1","url_pdf":"https://arxiv.org/pdf/2311.04219v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"otterhd-a-high-resolution-multi-modality","repo_url":"https://github.com/luodian/otter","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"high-resolution-input","method_name":"High-resolution input"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"OtterHD-8B","rank_in_archive_order":219,"of":231,"metrics":{"GPT-4 score":"26.3","Params":"8B"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.04219","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.04219"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luodian/otter","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6ef9ea8bea823588","entry":"dropout_add","repo":"luodian/otter","repo_kind":"official","path":"src/otter_ai/models/falcon/modelling_RW.py","file_url":"https://github.com/luodian/otter/blob/HEAD/src/otter_ai/models/falcon/modelling_RW.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ef9ea8bea823588"}},{"code_sha256_prefix":"029892ebb4321050","entry":"extract_frames","repo":"luodian/otter","repo_kind":"official","path":"pipeline/demos/interactive/otter_video.py","file_url":"https://github.com/luodian/otter/blob/HEAD/pipeline/demos/interactive/otter_video.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"029892ebb4321050"}},{"code_sha256_prefix":"dcd422d66b0581d8","entry":"get_cast_dtype","repo":"luodian/otter","repo_kind":"official","path":"pipeline/benchmarks/public_datasets_suite/models/otter.py","file_url":"https://github.com/luodian/otter/blob/HEAD/pipeline/benchmarks/public_datasets_suite/models/otter.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dcd422d66b0581d8"}},{"code_sha256_prefix":"3614ed4764f65f90","entry":"get_content_type","repo":"luodian/otter","repo_kind":"official","path":"pipeline/demos/interactive/otter_image.py","file_url":"https://github.com/luodian/otter/blob/HEAD/pipeline/demos/interactive/otter_image.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3614ed4764f65f90"}},{"code_sha256_prefix":"34167f78172c1abe","entry":"get_formatted_forward_prompt","repo":"luodian/otter","repo_kind":"official","path":"pipeline/benchmarks/models/otter_image.py","file_url":"https://github.com/luodian/otter/blob/HEAD/pipeline/benchmarks/models/otter_image.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"34167f78172c1abe"}},{"code_sha256_prefix":"b48b9a381723c691","entry":"get_formatted_prompt","repo":"luodian/otter","repo_kind":"official","path":"pipeline/benchmarks/models/otter_image.py","file_url":"https://github.com/luodian/otter/blob/HEAD/pipeline/benchmarks/models/otter_image.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b48b9a381723c691"}},{"code_sha256_prefix":"53f85a97033ac702","entry":"get_formatted_prompt","repo":"luodian/otter","repo_kind":"official","path":"pipeline/demos/interactive/otter_image_incontext.py","file_url":"https://github.com/luodian/otter/blob/HEAD/pipeline/demos/interactive/otter_image_incontext.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53f85a97033ac702"}},{"code_sha256_prefix":"fe65bfdd78032556","entry":"get_image","repo":"luodian/otter","repo_kind":"official","path":"pipeline/demos/interactive/otter_image.py","file_url":"https://github.com/luodian/otter/blob/HEAD/pipeline/demos/interactive/otter_image.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe65bfdd78032556"}},{"code_sha256_prefix":"bdd50d5971d6d069","entry":"get_image","repo":"luodian/otter","repo_kind":"official","path":"pipeline/demos/interactive/otter_video.py","file_url":"https://github.com/luodian/otter/blob/HEAD/pipeline/demos/interactive/otter_video.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bdd50d5971d6d069"}},{"code_sha256_prefix":"9679b4832e5de332","entry":"get_pil_image","repo":"luodian/otter","repo_kind":"official","path":"pipeline/benchmarks/models/otter_image.py","file_url":"https://github.com/luodian/otter/blob/HEAD/pipeline/benchmarks/models/otter_image.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9679b4832e5de332"}},{"code_sha256_prefix":"2502f906145be7f6","entry":"rotate_half","repo":"luodian/otter","repo_kind":"official","path":"src/otter_ai/models/falcon/modelling_RW.py","file_url":"https://github.com/luodian/otter/blob/HEAD/src/otter_ai/models/falcon/modelling_RW.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2502f906145be7f6"}},{"code_sha256_prefix":"b4acc4f0962c97ca","entry":"build_alibi_tensor","repo":"luodian/otter","repo_kind":"official","path":"src/otter_ai/models/falcon/modelling_RW.py","file_url":"https://github.com/luodian/otter/blob/HEAD/src/otter_ai/models/falcon/modelling_RW.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b4acc4f0962c97ca"}},{"code_sha256_prefix":"94a4076103662faf","entry":"get_autocast","repo":"luodian/otter","repo_kind":"official","path":"pipeline/benchmarks/public_datasets_suite/models/otter.py","file_url":"https://github.com/luodian/otter/blob/HEAD/pipeline/benchmarks/public_datasets_suite/models/otter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94a4076103662faf"}},{"code_sha256_prefix":"8dab437d86150725","entry":"get_image","repo":"luodian/otter","repo_kind":"official","path":"pipeline/demos/interactive/otter_image_incontext.py","file_url":"https://github.com/luodian/otter/blob/HEAD/pipeline/demos/interactive/otter_image_incontext.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8dab437d86150725"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}