{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deepseek-vl-towards-real-world-vision","title":"DeepSeek-VL: Towards Real-World Vision-Language Understanding","arxiv_id":"2403.05525","date":"2024-03-08","proceeding":null,"authors":["Haoyu Lu","Wen Liu","Bo Zhang","Bingxuan Wang","Kai Dong","Bo Liu","Jingxiang Sun","Tongzheng Ren","Zhuoshu Li","Hao Yang","Yaofeng Sun","Chengqi Deng","Hanwei Xu","Zhenda Xie","Chong Ruan"],"abstract":"We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. Our approach is structured around three key dimensions: We strive to ensure our data is diverse, scalable, and extensively covers real-world scenarios including web screenshots, PDFs, OCR, charts, and knowledge-based content, aiming for a comprehensive representation of practical contexts. Further, we create a use case taxonomy from real user scenarios and construct an instruction tuning dataset accordingly. The fine-tuning with this dataset substantially improves the model's user experience in practical applications. Considering efficiency and the demands of most real-world scenarios, DeepSeek-VL incorporates a hybrid vision encoder that efficiently processes high-resolution images (1024 x 1024), while maintaining a relatively low computational overhead. This design choice ensures the model's ability to capture critical semantic and detailed information across various visual tasks. We posit that a proficient Vision-Language Model should, foremost, possess strong language abilities. To ensure the preservation of LLM capabilities during pretraining, we investigate an effective VL pretraining strategy by integrating LLM training from the beginning and carefully managing the competitive dynamics observed between vision and language modalities. The DeepSeek-VL family (both 1.3B and 7B models) showcases superior user experiences as a vision-language chatbot in real-world applications, achieving state-of-the-art or competitive performance across a wide range of visual-language benchmarks at the same model size while maintaining robust performance on language-centric benchmarks. We have made both 1.3B and 7B models publicly accessible to foster innovations based on this foundation model.","url_abs":"https://arxiv.org/abs/2403.05525v2","url_pdf":"https://arxiv.org/pdf/2403.05525v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deepseek-vl-towards-real-world-vision","repo_url":"https://github.com/deepseek-ai/deepseek-vl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"chatbot","task_name":"Chatbot"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"multimodal-deep-learning","task_name":"Multimodal Deep Learning"},{"task_slug":"optical-character-recognition","task_name":"Optical Character Recognition (OCR)"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"DeepSeek-VL","rank_in_archive_order":101,"of":231,"metrics":{"GPT-4 score":"41.5"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2403.05525","atlas_url":"https://app.syntology.ai/?focus=2403.05525","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.05525"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepseek-ai/deepseek-vl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":3,"ran":3,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7bca1adaded75dc9","entry":"window_unpartition","repo":"deepseek-ai/deepseek-vl","repo_kind":"official","path":"deepseek_vl/models/sam.py","file_url":"https://github.com/deepseek-ai/deepseek-vl/blob/HEAD/deepseek_vl/models/sam.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7bca1adaded75dc9"}},{"code_sha256_prefix":"a63c3596d1105d5d","entry":"create_siglip_vit","repo":"deepseek-ai/deepseek-vl","repo_kind":"official","path":"deepseek_vl/models/siglip_vit.py","file_url":"https://github.com/deepseek-ai/deepseek-vl/blob/HEAD/deepseek_vl/models/siglip_vit.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a63c3596d1105d5d"}},{"code_sha256_prefix":"592b3c1a88f93d7c","entry":"expand2square","repo":"deepseek-ai/deepseek-vl","repo_kind":"official","path":"deepseek_vl/models/image_processing_vlm.py","file_url":"https://github.com/deepseek-ai/deepseek-vl/blob/HEAD/deepseek_vl/models/image_processing_vlm.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"592b3c1a88f93d7c"}},{"code_sha256_prefix":"733d7f0bedcb74c2","entry":"get_rel_pos","repo":"deepseek-ai/deepseek-vl","repo_kind":"official","path":"deepseek_vl/models/sam.py","file_url":"https://github.com/deepseek-ai/deepseek-vl/blob/HEAD/deepseek_vl/models/sam.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"733d7f0bedcb74c2"}},{"code_sha256_prefix":"10ce40a3d514f69a","entry":"to_gradio_chatbot","repo":"deepseek-ai/deepseek-vl","repo_kind":"official","path":"deepseek_vl/serve/app_deepseek.py","file_url":"https://github.com/deepseek-ai/deepseek-vl/blob/HEAD/deepseek_vl/serve/app_deepseek.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10ce40a3d514f69a"}},{"code_sha256_prefix":"f6e02412cf1ee318","entry":"to_gradio_history","repo":"deepseek-ai/deepseek-vl","repo_kind":"official","path":"deepseek_vl/serve/app_deepseek.py","file_url":"https://github.com/deepseek-ai/deepseek-vl/blob/HEAD/deepseek_vl/serve/app_deepseek.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6e02412cf1ee318"}},{"code_sha256_prefix":"5693c99bfc32e135","entry":"window_partition","repo":"deepseek-ai/deepseek-vl","repo_kind":"official","path":"deepseek_vl/models/sam.py","file_url":"https://github.com/deepseek-ai/deepseek-vl/blob/HEAD/deepseek_vl/models/sam.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5693c99bfc32e135"}},{"code_sha256_prefix":"ba86a364bce26e9b","entry":"compact_text_chunks","repo":"deepseek-ai/deepseek-vl","repo_kind":"official","path":"deepseek_vl/serve/app_modules/overwrites.py","file_url":"https://github.com/deepseek-ai/deepseek-vl/blob/HEAD/deepseek_vl/serve/app_modules/overwrites.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba86a364bce26e9b"}},{"code_sha256_prefix":"75628f3739d6fcd3","entry":"format_output","repo":"deepseek-ai/deepseek-vl","repo_kind":"official","path":"deepseek_vl/serve/app_modules/utils.py","file_url":"https://github.com/deepseek-ai/deepseek-vl/blob/HEAD/deepseek_vl/serve/app_modules/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75628f3739d6fcd3"}},{"code_sha256_prefix":"d1fcc287229a99c6","entry":"markdown_to_html_with_syntax_highlight","repo":"deepseek-ai/deepseek-vl","repo_kind":"official","path":"deepseek_vl/serve/app_modules/utils.py","file_url":"https://github.com/deepseek-ai/deepseek-vl/blob/HEAD/deepseek_vl/serve/app_modules/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1fcc287229a99c6"}},{"code_sha256_prefix":"f7fc8e891d83a3e1","entry":"strip_stop_words","repo":"deepseek-ai/deepseek-vl","repo_kind":"official","path":"deepseek_vl/serve/app_modules/utils.py","file_url":"https://github.com/deepseek-ai/deepseek-vl/blob/HEAD/deepseek_vl/serve/app_modules/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7fc8e891d83a3e1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}