{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cambrian-1-a-fully-open-vision-centric","title":"Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs","arxiv_id":"2406.16860","date":"2024-06-24","proceeding":null,"authors":["Shengbang Tong","Ellis Brown","Penghao Wu","Sanghyun Woo","Manoj Middepogu","Sai Charitha Akula","Jihan Yang","Shusheng Yang","Adithya Iyer","Xichen Pan","Ziteng Wang","Rob Fergus","Yann Lecun","Saining Xie"],"abstract":"We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations, offering new insights into different models and architectures -- self-supervised, strongly supervised, or combinations thereof -- based on experiments with over 20 vision encoders. We critically examine existing MLLM benchmarks, address the difficulties involved in consolidating and interpreting results from various tasks, and introduce a new vision-centric benchmark, CV-Bench. To further improve visual grounding, we propose the Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that integrates high-resolution vision features with LLMs while reducing the number of tokens. Additionally, we discuss the curation of high-quality visual instruction-tuning data from publicly available sources, emphasizing the importance of data source balancing and distribution ratio. Collectively, Cambrian-1 not only achieves state-of-the-art performance but also serves as a comprehensive, open cookbook for instruction-tuned MLLMs. We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes. We hope our release will inspire and accelerate advancements in multimodal systems and visual representation learning.","url_abs":"https://arxiv.org/abs/2406.16860v2","url_pdf":"https://arxiv.org/pdf/2406.16860v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cambrian-1-a-fully-open-vision-centric","repo_url":"https://github.com/cambrian-mllm/cambrian","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"}],"methods":[],"datasets_introduced":[{"slug":"cv-bench","name":"CV-Bench","full_name":"Cambrian Vision-Centric Benchmark"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.16860","atlas_url":"https://app.syntology.ai/?focus=2406.16860","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.16860"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cambrian-mllm/cambrian","reach":null}],"summary":{"ran_fixture":5,"ran":3,"ran_honours":2,"unverified":1},"by_repo_kind":{"official":{"samples":11,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7bca1adaded75dc9","entry":"window_unpartition","repo":"cambrian-mllm/cambrian","repo_kind":"official","path":"cambrian/model/multimodal_encoder/sam/encoder.py","file_url":"https://github.com/cambrian-mllm/cambrian/blob/HEAD/cambrian/model/multimodal_encoder/sam/encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7bca1adaded75dc9"}},{"code_sha256_prefix":"9b244ed4de98b8b3","entry":"AggregationBlock","repo":"cambrian-mllm/cambrian","repo_kind":"official","path":"cambrian/model/vision_sampler.py","file_url":"https://github.com/cambrian-mllm/cambrian/blob/HEAD/cambrian/model/vision_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b244ed4de98b8b3"}},{"code_sha256_prefix":"7f1838302ff8b2bc","entry":"CrossAttention","repo":"cambrian-mllm/cambrian","repo_kind":"official","path":"cambrian/model/vision_sampler.py","file_url":"https://github.com/cambrian-mllm/cambrian/blob/HEAD/cambrian/model/vision_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f1838302ff8b2bc"}},{"code_sha256_prefix":"a65d55cf801fa5fb","entry":"MLP","repo":"cambrian-mllm/cambrian","repo_kind":"official","path":"cambrian/model/vision_sampler.py","file_url":"https://github.com/cambrian-mllm/cambrian/blob/HEAD/cambrian/model/vision_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a65d55cf801fa5fb"}},{"code_sha256_prefix":"0970b04a87d6849f","entry":"get_chunk","repo":"cambrian-mllm/cambrian","repo_kind":"official","path":"eval/eval/omni/omni_eval.py","file_url":"https://github.com/cambrian-mllm/cambrian/blob/HEAD/eval/eval/omni/omni_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0970b04a87d6849f"}},{"code_sha256_prefix":"733d7f0bedcb74c2","entry":"get_rel_pos","repo":"cambrian-mllm/cambrian","repo_kind":"official","path":"cambrian/model/multimodal_encoder/sam/encoder.py","file_url":"https://github.com/cambrian-mllm/cambrian/blob/HEAD/cambrian/model/multimodal_encoder/sam/encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"733d7f0bedcb74c2"}},{"code_sha256_prefix":"9874efcce8755c53","entry":"split_list","repo":"cambrian-mllm/cambrian","repo_kind":"official","path":"eval/eval/omni/omni_eval.py","file_url":"https://github.com/cambrian-mllm/cambrian/blob/HEAD/eval/eval/omni/omni_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9874efcce8755c53"}},{"code_sha256_prefix":"fd89ea82cae32f5d","entry":"unmask_attention_mask","repo":"cambrian-mllm/cambrian","repo_kind":"official","path":"cambrian/model/cambrian_arch.py","file_url":"https://github.com/cambrian-mllm/cambrian/blob/HEAD/cambrian/model/cambrian_arch.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fd89ea82cae32f5d"}},{"code_sha256_prefix":"83d2977826140fdc","entry":"unpad_image","repo":"cambrian-mllm/cambrian","repo_kind":"official","path":"cambrian/model/cambrian_arch.py","file_url":"https://github.com/cambrian-mllm/cambrian/blob/HEAD/cambrian/model/cambrian_arch.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"83d2977826140fdc"}},{"code_sha256_prefix":"5693c99bfc32e135","entry":"window_partition","repo":"cambrian-mllm/cambrian","repo_kind":"official","path":"cambrian/model/multimodal_encoder/sam/encoder.py","file_url":"https://github.com/cambrian-mllm/cambrian/blob/HEAD/cambrian/model/multimodal_encoder/sam/encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5693c99bfc32e135"}},{"code_sha256_prefix":"b25d6d325c57c2a8","entry":"VisionAggregationLayer","repo":"cambrian-mllm/cambrian","repo_kind":"official","path":"cambrian/model/vision_sampler.py","file_url":"https://github.com/cambrian-mllm/cambrian/blob/HEAD/cambrian/model/vision_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b25d6d325c57c2a8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}