{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/u-llava-unifying-multi-modal-tasks-via-large","title":"u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model","arxiv_id":"2311.05348","date":"2023-11-09","proceeding":null,"authors":["Jinjin Xu","Liwu Xu","Yuzhe Yang","Xiang Li","Fanyi Wang","Yanchun Xie","Yi-Jie Huang","Yaqian Li"],"abstract":"Recent advancements in multi-modal large language models (MLLMs) have led to substantial improvements in visual understanding, primarily driven by sophisticated modality alignment strategies. However, predominant approaches prioritize global or regional comprehension, with less focus on fine-grained, pixel-level tasks. To address this gap, we introduce u-LLaVA, an innovative unifying multi-task framework that integrates pixel, regional, and global features to refine the perceptual faculties of MLLMs. We commence by leveraging an efficient modality alignment approach, harnessing both image and video datasets to bolster the model's foundational understanding across diverse visual contexts. Subsequently, a joint instruction tuning method with task-specific projectors and decoders for end-to-end downstream training is presented. Furthermore, this work contributes a novel mask-based multi-task dataset comprising 277K samples, crafted to challenge and assess the fine-grained perception capabilities of MLLMs. The overall framework is simple, effective, and achieves state-of-the-art performance across multiple benchmarks. We also make our model, data, and code publicly accessible at https://github.com/OPPOMKLab/u-LLaVA.","url_abs":"https://arxiv.org/abs/2311.05348v4","url_pdf":"https://arxiv.org/pdf/2311.05348v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"u-llava-unifying-multi-modal-tasks-via-large","repo_url":"https://github.com/OPPOMKLab/u-LLaVA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[{"slug":"ullava","name":"ullava","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.05348","atlas_url":"https://app.syntology.ai/?focus=2311.05348","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.05348"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/OPPOMKLab/u-LLaVA","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"ran_honours":1,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8b0261c8ffb51491","entry":"bbox_iou","repo":"OPPOMKLab/u-LLaVA","repo_kind":"official","path":"evaluation/tools.py","file_url":"https://github.com/OPPOMKLab/u-LLaVA/blob/HEAD/evaluation/tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b0261c8ffb51491"}},{"code_sha256_prefix":"ce749f837424dd8c","entry":"box_iou","repo":"OPPOMKLab/u-LLaVA","repo_kind":"official","path":"models/loss.py","file_url":"https://github.com/OPPOMKLab/u-LLaVA/blob/HEAD/models/loss.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce749f837424dd8c"}},{"code_sha256_prefix":"faf023207d110281","entry":"dice_loss","repo":"OPPOMKLab/u-LLaVA","repo_kind":"official","path":"models/loss.py","file_url":"https://github.com/OPPOMKLab/u-LLaVA/blob/HEAD/models/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"faf023207d110281"}},{"code_sha256_prefix":"3335a08070d5bb43","entry":"find_linear_layers","repo":"OPPOMKLab/u-LLaVA","repo_kind":"official","path":"train_ullava.py","file_url":"https://github.com/OPPOMKLab/u-LLaVA/blob/HEAD/train_ullava.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3335a08070d5bb43"}},{"code_sha256_prefix":"dc8fb9b6fb48ac37","entry":"generalized_box_iou","repo":"OPPOMKLab/u-LLaVA","repo_kind":"official","path":"models/loss.py","file_url":"https://github.com/OPPOMKLab/u-LLaVA/blob/HEAD/models/loss.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dc8fb9b6fb48ac37"}},{"code_sha256_prefix":"e7bb24cbb35466a5","entry":"to_cuda","repo":"OPPOMKLab/u-LLaVA","repo_kind":"official","path":"evaluation/tools.py","file_url":"https://github.com/OPPOMKLab/u-LLaVA/blob/HEAD/evaluation/tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7bb24cbb35466a5"}},{"code_sha256_prefix":"136834d66336e6b9","entry":"intersectionAndUnionGPU","repo":"OPPOMKLab/u-LLaVA","repo_kind":"official","path":"evaluation/tools.py","file_url":"https://github.com/OPPOMKLab/u-LLaVA/blob/HEAD/evaluation/tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"136834d66336e6b9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}