{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unifork-exploring-modality-alignment-for","title":"UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation","arxiv_id":"2506.17202","date":"2025-06-20","proceeding":null,"authors":["Teng Li","Quanfeng Lu","Lirui Zhao","Hao Li","Xizhou Zhu","Yu Qiao","Jun Zhang","Wenqi Shao"],"abstract":"Unified image understanding and generation has emerged as a promising paradigm in multimodal artificial intelligence. Despite recent progress, the optimal architectural design for such unified models remains an open challenge. In this work, we start by analyzing the modality alignment behaviors of task-specific expert models for understanding and generation, as well as current unified models. Our analysis reveals a crucial observation: understanding tasks benefit from a progressively increasing modality alignment across network depth, which helps build up semantic information for better comprehension; In contrast, generation tasks follow a different trend: modality alignment increases in the early layers but decreases in the deep layers to recover spatial details. These divergent alignment patterns create a fundamental conflict in fully shared Transformer backbones, where a uniform representational flow often leads to performance compromises across two tasks. Motivated by this finding, we introduce UniFork, a novel Y-shaped architecture that shares the shallow layers for cross-task representation learning, while employing task-specific branches in deeper layers to avoid task interference. This design effectively balances shared learning and task specialization. Through extensive ablation experiments, we demonstrate that Unifork consistently outperforms conventional fully shared Transformer architectures, and achieves performance on par with or better than task-specific models.","url_abs":"https://arxiv.org/abs/2506.17202v1","url_pdf":"https://arxiv.org/pdf/2506.17202v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unifork-exploring-modality-alignment-for","repo_url":"https://github.com/tliby/unifork","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.17202","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.17202"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tliby/unifork","reach":{"status":"ok"}}],"summary":{"ran":2,"ran_draft_wrong":2,"ran_fixture":2,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"7e03b180fa317c9a","entry":"divide_to_patches","repo":"tliby/unifork","repo_kind":"official","path":"unifork/mm_utils.py","file_url":"https://github.com/tliby/unifork/blob/HEAD/unifork/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e03b180fa317c9a"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"tliby/unifork","repo_kind":"official","path":"unifork/eval/model_vqa.py","file_url":"https://github.com/tliby/unifork/blob/HEAD/unifork/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"468eedeba67f1b00","entry":"resize_and_pad_image","repo":"tliby/unifork","repo_kind":"official","path":"unifork/mm_utils.py","file_url":"https://github.com/tliby/unifork/blob/HEAD/unifork/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"468eedeba67f1b00"}},{"code_sha256_prefix":"3999ff487573f32c","entry":"select_best_resolution","repo":"tliby/unifork","repo_kind":"official","path":"unifork/mm_utils.py","file_url":"https://github.com/tliby/unifork/blob/HEAD/unifork/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3999ff487573f32c"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"tliby/unifork","repo_kind":"official","path":"unifork/eval/model_vqa.py","file_url":"https://github.com/tliby/unifork/blob/HEAD/unifork/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"55c32993da87759b","entry":"unpad_image","repo":"tliby/unifork","repo_kind":"official","path":"unifork/model/llava_arch.py","file_url":"https://github.com/tliby/unifork/blob/HEAD/unifork/model/llava_arch.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55c32993da87759b"}},{"code_sha256_prefix":"d9010a83f7ea789c","entry":"build_vision_projector","repo":"tliby/unifork","repo_kind":"official","path":"unifork/model/projector.py","file_url":"https://github.com/tliby/unifork/blob/HEAD/unifork/model/projector.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9010a83f7ea789c"}},{"code_sha256_prefix":"9738a62bfa4c292d","entry":"collate_fn","repo":"tliby/unifork","repo_kind":"official","path":"unifork/eval/model_vqa_loader.py","file_url":"https://github.com/tliby/unifork/blob/HEAD/unifork/eval/model_vqa_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9738a62bfa4c292d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}