{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mmrl-multi-modal-representation-learning-for","title":"MMRL: Multi-Modal Representation Learning for Vision-Language Models","arxiv_id":"2503.08497","date":"2025-03-11","proceeding":"CVPR 2025 1","authors":["Yuncheng Guo","Xiaodong Gu"],"abstract":"Large-scale pre-trained Vision-Language Models (VLMs) have become essential for transfer learning across diverse tasks. However, adapting these models with limited few-shot data often leads to overfitting, diminishing their performance on new tasks. To tackle this issue, we propose a novel Multi-Modal Representation Learning (MMRL) framework that introduces a shared, learnable, and modality-agnostic representation space. MMRL projects the space tokens to text and image representation tokens, facilitating more effective multi-modal interactions. Unlike previous approaches that solely optimize class token features, MMRL integrates representation tokens at higher layers of the encoders--where dataset-specific features are more prominent--while preserving generalized knowledge in the lower layers. During training, both representation and class features are optimized, with trainable projection layer applied to the representation tokens, whereas the class token projection layer remains frozen to retain pre-trained knowledge. Furthermore, a regularization term is introduced to align the class features and text features with the zero-shot features from the frozen VLM, thereby safeguarding the model's generalization capacity. For inference, a decoupling strategy is employed, wherein both representation and class features are utilized for base classes, while only the class features, which retain more generalized knowledge, are used for new tasks. Extensive experiments across 15 datasets demonstrate that MMRL outperforms state-of-the-art methods, achieving a balanced trade-off between task-specific adaptation and generalization. Code is available at https://github.com/yunncheng/MMRL.","url_abs":"https://arxiv.org/abs/2503.08497v2","url_pdf":"https://arxiv.org/pdf/2503.08497v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mmrl-multi-modal-representation-learning-for","repo_url":"https://github.com/yunncheng/MMRL","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"coop","method_name":"CoOp"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/prompt-engineering-on-caltech-101","task":"Prompt Engineering","dataset":"Caltech-101","model":"MMRL","rank_in_archive_order":4,"of":14,"metrics":{"Harmonic mean":"96.68"},"uses_additional_data":false},{"leaderboard":"/sota/prompt-engineering-on-dtd","task":"Prompt Engineering","dataset":"DTD","model":"MMRL","rank_in_archive_order":4,"of":14,"metrics":{"Harmonic mean":"73.82"},"uses_additional_data":false},{"leaderboard":"/sota/prompt-engineering-on-eurosat","task":"Prompt Engineering","dataset":"EuroSAT","model":"MMRL","rank_in_archive_order":4,"of":14,"metrics":{"Harmonic mean":"87.21"},"uses_additional_data":false},{"leaderboard":"/sota/prompt-engineering-on-fgvc-aircraft","task":"Prompt Engineering","dataset":"FGVC-Aircraft","model":"MMRL","rank_in_archive_order":4,"of":14,"metrics":{"Harmonic mean":"41.15"},"uses_additional_data":false},{"leaderboard":"/sota/prompt-engineering-on-food-101","task":"Prompt Engineering","dataset":"Food-101","model":"MMRL","rank_in_archive_order":10,"of":13,"metrics":{"Harmonic mean":"91.03"},"uses_additional_data":false},{"leaderboard":"/sota/prompt-engineering-on-imagenet","task":"Prompt Engineering","dataset":"ImageNet","model":"MMRL","rank_in_archive_order":3,"of":15,"metrics":{"Harmonic mean":"74.45"},"uses_additional_data":false},{"leaderboard":"/sota/prompt-engineering-on-imagenet-v2","task":"Prompt Engineering","dataset":"ImageNet V2","model":"MMRL","rank_in_archive_order":3,"of":8,"metrics":{"Top-1 accuracy %":"64.47"},"uses_additional_data":false},{"leaderboard":"/sota/prompt-engineering-on-imagenet-a","task":"Prompt Engineering","dataset":"ImageNet-A","model":"MMRL","rank_in_archive_order":2,"of":9,"metrics":{"Top-1 accuracy %":"51.20"},"uses_additional_data":false},{"leaderboard":"/sota/prompt-engineering-on-imagenet-r","task":"Prompt Engineering","dataset":"ImageNet-R","model":"MMRL","rank_in_archive_order":3,"of":9,"metrics":{"Top-1 accuracy %":"77.53"},"uses_additional_data":false},{"leaderboard":"/sota/prompt-engineering-on-imagenet-s","task":"Prompt Engineering","dataset":"ImageNet-S","model":"MMRL","rank_in_archive_order":6,"of":9,"metrics":{"Top-1 accuracy %":"49.17"},"uses_additional_data":false},{"leaderboard":"/sota/prompt-engineering-on-oxford-102-flower","task":"Prompt Engineering","dataset":"Oxford 102 Flower","model":"MMRL","rank_in_archive_order":4,"of":14,"metrics":{"Harmonic mean":"86.78"},"uses_additional_data":false},{"leaderboard":"/sota/prompt-engineering-on-oxford-iiit-pet-dataset","task":"Prompt Engineering","dataset":"Oxford-IIIT Pet Dataset","model":"MMRL","rank_in_archive_order":4,"of":14,"metrics":{"Harmonic mean":"96.74"},"uses_additional_data":false},{"leaderboard":"/sota/prompt-engineering-on-sun397","task":"Prompt Engineering","dataset":"SUN397","model":"MMRL","rank_in_archive_order":4,"of":14,"metrics":{"Harmonic mean":"81.2"},"uses_additional_data":false},{"leaderboard":"/sota/prompt-engineering-on-stanford-cars-1","task":"Prompt Engineering","dataset":"Stanford Cars","model":"MMRL","rank_in_archive_order":3,"of":14,"metrics":{"Harmonic mean":"78.06"},"uses_additional_data":false},{"leaderboard":"/sota/prompt-engineering-on-ucf101","task":"Prompt Engineering","dataset":"UCF101","model":"MMRL","rank_in_archive_order":2,"of":14,"metrics":{"Harmonic mean":"83.89"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.08497","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.08497"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/yunncheng/MMRL","reach":null}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8121fdbb883925aa","entry":"MultiModalRepresentationLearner","repo":"yunncheng/MMRL","repo_kind":"official","path":"trainers/mmrl.py","file_url":"https://github.com/yunncheng/MMRL/blob/HEAD/trainers/mmrl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8121fdbb883925aa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}