{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multimodal-task-vectors-enable-many-shot","title":"Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning","arxiv_id":"2406.15334","date":"2024-06-21","proceeding":null,"authors":["Brandon Huang","Chancharik Mitra","Assaf Arbelle","Leonid Karlinsky","Trevor Darrell","Roei Herzig"],"abstract":"The recent success of interleaved Large Multimodal Models (LMMs) in few-shot learning suggests that in-context learning (ICL) with many examples can be promising for learning new tasks. However, this many-shot multimodal ICL setting has one crucial problem: it is fundamentally limited by the model's context length set at pretraining. The problem is especially prominent in the multimodal domain, which processes both text and images, requiring additional tokens. This motivates the need for a multimodal method to compress many shots into fewer tokens without finetuning. In this work, we enable LMMs to perform multimodal, many-shot in-context learning by leveraging Multimodal Task Vectors (MTV) -- compact implicit representations of in-context examples compressed in the model's attention heads. Specifically, we first demonstrate the existence of such MTV in LMMs and then leverage these extracted MTV to enable many-shot in-context learning for various vision-and-language tasks. Our experiments suggest that MTV can scale in performance with the number of compressed shots and generalize to similar out-of-domain tasks without additional context length for inference. Code: https://github.com/Brandon3964/MultiModal-Task-Vector","url_abs":"https://arxiv.org/abs/2406.15334v3","url_pdf":"https://arxiv.org/pdf/2406.15334v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multimodal-task-vectors-enable-many-shot","repo_url":"https://github.com/brandon3964/multimodal-task-vector","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.15334","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.15334"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Brandon3964/MultiModal-Task-Vector","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/QwenLM/Qwen-VL","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/brandon3964/multimodal-task-vector","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_honours":4,"ran":7,"ran_violates":2,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":19,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":19,"samples":[{"code_sha256_prefix":"ce749f837424dd8c","entry":"box_iou","repo":"Brandon3964/MultiModal-Task-Vector","repo_kind":"official","path":"eval_mm/evaluate_grounding.py","file_url":"https://github.com/Brandon3964/MultiModal-Task-Vector/blob/HEAD/eval_mm/evaluate_grounding.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce749f837424dd8c"}},{"code_sha256_prefix":"a7ddf937ddb1ba60","entry":"collate_fn","repo":"Brandon3964/MultiModal-Task-Vector","repo_kind":"official","path":"eval_mm/evaluate_caption.py","file_url":"https://github.com/Brandon3964/MultiModal-Task-Vector/blob/HEAD/eval_mm/evaluate_caption.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a7ddf937ddb1ba60"}},{"code_sha256_prefix":"d19fe23145498c35","entry":"collate_fn","repo":"Brandon3964/MultiModal-Task-Vector","repo_kind":"official","path":"eval_mm/evaluate_grounding.py","file_url":"https://github.com/Brandon3964/MultiModal-Task-Vector/blob/HEAD/eval_mm/evaluate_grounding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d19fe23145498c35"}},{"code_sha256_prefix":"37a27553c04d476e","entry":"collate_fn","repo":"Brandon3964/MultiModal-Task-Vector","repo_kind":"official","path":"eval_mm/evaluate_multiple_choice.py","file_url":"https://github.com/Brandon3964/MultiModal-Task-Vector/blob/HEAD/eval_mm/evaluate_multiple_choice.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"37a27553c04d476e"}},{"code_sha256_prefix":"e159a001bd4cbfaa","entry":"evaluate_exact_match_accuracy","repo":"brandon3964/multimodal-task-vector","repo_kind":"official","path":"eval_mm/evaluate_vqa.py","file_url":"https://github.com/brandon3964/multimodal-task-vector/blob/HEAD/eval_mm/evaluate_vqa.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e159a001bd4cbfaa"}},{"code_sha256_prefix":"4ffcad52354b4806","entry":"evaluate_relaxed_accuracy","repo":"brandon3964/multimodal-task-vector","repo_kind":"official","path":"eval_mm/evaluate_vqa.py","file_url":"https://github.com/brandon3964/multimodal-task-vector/blob/HEAD/eval_mm/evaluate_vqa.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4ffcad52354b4806"}},{"code_sha256_prefix":"d3495f004eb282e1","entry":"format_vizwiz","repo":"Brandon3964/MultiModal-Task-Vector","repo_kind":"official","path":"MTV/preprocess.py","file_url":"https://github.com/Brandon3964/MultiModal-Task-Vector/blob/HEAD/MTV/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d3495f004eb282e1"}},{"code_sha256_prefix":"90a8e73fa6de72a4","entry":"levenshtein_distance","repo":"Brandon3964/MultiModal-Task-Vector","repo_kind":"official","path":"eval_mm/infographicsvqa_eval.py","file_url":"https://github.com/Brandon3964/MultiModal-Task-Vector/blob/HEAD/eval_mm/infographicsvqa_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"90a8e73fa6de72a4"}},{"code_sha256_prefix":"3275ed06c9e635d8","entry":"load_image","repo":"brandon3964/multimodal-task-vector","repo_kind":"official","path":"MTV/models.py","file_url":"https://github.com/brandon3964/multimodal-task-vector/blob/HEAD/MTV/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3275ed06c9e635d8"}},{"code_sha256_prefix":"8f69c6173ca1fadb","entry":"load_images","repo":"brandon3964/multimodal-task-vector","repo_kind":"official","path":"MTV/models.py","file_url":"https://github.com/brandon3964/multimodal-task-vector/blob/HEAD/MTV/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8f69c6173ca1fadb"}},{"code_sha256_prefix":"c62b6812691c4ecc","entry":"open_data","repo":"Brandon3964/MultiModal-Task-Vector","repo_kind":"official","path":"MTV/preprocess.py","file_url":"https://github.com/Brandon3964/MultiModal-Task-Vector/blob/HEAD/MTV/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c62b6812691c4ecc"}},{"code_sha256_prefix":"8b733c9fc590e4c1","entry":"preprocess","repo":"Brandon3964/MultiModal-Task-Vector","repo_kind":"official","path":"finetune.py","file_url":"https://github.com/Brandon3964/MultiModal-Task-Vector/blob/HEAD/finetune.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8b733c9fc590e4c1"}},{"code_sha256_prefix":"b3c8680ebd8739af","entry":"relaxed_correctness","repo":"brandon3964/multimodal-task-vector","repo_kind":"official","path":"eval_mm/evaluate_vqa.py","file_url":"https://github.com/brandon3964/multimodal-task-vector/blob/HEAD/eval_mm/evaluate_vqa.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b3c8680ebd8739af"}},{"code_sha256_prefix":"d4db6c913fb49a77","entry":"split_activations_by_head","repo":"Brandon3964/MultiModal-Task-Vector","repo_kind":"official","path":"MTV/mtv_utils.py","file_url":"https://github.com/Brandon3964/MultiModal-Task-Vector/blob/HEAD/MTV/mtv_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d4db6c913fb49a77"}},{"code_sha256_prefix":"385a35a03fe04cde","entry":"add_extra_stop_words","repo":"Brandon3964/MultiModal-Task-Vector","repo_kind":"official","path":"openai_api.py","file_url":"https://github.com/Brandon3964/MultiModal-Task-Vector/blob/HEAD/openai_api.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"385a35a03fe04cde"}},{"code_sha256_prefix":"b90651bcc771b76f","entry":"evaluate_method","repo":"Brandon3964/MultiModal-Task-Vector","repo_kind":"official","path":"eval_mm/infographicsvqa_eval.py","file_url":"https://github.com/Brandon3964/MultiModal-Task-Vector/blob/HEAD/eval_mm/infographicsvqa_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b90651bcc771b76f"}},{"code_sha256_prefix":"9d2410fb29f7ac7e","entry":"get_format_func","repo":"Brandon3964/MultiModal-Task-Vector","repo_kind":"official","path":"MTV/preprocess.py","file_url":"https://github.com/Brandon3964/MultiModal-Task-Vector/blob/HEAD/MTV/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9d2410fb29f7ac7e"}},{"code_sha256_prefix":"2534a18f2f56fd6c","entry":"load_model","repo":"Brandon3964/MultiModal-Task-Vector","repo_kind":"official","path":"MTV/mtv_utils.py","file_url":"https://github.com/Brandon3964/MultiModal-Task-Vector/blob/HEAD/MTV/mtv_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2534a18f2f56fd6c"}},{"code_sha256_prefix":"469cf12b400c2c5a","entry":"trim_stop_words","repo":"Brandon3964/MultiModal-Task-Vector","repo_kind":"official","path":"openai_api.py","file_url":"https://github.com/Brandon3964/MultiModal-Task-Vector/blob/HEAD/openai_api.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"469cf12b400c2c5a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}