{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2601-21426","title":"MultiModal Fine-tuning with Synthetic Captions","arxiv_id":"2601.21426","date":"2026-01-29","proceeding":null,"authors":["Shohei Enomoto","Shin'ya Yamaguchi"],"abstract":"In this paper, we address a fundamental gap between pre-training and fine-tuning of deep neural networks: while pre-training has shifted from unimodal to multimodal learning with enhanced visual understanding, fine-tuning predominantly remains unimodal, limiting the benefits of rich pre-trained representations. To bridge this gap, we propose a novel approach that transforms unimodal datasets into multimodal ones using Multimodal Large Language Models (MLLMs) to generate synthetic image captions for fine-tuning models with a multimodal objective. Our method employs carefully designed prompts incorporating class labels and domain context to produce high-quality captions tailored for classification tasks. Furthermore, we introduce a supervised contrastive loss function that explicitly encourages clustering of same-class representations during fine-tuning, along with a new inference technique that leverages class-averaged text embeddings from multiple synthetic captions per image. Extensive experiments across 13 image classification benchmarks demonstrate that our approach outperforms baseline methods, with particularly significant improvements in few-shot learning scenarios. Our work establishes a new paradigm for dataset enhancement that effectively bridges the gap between multimodal pre-training and fine-tuning. Our code is available at https://github.com/s-enmt/MMFT.","url_abs":"https://arxiv.org/abs/2601.21426","url_pdf":"https://arxiv.org/pdf/2601.21426","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2601.21426"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/s-enmt/MMFT","reach":null}],"summary":{"unverified":15},"by_repo_kind":{"found_in_text":{"samples":15,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"0104870a9a95d4ea","entry":"contrastive_loss","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"utils/model_utils.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0104870a9a95d4ea"}},{"code_sha256_prefix":"9d13831bf4ae7741","entry":"create_experiment_hash","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"utils/experiment_utils.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/utils/experiment_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9d13831bf4ae7741"}},{"code_sha256_prefix":"75660717b6b67b02","entry":"extract_retry_delay","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"generate_captions.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/generate_captions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"75660717b6b67b02"}},{"code_sha256_prefix":"ec0478398d7cae5e","entry":"find_completed_experiments","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"collect_results.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/collect_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ec0478398d7cae5e"}},{"code_sha256_prefix":"f79228f9a6e405b9","entry":"get_dataset","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"utils/dataset_utils.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/utils/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f79228f9a6e405b9"}},{"code_sha256_prefix":"5bb4cdc1a9d40080","entry":"get_filename_from_dataset","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"utils/dataset_utils.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/utils/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5bb4cdc1a9d40080"}},{"code_sha256_prefix":"85db79d26d53acd4","entry":"load_existing_captions","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"generate_captions.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/generate_captions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"85db79d26d53acd4"}},{"code_sha256_prefix":"b20dc29f5530fbf8","entry":"load_experiment_config","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"utils/experiment_utils.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/utils/experiment_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b20dc29f5530fbf8"}},{"code_sha256_prefix":"7e4bbc333693ea16","entry":"load_json_file","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"merge_captions.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/merge_captions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7e4bbc333693ea16"}},{"code_sha256_prefix":"2bc7277fbb33cc3f","entry":"loads_data","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"utils/ILM_Dataloader.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/utils/ILM_Dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2bc7277fbb33cc3f"}},{"code_sha256_prefix":"0e309647c3314989","entry":"merge_caption_files","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"merge_captions.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/merge_captions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0e309647c3314989"}},{"code_sha256_prefix":"74368afa39aa1fce","entry":"read_experiment_data","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"collect_results.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/collect_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"74368afa39aa1fce"}},{"code_sha256_prefix":"b32782e33156c848","entry":"refine_classname","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"utils/dataset_utils.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/utils/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b32782e33156c848"}},{"code_sha256_prefix":"89a3435699fa9eab","entry":"setup_experiment_dir","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"utils/experiment_utils.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/utils/experiment_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"89a3435699fa9eab"}},{"code_sha256_prefix":"35720a0e5aeee40e","entry":"split_shot","repo":"s-enmt/MMFT","repo_kind":"found_in_text","path":"collect_results.py","file_url":"https://github.com/s-enmt/MMFT/blob/HEAD/collect_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"35720a0e5aeee40e"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CV","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}