{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/balanced-multimodal-learning-via-on-the-fly","title":"Balanced Multimodal Learning via On-the-fly Gradient Modulation","arxiv_id":"2203.15332","date":"2022-03-29","proceeding":"CVPR 2022 1","authors":["Xiaokang Peng","Yake Wei","Andong Deng","Dong Wang","Di Hu"],"abstract":"Multimodal learning helps to comprehensively understand the world, by integrating different senses. Accordingly, multiple input modalities are expected to boost model performance, but we actually find that they are not fully exploited even when the multimodal model outperforms its uni-modal counterpart. Specifically, in this paper we point out that existing multimodal discriminative models, in which uniform objective is designed for all modalities, could remain under-optimized uni-modal representations, caused by another dominated modality in some scenarios, e.g., sound in blowing wind event, vision in drawing picture event, etc. To alleviate this optimization imbalance, we propose on-the-fly gradient modulation to adaptively control the optimization of each modality, via monitoring the discrepancy of their contribution towards the learning objective. Further, an extra Gaussian noise that changes dynamically is introduced to avoid possible generalization drop caused by gradient modulation. As a result, we achieve considerable improvement over common fusion methods on different multimodal tasks, and this simple strategy can also boost existing multimodal methods, which illustrates its efficacy and versatility. The source code is available at \\url{https://github.com/GeWu-Lab/OGM-GE_CVPR2022}.","url_abs":"https://arxiv.org/abs/2203.15332v1","url_pdf":"https://arxiv.org/pdf/2203.15332v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"balanced-multimodal-learning-via-on-the-fly","repo_url":"https://github.com/gewu-lab/ogm-ge_cvpr2022","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"balanced-multimodal-learning-via-on-the-fly","repo_url":"https://github.com/gewu-lab/bml_tpami2024","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2203.15332","atlas_url":"https://app.syntology.ai/?focus=2203.15332","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.15332"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gewu-lab/ogm-ge_cvpr2022","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gewu-lab/bml_tpami2024","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/GeWu-Lab/OGM-GE_CVPR2022","reach":null}],"summary":{"ran":3,"unverified":5},"by_repo_kind":{"official":{"samples":8,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"48177bc8c5883df6","entry":"ConcatFusion","repo":"GeWu-Lab/OGM-GE_CVPR2022","repo_kind":"official","path":"models/basic_model.py","file_url":"https://github.com/GeWu-Lab/OGM-GE_CVPR2022/blob/HEAD/models/basic_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48177bc8c5883df6"}},{"code_sha256_prefix":"de31f8fee4527657","entry":"GatedFusion","repo":"GeWu-Lab/OGM-GE_CVPR2022","repo_kind":"official","path":"models/basic_model.py","file_url":"https://github.com/GeWu-Lab/OGM-GE_CVPR2022/blob/HEAD/models/basic_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de31f8fee4527657"}},{"code_sha256_prefix":"579830cf6d703855","entry":"SumFusion","repo":"GeWu-Lab/OGM-GE_CVPR2022","repo_kind":"official","path":"models/basic_model.py","file_url":"https://github.com/GeWu-Lab/OGM-GE_CVPR2022/blob/HEAD/models/basic_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"579830cf6d703855"}},{"code_sha256_prefix":"58acbad04b63ec6f","entry":"AVClassifier","repo":"GeWu-Lab/OGM-GE_CVPR2022","repo_kind":"official","path":"models/basic_model.py","file_url":"https://github.com/GeWu-Lab/OGM-GE_CVPR2022/blob/HEAD/models/basic_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58acbad04b63ec6f"}},{"code_sha256_prefix":"713cd9da0b658745","entry":"FiLM","repo":"GeWu-Lab/OGM-GE_CVPR2022","repo_kind":"official","path":"models/basic_model.py","file_url":"https://github.com/GeWu-Lab/OGM-GE_CVPR2022/blob/HEAD/models/basic_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"713cd9da0b658745"}},{"code_sha256_prefix":"477f37d037723674","entry":"ResNet","repo":"GeWu-Lab/OGM-GE_CVPR2022","repo_kind":"official","path":"models/basic_model.py","file_url":"https://github.com/GeWu-Lab/OGM-GE_CVPR2022/blob/HEAD/models/basic_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"477f37d037723674"}},{"code_sha256_prefix":"fc6115557015275e","entry":"_resnet","repo":"GeWu-Lab/OGM-GE_CVPR2022","repo_kind":"official","path":"models/basic_model.py","file_url":"https://github.com/GeWu-Lab/OGM-GE_CVPR2022/blob/HEAD/models/basic_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fc6115557015275e"}},{"code_sha256_prefix":"f3c834e82ebe728e","entry":"resnet18","repo":"GeWu-Lab/OGM-GE_CVPR2022","repo_kind":"official","path":"models/basic_model.py","file_url":"https://github.com/GeWu-Lab/OGM-GE_CVPR2022/blob/HEAD/models/basic_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3c834e82ebe728e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}