{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/classifier-guided-gradient-modulation-for","title":"Classifier-guided Gradient Modulation for Enhanced Multimodal Learning","arxiv_id":"2411.01409","date":"2024-11-03","proceeding":null,"authors":["Zirun Guo","Tao Jin","Jingyuan Chen","Zhou Zhao"],"abstract":"Multimodal learning has developed very fast in recent years. However, during the multimodal training process, the model tends to rely on only one modality based on which it could learn faster, thus leading to inadequate use of other modalities. Existing methods to balance the training process always have some limitations on the loss functions, optimizers and the number of modalities and only consider modulating the magnitude of the gradients while ignoring the directions of the gradients. To solve these problems, in this paper, we present a novel method to balance multimodal learning with Classifier-Guided Gradient Modulation (CGGM), considering both the magnitude and directions of the gradients. We conduct extensive experiments on four multimodal datasets: UPMC-Food 101, CMU-MOSI, IEMOCAP and BraTS 2021, covering classification, regression and segmentation tasks. The results show that CGGM outperforms all the baselines and other state-of-the-art methods consistently, demonstrating its effectiveness and versatility. Our code is available at https://github.com/zrguo/CGGM.","url_abs":"https://arxiv.org/abs/2411.01409v1","url_pdf":"https://arxiv.org/pdf/2411.01409v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"classifier-guided-gradient-modulation-for","repo_url":"https://github.com/zrguo/cggm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.01409","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.01409"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/zrguo/CGGM","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zrguo/cggm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran_violates":1,"ran":5,"ran_honours":2,"ran_fixture":1,"unverified":15},"by_repo_kind":{"official":{"samples":25,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"10865bbb99140edd","entry":"Linear","repo":"zrguo/CGGM","repo_kind":"official","path":"modules/transformer.py","file_url":"https://github.com/zrguo/CGGM/blob/HEAD/modules/transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10865bbb99140edd"}},{"code_sha256_prefix":"184a5eddedaf6a42","entry":"buffered_future_mask","repo":"zrguo/CGGM","repo_kind":"official","path":"modules/transformer.py","file_url":"https://github.com/zrguo/CGGM/blob/HEAD/modules/transformer.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"184a5eddedaf6a42"}},{"code_sha256_prefix":"ed6f4e9e545f8f07","entry":"cal_cos","repo":"zrguo/cggm","repo_kind":"official","path":"src/foodtrain.py","file_url":"https://github.com/zrguo/cggm/blob/HEAD/src/foodtrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed6f4e9e545f8f07"}},{"code_sha256_prefix":"d67ae69618e8f777","entry":"eval_food","repo":"zrguo/cggm","repo_kind":"official","path":"src/foodtrain.py","file_url":"https://github.com/zrguo/cggm/blob/HEAD/src/foodtrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d67ae69618e8f777"}},{"code_sha256_prefix":"8f266b9d616de9b9","entry":"fill_with_neg_inf","repo":"zrguo/CGGM","repo_kind":"official","path":"modules/transformer.py","file_url":"https://github.com/zrguo/CGGM/blob/HEAD/modules/transformer.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f266b9d616de9b9"}},{"code_sha256_prefix":"21958d540bd87a8c","entry":"make_positions","repo":"zrguo/CGGM","repo_kind":"official","path":"modules/position_embedding.py","file_url":"https://github.com/zrguo/CGGM/blob/HEAD/modules/position_embedding.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21958d540bd87a8c"}},{"code_sha256_prefix":"b39cbc0e9aa13c48","entry":"train_eval_food","repo":"zrguo/cggm","repo_kind":"official","path":"src/foodtrain.py","file_url":"https://github.com/zrguo/cggm/blob/HEAD/src/foodtrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b39cbc0e9aa13c48"}},{"code_sha256_prefix":"ad9faeb1beaacf40","entry":"train_eval_iemo","repo":"zrguo/cggm","repo_kind":"official","path":"src/foodtrain.py","file_url":"https://github.com/zrguo/cggm/blob/HEAD/src/foodtrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad9faeb1beaacf40"}},{"code_sha256_prefix":"7b0ce64e017cea1a","entry":"train_eval_senti","repo":"zrguo/cggm","repo_kind":"official","path":"src/foodtrain.py","file_url":"https://github.com/zrguo/cggm/blob/HEAD/src/foodtrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b0ce64e017cea1a"}},{"code_sha256_prefix":"6d4da8365bb0f5d1","entry":"weighted_accuracy","repo":"zrguo/CGGM","repo_kind":"official","path":"src/eval_metrics.py","file_url":"https://github.com/zrguo/CGGM/blob/HEAD/src/eval_metrics.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d4da8365bb0f5d1"}},{"code_sha256_prefix":"f70f65ad58a53350","entry":"Classifier","repo":"zrguo/cggm","repo_kind":"official","path":"src/foodtrain.py","file_url":"https://github.com/zrguo/cggm/blob/HEAD/src/foodtrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f70f65ad58a53350"}},{"code_sha256_prefix":"f927307b92481f41","entry":"ClassifierGuided","repo":"zrguo/cggm","repo_kind":"official","path":"src/foodtrain.py","file_url":"https://github.com/zrguo/cggm/blob/HEAD/src/foodtrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f927307b92481f41"}},{"code_sha256_prefix":"1cd046b456ad9f31","entry":"FoodModel","repo":"zrguo/cggm","repo_kind":"official","path":"src/foodtrain.py","file_url":"https://github.com/zrguo/cggm/blob/HEAD/src/foodtrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1cd046b456ad9f31"}},{"code_sha256_prefix":"6aa213f79f005ec4","entry":"ResNet101","repo":"zrguo/CGGM","repo_kind":"official","path":"models/segmodel.py","file_url":"https://github.com/zrguo/CGGM/blob/HEAD/models/segmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6aa213f79f005ec4"}},{"code_sha256_prefix":"34ea1f3d71361140","entry":"TransformerEncoder","repo":"zrguo/cggm","repo_kind":"official","path":"src/foodtrain.py","file_url":"https://github.com/zrguo/cggm/blob/HEAD/src/foodtrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"34ea1f3d71361140"}},{"code_sha256_prefix":"3dfcc20147c77637","entry":"TransformerEncoderLayer","repo":"zrguo/cggm","repo_kind":"official","path":"src/foodtrain.py","file_url":"https://github.com/zrguo/cggm/blob/HEAD/src/foodtrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3dfcc20147c77637"}},{"code_sha256_prefix":"1fe4d67c3bc4510b","entry":"augment_gaussian_noise","repo":"zrguo/CGGM","repo_kind":"official","path":"datasets/BratsDataset.py","file_url":"https://github.com/zrguo/CGGM/blob/HEAD/datasets/BratsDataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1fe4d67c3bc4510b"}},{"code_sha256_prefix":"137ee892a73b5311","entry":"build_aspp","repo":"zrguo/CGGM","repo_kind":"official","path":"models/segmodel.py","file_url":"https://github.com/zrguo/CGGM/blob/HEAD/models/segmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"137ee892a73b5311"}},{"code_sha256_prefix":"22ecd809e4d1de39","entry":"build_decoder","repo":"zrguo/CGGM","repo_kind":"official","path":"models/segmodel.py","file_url":"https://github.com/zrguo/CGGM/blob/HEAD/models/segmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"22ecd809e4d1de39"}},{"code_sha256_prefix":"653ae4081b06cab6","entry":"cutout","repo":"zrguo/CGGM","repo_kind":"official","path":"datasets/BratsDataset.py","file_url":"https://github.com/zrguo/CGGM/blob/HEAD/datasets/BratsDataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"653ae4081b06cab6"}},{"code_sha256_prefix":"cae47ad9ef47ac82","entry":"eval_senti","repo":"zrguo/CGGM","repo_kind":"official","path":"src/eval_metrics.py","file_url":"https://github.com/zrguo/CGGM/blob/HEAD/src/eval_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cae47ad9ef47ac82"}},{"code_sha256_prefix":"e2fe3744d6f3b381","entry":"initiate","repo":"zrguo/cggm","repo_kind":"official","path":"src/foodtrain.py","file_url":"https://github.com/zrguo/cggm/blob/HEAD/src/foodtrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2fe3744d6f3b381"}},{"code_sha256_prefix":"65494cdd1508772e","entry":"multiclass_acc","repo":"zrguo/CGGM","repo_kind":"official","path":"src/eval_metrics.py","file_url":"https://github.com/zrguo/CGGM/blob/HEAD/src/eval_metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"65494cdd1508772e"}},{"code_sha256_prefix":"341db6e7ca441245","entry":"train_model","repo":"zrguo/cggm","repo_kind":"official","path":"src/foodtrain.py","file_url":"https://github.com/zrguo/cggm/blob/HEAD/src/foodtrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"341db6e7ca441245"}},{"code_sha256_prefix":"27a4ff497fab5848","entry":"train_model","repo":"zrguo/CGGM","repo_kind":"official","path":"src/segtrain.py","file_url":"https://github.com/zrguo/CGGM/blob/HEAD/src/segtrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27a4ff497fab5848"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}