{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diagnosing-and-re-learning-for-balanced","title":"Diagnosing and Re-learning for Balanced Multimodal Learning","arxiv_id":"2407.09705","date":"2024-07-12","proceeding":null,"authors":["Yake Wei","Siwei Li","Ruoxuan Feng","Di Hu"],"abstract":"To overcome the imbalanced multimodal learning problem, where models prefer the training of specific modalities, existing methods propose to control the training of uni-modal encoders from different perspectives, taking the inter-modal performance discrepancy as the basis. However, the intrinsic limitation of modality capacity is ignored. The scarcely informative modalities can be recognized as ``worse-learnt'' ones, which could force the model to memorize more noise, counterproductively affecting the multimodal model ability. Moreover, the current modality modulation methods narrowly concentrate on selected worse-learnt modalities, even suppressing the training of others. Hence, it is essential to consider the intrinsic limitation of modality capacity and take all modalities into account during balancing. To this end, we propose the Diagnosing \\& Re-learning method. The learning state of each modality is firstly estimated based on the separability of its uni-modal representation space, and then used to softly re-initialize the corresponding uni-modal encoder. In this way, the over-emphasizing of scarcely informative modalities is avoided. In addition, encoders of worse-learnt modalities are enhanced, simultaneously avoiding the over-training of other modalities. Accordingly, multimodal learning is effectively balanced and enhanced. Experiments covering multiple types of modalities and multimodal frameworks demonstrate the superior performance of our simple-yet-effective method for balanced multimodal learning. The source code and dataset are available at \\url{https://github.com/GeWu-Lab/Diagnosing_Relearning_ECCV2024}.","url_abs":"https://arxiv.org/abs/2407.09705v1","url_pdf":"https://arxiv.org/pdf/2407.09705v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diagnosing-and-re-learning-for-balanced","repo_url":"https://github.com/gewu-lab/diagnosing_relearning_eccv2024","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.09705","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.09705"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gewu-lab/diagnosing_relearning_eccv2024","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"d9def42110729a85","entry":"conv1x1","repo":"gewu-lab/diagnosing_relearning_eccv2024","repo_kind":"official","path":"code/models/backbone.py","file_url":"https://github.com/gewu-lab/diagnosing_relearning_eccv2024/blob/HEAD/code/models/backbone.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9def42110729a85"}},{"code_sha256_prefix":"160bb14bd76201b4","entry":"conv3x3","repo":"gewu-lab/diagnosing_relearning_eccv2024","repo_kind":"official","path":"code/models/backbone.py","file_url":"https://github.com/gewu-lab/diagnosing_relearning_eccv2024/blob/HEAD/code/models/backbone.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"160bb14bd76201b4"}},{"code_sha256_prefix":"d102ce926eeacdf4","entry":"purity_score","repo":"gewu-lab/diagnosing_relearning_eccv2024","repo_kind":"official","path":"code/ours.py","file_url":"https://github.com/gewu-lab/diagnosing_relearning_eccv2024/blob/HEAD/code/ours.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d102ce926eeacdf4"}},{"code_sha256_prefix":"b38d3192d06c58da","entry":"reinit","repo":"gewu-lab/diagnosing_relearning_eccv2024","repo_kind":"official","path":"code/ours.py","file_url":"https://github.com/gewu-lab/diagnosing_relearning_eccv2024/blob/HEAD/code/ours.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b38d3192d06c58da"}},{"code_sha256_prefix":"67a6282ef1490440","entry":"reinit_score","repo":"gewu-lab/diagnosing_relearning_eccv2024","repo_kind":"official","path":"code/ours.py","file_url":"https://github.com/gewu-lab/diagnosing_relearning_eccv2024/blob/HEAD/code/ours.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"67a6282ef1490440"}},{"code_sha256_prefix":"33731e9aed0fd4cd","entry":"train_epoch","repo":"gewu-lab/diagnosing_relearning_eccv2024","repo_kind":"official","path":"code/baseline.py","file_url":"https://github.com/gewu-lab/diagnosing_relearning_eccv2024/blob/HEAD/code/baseline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"33731e9aed0fd4cd"}},{"code_sha256_prefix":"7ac23ebfb3035b34","entry":"valid","repo":"gewu-lab/diagnosing_relearning_eccv2024","repo_kind":"official","path":"code/baseline.py","file_url":"https://github.com/gewu-lab/diagnosing_relearning_eccv2024/blob/HEAD/code/baseline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7ac23ebfb3035b34"}},{"code_sha256_prefix":"f186c6d95e90615a","entry":"resnet18","repo":"gewu-lab/diagnosing_relearning_eccv2024","repo_kind":"official","path":"code/models/backbone.py","file_url":"https://github.com/gewu-lab/diagnosing_relearning_eccv2024/blob/HEAD/code/models/backbone.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f186c6d95e90615a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}