{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reconboost-boosting-can-achieve-modality","title":"ReconBoost: Boosting Can Achieve Modality Reconcilement","arxiv_id":"2405.09321","date":"2024-05-15","proceeding":null,"authors":["Cong Hua","Qianqian Xu","Shilong Bao","Zhiyong Yang","Qingming Huang"],"abstract":"This paper explores a novel multi-modal alternating learning paradigm pursuing a reconciliation between the exploitation of uni-modal features and the exploration of cross-modal interactions. This is motivated by the fact that current paradigms of multi-modal learning tend to explore multi-modal features simultaneously. The resulting gradient prohibits further exploitation of the features in the weak modality, leading to modality competition, where the dominant modality overpowers the learning process. To address this issue, we study the modality-alternating learning paradigm to achieve reconcilement. Specifically, we propose a new method called ReconBoost to update a fixed modality each time. Herein, the learning objective is dynamically adjusted with a reconcilement regularization against competition with the historical models. By choosing a KL-based reconcilement, we show that the proposed method resembles Friedman's Gradient-Boosting (GB) algorithm, where the updated learner can correct errors made by others and help enhance the overall performance. The major difference with the classic GB is that we only preserve the newest model for each modality to avoid overfitting caused by ensembling strong learners. Furthermore, we propose a memory consolidation scheme and a global rectification scheme to make this strategy more effective. Experiments over six multi-modal benchmarks speak to the efficacy of the method. We release the code at https://github.com/huacong/ReconBoost.","url_abs":"https://arxiv.org/abs/2405.09321v1","url_pdf":"https://arxiv.org/pdf/2405.09321v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reconboost-boosting-can-achieve-modality","repo_url":"https://github.com/huacong/reconboost","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.09321","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.09321"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huacong/reconboost","reach":{"status":"ok"}}],"summary":{"ran":5,"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"570d804f2bbd38c7","entry":"acc_score","repo":"huacong/reconboost","repo_kind":"official","path":"utils.py","file_url":"https://github.com/huacong/reconboost/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"570d804f2bbd38c7"}},{"code_sha256_prefix":"2ef957ef924eaf7e","entry":"check_status","repo":"huacong/reconboost","repo_kind":"official","path":"utils.py","file_url":"https://github.com/huacong/reconboost/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ef957ef924eaf7e"}},{"code_sha256_prefix":"d9def42110729a85","entry":"conv1x1","repo":"huacong/reconboost","repo_kind":"official","path":"models/CREMA/backbone.py","file_url":"https://github.com/huacong/reconboost/blob/HEAD/models/CREMA/backbone.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9def42110729a85"}},{"code_sha256_prefix":"160bb14bd76201b4","entry":"conv3x3","repo":"huacong/reconboost","repo_kind":"official","path":"models/CREMA/backbone.py","file_url":"https://github.com/huacong/reconboost/blob/HEAD/models/CREMA/backbone.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"160bb14bd76201b4"}},{"code_sha256_prefix":"0f6a89565e264586","entry":"init_pretrain","repo":"huacong/reconboost","repo_kind":"official","path":"train_MSA.py","file_url":"https://github.com/huacong/reconboost/blob/HEAD/train_MSA.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f6a89565e264586"}},{"code_sha256_prefix":"60ad73ba838017bd","entry":"res2tab","repo":"huacong/reconboost","repo_kind":"official","path":"utils.py","file_url":"https://github.com/huacong/reconboost/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"60ad73ba838017bd"}},{"code_sha256_prefix":"1fddfe7ac4ed4e2c","entry":"schedule_model","repo":"huacong/reconboost","repo_kind":"official","path":"schedule.py","file_url":"https://github.com/huacong/reconboost/blob/HEAD/schedule.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1fddfe7ac4ed4e2c"}},{"code_sha256_prefix":"c604bd495c385ed7","entry":"MMDataLoader","repo":"huacong/reconboost","repo_kind":"official","path":"loaders/MOSEIDataset.py","file_url":"https://github.com/huacong/reconboost/blob/HEAD/loaders/MOSEIDataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c604bd495c385ed7"}},{"code_sha256_prefix":"3fcaa8d813c082b1","entry":"resnet18","repo":"huacong/reconboost","repo_kind":"official","path":"models/CREMA/backbone.py","file_url":"https://github.com/huacong/reconboost/blob/HEAD/models/CREMA/backbone.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3fcaa8d813c082b1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}