{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-deep-multimodal-feature","title":"Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion","arxiv_id":"2108.05009","date":"2021-08-11","proceeding":null,"authors":["Yikai Wang","Fuchun Sun","Ming Lu","Anbang Yao"],"abstract":"We propose a compact and effective framework to fuse multimodal features at multiple layers in a single network. The framework consists of two innovative fusion schemes. Firstly, unlike existing multimodal methods that necessitate individual encoders for different modalities, we verify that multimodal features can be learnt within a shared single network by merely maintaining modality-specific batch normalization layers in the encoder, which also enables implicit fusion via joint feature representation learning. Secondly, we propose a bidirectional multi-layer fusion scheme, where multimodal features can be exploited progressively. To take advantage of such scheme, we introduce two asymmetric fusion operations including channel shuffle and pixel shift, which learn different fused features with respect to different fusion directions. These two operations are parameter-free and strengthen the multimodal feature interactions across channels as well as enhance the spatial feature discrimination within channels. We conduct extensive experiments on semantic segmentation and image translation tasks, based on three publicly available datasets covering diverse modalities. Results indicate that our proposed framework is general, compact and is superior to state-of-the-art fusion frameworks.","url_abs":"https://arxiv.org/abs/2108.05009v1","url_pdf":"https://arxiv.org/pdf/2108.05009v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-deep-multimodal-feature","repo_url":"https://github.com/yikaiw/AsymFusion","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"channel-shuffle","method_name":"Channel Shuffle"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/semantic-segmentation-on-nyu-depth-v2","task":"Semantic Segmentation","dataset":"NYU Depth v2","model":"AsymFusion (ResNet-152)","rank_in_archive_order":54,"of":121,"metrics":{"Mean IoU":"51.2%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2108.05009","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2108.05009"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yikaiw/AsymFusion","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f8a8c9728d1e4d18","entry":"compute_params","repo":"yikaiw/AsymFusion","repo_kind":"official","path":"utils/meter.py","file_url":"https://github.com/yikaiw/AsymFusion/blob/HEAD/utils/meter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f8a8c9728d1e4d18"}},{"code_sha256_prefix":"be06fe01faab93f9","entry":"confusion_matrix","repo":"yikaiw/AsymFusion","repo_kind":"official","path":"utils/meter.py","file_url":"https://github.com/yikaiw/AsymFusion/blob/HEAD/utils/meter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be06fe01faab93f9"}},{"code_sha256_prefix":"ee58726ccfd1a5c8","entry":"conv1x1","repo":"yikaiw/AsymFusion","repo_kind":"official","path":"models/model.py","file_url":"https://github.com/yikaiw/AsymFusion/blob/HEAD/models/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee58726ccfd1a5c8"}},{"code_sha256_prefix":"7c930e46d06bfa25","entry":"conv3x3","repo":"yikaiw/AsymFusion","repo_kind":"official","path":"models/model.py","file_url":"https://github.com/yikaiw/AsymFusion/blob/HEAD/models/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c930e46d06bfa25"}},{"code_sha256_prefix":"7cb6905948e4acb0","entry":"getScores","repo":"yikaiw/AsymFusion","repo_kind":"official","path":"utils/meter.py","file_url":"https://github.com/yikaiw/AsymFusion/blob/HEAD/utils/meter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7cb6905948e4acb0"}},{"code_sha256_prefix":"e4fefad569c9bb5c","entry":"line_to_paths_fn_nyudv2","repo":"yikaiw/AsymFusion","repo_kind":"official","path":"utils/datasets.py","file_url":"https://github.com/yikaiw/AsymFusion/blob/HEAD/utils/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4fefad569c9bb5c"}},{"code_sha256_prefix":"b61821ce9e53818d","entry":"make_list","repo":"yikaiw/AsymFusion","repo_kind":"official","path":"utils/transforms.py","file_url":"https://github.com/yikaiw/AsymFusion/blob/HEAD/utils/transforms.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b61821ce9e53818d"}},{"code_sha256_prefix":"14fa05652664a293","entry":"make_validation_img","repo":"yikaiw/AsymFusion","repo_kind":"official","path":"utils/helpers.py","file_url":"https://github.com/yikaiw/AsymFusion/blob/HEAD/utils/helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14fa05652664a293"}},{"code_sha256_prefix":"5d35d232de98d91d","entry":"maybe_download","repo":"yikaiw/AsymFusion","repo_kind":"official","path":"utils/helpers.py","file_url":"https://github.com/yikaiw/AsymFusion/blob/HEAD/utils/helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d35d232de98d91d"}},{"code_sha256_prefix":"0414c633729593df","entry":"prepare_img","repo":"yikaiw/AsymFusion","repo_kind":"official","path":"utils/helpers.py","file_url":"https://github.com/yikaiw/AsymFusion/blob/HEAD/utils/helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0414c633729593df"}},{"code_sha256_prefix":"1bc30127718ac5ec","entry":"refinenet","repo":"yikaiw/AsymFusion","repo_kind":"official","path":"models/model.py","file_url":"https://github.com/yikaiw/AsymFusion/blob/HEAD/models/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1bc30127718ac5ec"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}