{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multi-modality-guidance-network-for-missing","title":"Multimodal Guidance Network for Missing-Modality Inference in Content Moderation","arxiv_id":"2309.03452","date":"2023-09-07","proceeding":null,"authors":["Zhuokai Zhao","Harish Palani","Tianyi Liu","Lena Evans","Ruth Toner"],"abstract":"Multimodal deep learning, especially vision-language models, have gained significant traction in recent years, greatly improving performance on many downstream tasks, including content moderation and violence detection. However, standard multimodal approaches often assume consistent modalities between training and inference, limiting applications in many real-world use cases, as some modalities may not be available during inference. While existing research mitigates this problem through reconstructing the missing modalities, they unavoidably increase unnecessary computational cost, which could be just as critical, especially for large, deployed infrastructures in industry. To this end, we propose a novel guidance network that promotes knowledge sharing during training, taking advantage of the multimodal representations to train better single-modality models to be used for inference. Real-world experiments in violence detection shows that our proposed framework trains single-modality models that significantly outperform traditionally trained counterparts, while avoiding increases in computational cost for inference.","url_abs":"https://arxiv.org/abs/2309.03452v2","url_pdf":"https://arxiv.org/pdf/2309.03452v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multi-modality-guidance-network-for-missing","repo_url":"https://github.com/zhuokaizhao/multimodal-guidance-network","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multimodal-deep-learning","task_name":"Multimodal Deep Learning"}],"methods":[{"method_slug":"fail","method_name":"fail"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.03452","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.03452"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhuokaizhao/multimodal-guidance-network","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":6},"by_repo_kind":{"official":{"samples":10,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ea664c7167d26a47","entry":"load_data_for_testing","repo":"zhuokaizhao/multimodal-guidance-network","repo_kind":"official","path":"CLIP/vl_dataset.py","file_url":"https://github.com/zhuokaizhao/multimodal-guidance-network/blob/HEAD/CLIP/vl_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea664c7167d26a47"}},{"code_sha256_prefix":"31fc31433de953de","entry":"load_data_for_training","repo":"zhuokaizhao/multimodal-guidance-network","repo_kind":"official","path":"CLIP/vl_dataset.py","file_url":"https://github.com/zhuokaizhao/multimodal-guidance-network/blob/HEAD/CLIP/vl_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31fc31433de953de"}},{"code_sha256_prefix":"83c7619610642884","entry":"load_model","repo":"zhuokaizhao/multimodal-guidance-network","repo_kind":"official","path":"MobileOne/finetune_mobileone_single.py","file_url":"https://github.com/zhuokaizhao/multimodal-guidance-network/blob/HEAD/MobileOne/finetune_mobileone_single.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83c7619610642884"}},{"code_sha256_prefix":"26d8eb2d92c74ef9","entry":"reparameterize_model","repo":"zhuokaizhao/multimodal-guidance-network","repo_kind":"official","path":"vl_model/mobileone.py","file_url":"https://github.com/zhuokaizhao/multimodal-guidance-network/blob/HEAD/vl_model/mobileone.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26d8eb2d92c74ef9"}},{"code_sha256_prefix":"546d4f4b91ff8133","entry":"get_tokenizer_settings","repo":"zhuokaizhao/multimodal-guidance-network","repo_kind":"official","path":"vl_model/vl_dateset.py","file_url":"https://github.com/zhuokaizhao/multimodal-guidance-network/blob/HEAD/vl_model/vl_dateset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"546d4f4b91ff8133"}},{"code_sha256_prefix":"46ce8ca40f317139","entry":"load_data_for_testing","repo":"zhuokaizhao/multimodal-guidance-network","repo_kind":"official","path":"vl_model/vl_dateset.py","file_url":"https://github.com/zhuokaizhao/multimodal-guidance-network/blob/HEAD/vl_model/vl_dateset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"46ce8ca40f317139"}},{"code_sha256_prefix":"400aef55175bd438","entry":"load_data_for_training","repo":"zhuokaizhao/multimodal-guidance-network","repo_kind":"official","path":"vl_model/vl_dateset.py","file_url":"https://github.com/zhuokaizhao/multimodal-guidance-network/blob/HEAD/vl_model/vl_dateset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"400aef55175bd438"}},{"code_sha256_prefix":"e4a846c5456a0af7","entry":"load_model","repo":"zhuokaizhao/multimodal-guidance-network","repo_kind":"official","path":"MobileOne/finetune_mobileone.py","file_url":"https://github.com/zhuokaizhao/multimodal-guidance-network/blob/HEAD/MobileOne/finetune_mobileone.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4a846c5456a0af7"}},{"code_sha256_prefix":"c530aa04420eb8d2","entry":"mobileone","repo":"zhuokaizhao/multimodal-guidance-network","repo_kind":"official","path":"MobileOne/mobileone.py","file_url":"https://github.com/zhuokaizhao/multimodal-guidance-network/blob/HEAD/MobileOne/mobileone.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c530aa04420eb8d2"}},{"code_sha256_prefix":"a450e5652620e756","entry":"mobileone_encoder","repo":"zhuokaizhao/multimodal-guidance-network","repo_kind":"official","path":"vl_model/mobileone.py","file_url":"https://github.com/zhuokaizhao/multimodal-guidance-network/blob/HEAD/vl_model/mobileone.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a450e5652620e756"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}