{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-trimodal-relation-for-avqa-with","title":"Learning Trimodal Relation for AVQA with Missing Modality","arxiv_id":"2407.16171","date":"2024-07-23","proceeding":null,"authors":["Kyu Ri Park","Hong Joo Lee","Jung Uk Kim"],"abstract":"Recent Audio-Visual Question Answering (AVQA) methods rely on complete visual and audio input to answer questions accurately. However, in real-world scenarios, issues such as device malfunctions and data transmission errors frequently result in missing audio or visual modality. In such cases, existing AVQA methods suffer significant performance degradation. In this paper, we propose a framework that ensures robust AVQA performance even when a modality is missing. First, we propose a Relation-aware Missing Modal (RMM) generator with Relation-aware Missing Modal Recalling (RMMR) loss to enhance the ability of the generator to recall missing modal information by understanding the relationships and context among the available modalities. Second, we design an Audio-Visual Relation-aware (AVR) diffusion model with Audio-Visual Enhancing (AVE) loss to further enhance audio-visual features by leveraging the relationships and shared cues between the audio-visual modalities. As a result, our method can provide accurate answers by effectively utilizing available information even when input modalities are missing. We believe our method holds potential applications not only in AVQA research but also in various multi-modal scenarios.","url_abs":"https://arxiv.org/abs/2407.16171v1","url_pdf":"https://arxiv.org/pdf/2407.16171v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-trimodal-relation-for-avqa-with","repo_url":"https://github.com/visualaikhu/missing-avqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"audio-visual-question-answering-avqa","task_name":"Audio-Visual Question Answering (AVQA)"},{"task_slug":"audio-visual-question-answering","task_name":"Audio-visual Question Answering"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":null,"task_name":"Relation"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.16171","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.16171"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/VisualAIKHU/Missing-AVQA","reach":null}],"summary":{"ran":14,"ran_draft_wrong":2,"ran_honours":4,"unverified":4},"by_repo_kind":{"official":{"samples":21,"ran":17,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"b19aa687804ddbdd","entry":"AVQA_Memory","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b19aa687804ddbdd"}},{"code_sha256_prefix":"a461fe5c0890e880","entry":"Attend","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a461fe5c0890e880"}},{"code_sha256_prefix":"100973e0dc95d746","entry":"Attention","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"100973e0dc95d746"}},{"code_sha256_prefix":"f3b98aa33cb4d3fc","entry":"BasicBlock","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3b98aa33cb4d3fc"}},{"code_sha256_prefix":"1b1fcc6ab25fa4ef","entry":"Block","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b1fcc6ab25fa4ef"}},{"code_sha256_prefix":"0df99f2ca89a9f10","entry":"Bottleneck","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0df99f2ca89a9f10"}},{"code_sha256_prefix":"b5bf8353f4199efb","entry":"Downsample","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5bf8353f4199efb"}},{"code_sha256_prefix":"1cbe92e0187480a4","entry":"GaussianDiffusion","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1cbe92e0187480a4"}},{"code_sha256_prefix":"49dea4e895a85ad5","entry":"LinearAttention","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"49dea4e895a85ad5"}},{"code_sha256_prefix":"bb260ea644c95e3a","entry":"QstEncoder","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb260ea644c95e3a"}},{"code_sha256_prefix":"30e7fae83920ab01","entry":"RMSNorm","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30e7fae83920ab01"}},{"code_sha256_prefix":"b100b6c72c401001","entry":"RandomOrLearnedSinusoidalPosEmb","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b100b6c72c401001"}},{"code_sha256_prefix":"6f0b7d14d56b9a7b","entry":"ResNet","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f0b7d14d56b9a7b"}},{"code_sha256_prefix":"b18e27ad640dde3d","entry":"ResnetBlock","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b18e27ad640dde3d"}},{"code_sha256_prefix":"2b8015ff3c87273a","entry":"SinusoidalPosEmb","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b8015ff3c87273a"}},{"code_sha256_prefix":"410a7d2e8c88e682","entry":"Upsample","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"410a7d2e8c88e682"}},{"code_sha256_prefix":"d229662b12e581b6","entry":"cast_tuple","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"d229662b12e581b6"}},{"code_sha256_prefix":"079814bd981435a5","entry":"cosine_beta_schedule","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"079814bd981435a5"}},{"code_sha256_prefix":"a1229a9f4df127e2","entry":"normalize_to_neg_one_to_one","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"a1229a9f4df127e2"}},{"code_sha256_prefix":"97e1bbd232dfe092","entry":"unnormalize_to_zero_to_one","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"97e1bbd232dfe092"}},{"code_sha256_prefix":"a787fa87e1afe3f0","entry":"AVQA_Fusion_Net","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a787fa87e1afe3f0"}},{"code_sha256_prefix":"44756209b5482bd6","entry":"Unet","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44756209b5482bd6"}},{"code_sha256_prefix":"023ac56ffd33d3bc","entry":"_resnet","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"023ac56ffd33d3bc"}},{"code_sha256_prefix":"b03e2f9d7a1ce561","entry":"resnet18","repo":"VisualAIKHU/Missing-AVQA","repo_kind":"official","path":"net_grd_avst/net_avst.py","file_url":"https://github.com/VisualAIKHU/Missing-AVQA/blob/HEAD/net_grd_avst/net_avst.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b03e2f9d7a1ce561"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}