{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multi-modality-associative-bridging-through-1","title":"Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video","arxiv_id":"2204.01265","date":"2022-04-04","proceeding":"ICCV 2021 10","authors":["Minsu Kim","Joanna Hong","Se Jin Park","Yong Man Ro"],"abstract":"In this paper, we introduce a novel audio-visual multi-modal bridging framework that can utilize both audio and visual information, even with uni-modal inputs. We exploit a memory network that stores source (i.e., visual) and target (i.e., audio) modal representations, where source modal representation is what we are given, and target modal representations are what we want to obtain from the memory network. We then construct an associative bridge between source and target memories that considers the interrelationship between the two memories. By learning the interrelationship through the associative bridge, the proposed bridging framework is able to obtain the target modal representations inside the memory network, even with the source modal input only, and it provides rich information for its downstream tasks. We apply the proposed framework to two tasks: lip reading and speech reconstruction from silent video. Through the proposed associative bridge and modality-specific memories, each task knowledge is enriched with the recalled audio context, achieving state-of-the-art performance. We also verify that the associative bridge properly relates the source and target memories.","url_abs":"https://arxiv.org/abs/2204.01265v1","url_pdf":"https://arxiv.org/pdf/2204.01265v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multi-modality-associative-bridging-through-1","repo_url":"https://github.com/ms-dot-k/Visual-Audio-Memory","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"lip-reading","task_name":"Lip Reading"}],"methods":[{"method_slug":"memory-network","method_name":"Memory Network"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/lipreading-on-lrw-1000","task":"Lipreading","dataset":"CAS-VSR-W1k (LRW-1000)","model":"3D Conv + ResNet-18 + Bi-GRU + Visual-Audio Memory","rank_in_archive_order":4,"of":9,"metrics":{"Top-1 Accuracy":"50.82%"},"uses_additional_data":false},{"leaderboard":"/sota/lipreading-on-lip-reading-in-the-wild","task":"Lipreading","dataset":"Lip Reading in the Wild","model":"3D Conv + ResNet-18 + Bi-GRU + Visual-Audio Memory","rank_in_archive_order":11,"of":22,"metrics":{"Top-1 Accuracy":"85.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2204.01265","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.01265"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ms-dot-k/Visual-Audio-Memory","reach":null}],"summary":{"ran":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"d0a7a7a21b4d5d9a","entry":"Memory","repo":"ms-dot-k/Visual-Audio-Memory","repo_kind":"listed","path":"src/models/memory.py","file_url":"https://github.com/ms-dot-k/Visual-Audio-Memory/blob/HEAD/src/models/memory.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d0a7a7a21b4d5d9a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}