{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scanet-a-self-and-cross-attention-network-for","title":"IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separation","arxiv_id":"2308.08143","date":"2023-08-16","proceeding":null,"authors":["Kai Li","Runxuan Yang","Fuchun Sun","Xiaolin Hu"],"abstract":"Recent research has made significant progress in designing fusion modules for audio-visual speech separation. However, they predominantly focus on multi-modal fusion at a single temporal scale of auditory and visual features without employing selective attention mechanisms, which is in sharp contrast with the brain. To address this issue, We propose a novel model called Intra- and Inter-Attention Network (IIANet), which leverages the attention mechanism for efficient audio-visual feature fusion. IIANet consists of two types of attention blocks: intra-attention (IntraA) and inter-attention (InterA) blocks, where the InterA blocks are distributed at the top, middle and bottom of IIANet. Heavily inspired by the way how human brain selectively focuses on relevant content at various temporal scales, these blocks maintain the ability to learn modality-specific features and enable the extraction of different semantics from audio-visual features. Comprehensive experiments on three standard audio-visual separation benchmarks (LRS2, LRS3, and VoxCeleb2) demonstrate the effectiveness of IIANet, outperforming previous state-of-the-art methods while maintaining comparable inference time. In particular, the fast version of IIANet (IIANet-fast) has only 7% of CTCNet's MACs and is 40% faster than CTCNet on CPUs while achieving better separation quality, showing the great potential of attention mechanism for efficient and effective multimodal fusion.","url_abs":"https://arxiv.org/abs/2308.08143v3","url_pdf":"https://arxiv.org/pdf/2308.08143v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scanet-a-self-and-cross-attention-network-for","repo_url":"https://github.com/JusperLee/IIANet","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"speech-separation","task_name":"Speech Separation"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/speech-separation-on-lrs2","task":"Speech Separation","dataset":"LRS2","model":"IIANet","rank_in_archive_order":1,"of":8,"metrics":{"SDRi":"16.6","SI-SNRi":"16.4"},"uses_additional_data":false},{"leaderboard":"/sota/speech-separation-on-lrs3","task":"Speech Separation","dataset":"LRS3","model":"IIANet","rank_in_archive_order":1,"of":5,"metrics":{"SDRi":"18.5","SI-SNRi":"18.3"},"uses_additional_data":false},{"leaderboard":"/sota/speech-separation-on-voxceleb2","task":"Speech Separation","dataset":"VoxCeleb2","model":"IIANet","rank_in_archive_order":1,"of":5,"metrics":{"SDRi":"15.1","SI-SNRi":"14.0"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.08143","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.08143"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JusperLee/IIANet","reach":null}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"57e4b39077199c2e","entry":"linear_interpolate","repo":"JusperLee/IIANet","repo_kind":"official","path":"inference.py","file_url":"https://github.com/JusperLee/IIANet/blob/HEAD/inference.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57e4b39077199c2e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}