{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-audio-visual-segmentation-with","title":"Improving Audio-Visual Segmentation with Bidirectional Generation","arxiv_id":"2308.08288","date":"2023-08-16","proceeding":null,"authors":["Dawei Hao","Yuxin Mao","Bowen He","Xiaodong Han","Yuchao Dai","Yiran Zhong"],"abstract":"The aim of audio-visual segmentation (AVS) is to precisely differentiate audible objects within videos down to the pixel level. Traditional approaches often tackle this challenge by combining information from various modalities, where the contribution of each modality is implicitly or explicitly modeled. Nevertheless, the interconnections between different modalities tend to be overlooked in audio-visual modeling. In this paper, inspired by the human ability to mentally simulate the sound of an object and its visual appearance, we introduce a bidirectional generation framework. This framework establishes robust correlations between an object's visual characteristics and its associated sound, thereby enhancing the performance of AVS. To achieve this, we employ a visual-to-audio projection component that reconstructs audio features from object segmentation masks and minimizes reconstruction errors. Moreover, recognizing that many sounds are linked to object movements, we introduce an implicit volumetric motion estimation module to handle temporal dynamics that may be challenging to capture using conventional optical flow methods. To showcase the effectiveness of our approach, we conduct comprehensive experiments and analyses on the widely recognized AVSBench benchmark. As a result, we establish a new state-of-the-art performance level in the AVS benchmark, particularly excelling in the challenging MS3 subset which involves segmenting multiple sound sources. To facilitate reproducibility, we plan to release both the source code and the pre-trained model.","url_abs":"https://arxiv.org/abs/2308.08288v2","url_pdf":"https://arxiv.org/pdf/2308.08288v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-audio-visual-segmentation-with","repo_url":"https://github.com/opennlplab/avs-bidirectional","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"motion-estimation","task_name":"Motion Estimation"},{"task_slug":"object","task_name":"Object"},{"task_slug":"optical-flow-estimation","task_name":"Optical Flow Estimation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.08288","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.08288"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opennlplab/avs-bidirectional","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"2e405eca70ac9fb0","entry":"load_audio_lm","repo":"opennlplab/avs-bidirectional","repo_kind":"official","path":"datasets/dataset_utils.py","file_url":"https://github.com/opennlplab/avs-bidirectional/blob/HEAD/datasets/dataset_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2e405eca70ac9fb0"}},{"code_sha256_prefix":"fe732dcb1c93487d","entry":"load_image_in_PIL_to_Tensor","repo":"opennlplab/avs-bidirectional","repo_kind":"official","path":"datasets/dataset_utils.py","file_url":"https://github.com/opennlplab/avs-bidirectional/blob/HEAD/datasets/dataset_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe732dcb1c93487d"}},{"code_sha256_prefix":"513e857ce68fed87","entry":"structure_loss_ms3","repo":"opennlplab/avs-bidirectional","repo_kind":"official","path":"loss.py","file_url":"https://github.com/opennlplab/avs-bidirectional/blob/HEAD/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"513e857ce68fed87"}},{"code_sha256_prefix":"56f1b6a24f85080a","entry":"F1_IoU_BCELoss","repo":"opennlplab/avs-bidirectional","repo_kind":"official","path":"loss.py","file_url":"https://github.com/opennlplab/avs-bidirectional/blob/HEAD/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56f1b6a24f85080a"}},{"code_sha256_prefix":"3c639ab917664e52","entry":"structure_loss_s4","repo":"opennlplab/avs-bidirectional","repo_kind":"official","path":"loss.py","file_url":"https://github.com/opennlplab/avs-bidirectional/blob/HEAD/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c639ab917664e52"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}