{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/continual-audio-visual-sound-separation","title":"Continual Audio-Visual Sound Separation","arxiv_id":"2411.02860","date":"2024-11-05","proceeding":null,"authors":["Weiguo Pian","Yiyang Nan","Shijian Deng","Shentong Mo","Yunhui Guo","Yapeng Tian"],"abstract":"In this paper, we introduce a novel continual audio-visual sound separation task, aiming to continuously separate sound sources for new classes while preserving performance on previously learned classes, with the aid of visual guidance. This problem is crucial for practical visually guided auditory perception as it can significantly enhance the adaptability and robustness of audio-visual sound separation models, making them more applicable for real-world scenarios where encountering new sound sources is commonplace. The task is inherently challenging as our models must not only effectively utilize information from both modalities in current tasks but also preserve their cross-modal association in old tasks to mitigate catastrophic forgetting during audio-visual continual learning. To address these challenges, we propose a novel approach named ContAV-Sep (\\textbf{Cont}inual \\textbf{A}udio-\\textbf{V}isual Sound \\textbf{Sep}aration). ContAV-Sep presents a novel Cross-modal Similarity Distillation Constraint (CrossSDC) to uphold the cross-modal semantic similarity through incremental tasks and retain previously acquired knowledge of semantic similarity in old models, mitigating the risk of catastrophic forgetting. The CrossSDC can seamlessly integrate into the training process of different audio-visual sound separation frameworks. Experiments demonstrate that ContAV-Sep can effectively mitigate catastrophic forgetting and achieve significantly better performance compared to other continual learning baselines for audio-visual sound separation. Code is available at: \\url{https://github.com/weiguoPian/ContAV-Sep_NeurIPS2024}.","url_abs":"https://arxiv.org/abs/2411.02860v1","url_pdf":"https://arxiv.org/pdf/2411.02860v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"continual-audio-visual-sound-separation","repo_url":"https://github.com/weiguopian/contav-sep_neurips2024","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"}],"methods":[{"method_slug":null,"method_name":"Uphold"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.02860","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.02860"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/weiguoPian/ContAV-Sep_NeurIPS2024","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/weiguopian/contav-sep_neurips2024","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"a4a08206de81190b","entry":"warpgrid","repo":"weiguoPian/ContAV-Sep_NeurIPS2024","repo_kind":"official","path":"utils.py","file_url":"https://github.com/weiguoPian/ContAV-Sep_NeurIPS2024/blob/HEAD/utils.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4a08206de81190b"}},{"code_sha256_prefix":"e52e3c5bf79fea6f","entry":"magnitude2heatmap","repo":"weiguoPian/ContAV-Sep_NeurIPS2024","repo_kind":"official","path":"utils.py","file_url":"https://github.com/weiguoPian/ContAV-Sep_NeurIPS2024/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e52e3c5bf79fea6f"}},{"code_sha256_prefix":"474284d4bfef4f7c","entry":"positionalencoding1d","repo":"weiguoPian/ContAV-Sep_NeurIPS2024","repo_kind":"official","path":"models/maskformer_predictor_motion.py","file_url":"https://github.com/weiguoPian/ContAV-Sep_NeurIPS2024/blob/HEAD/models/maskformer_predictor_motion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"474284d4bfef4f7c"}},{"code_sha256_prefix":"3e5b3b6994917c24","entry":"recover_rgb","repo":"weiguoPian/ContAV-Sep_NeurIPS2024","repo_kind":"official","path":"utils.py","file_url":"https://github.com/weiguoPian/ContAV-Sep_NeurIPS2024/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e5b3b6994917c24"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}