{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/joint-modal-label-denoising-for-weakly","title":"Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing","arxiv_id":"2204.11573","date":"2022-04-25","proceeding":null,"authors":["Haoyue Cheng","Zhaoyang Liu","Hang Zhou","Chen Qian","Wayne Wu","LiMin Wang"],"abstract":"This paper focuses on the weakly-supervised audio-visual video parsing task, which aims to recognize all events belonging to each modality and localize their temporal boundaries. This task is challenging because only overall labels indicating the video events are provided for training. However, an event might be labeled but not appear in one of the modalities, which results in a modality-specific noisy label problem. In this work, we propose a training strategy to identify and remove modality-specific noisy labels dynamically. It is motivated by two key observations: 1) networks tend to learn clean samples first; and 2) a labeled event would appear in at least one modality. Specifically, we sort the losses of all instances within a mini-batch individually in each modality, and then select noisy samples according to the relationships between intra-modal and inter-modal losses. Besides, we also propose a simple but valid noise ratio estimation method by calculating the proportion of instances whose confidence is below a preset threshold. Our method makes large improvements over the previous state of the arts (e.g. from 60.0\\% to 63.8\\% in segment-level visual metric), which demonstrates the effectiveness of our approach. Code and trained models are publicly available at \\url{https://github.com/MCG-NJU/JoMoLD}.","url_abs":"https://arxiv.org/abs/2204.11573v4","url_pdf":"https://arxiv.org/pdf/2204.11573v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"joint-modal-label-denoising-for-weakly","repo_url":"https://github.com/mcg-nju/jomold","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"joint-modal-label-denoising-for-weakly","repo_url":"https://github.com/mengyuanchen21/cvpr2023-cmpae","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":null,"task_name":"valid"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2204.11573","atlas_url":"https://app.syntology.ai/?focus=2204.11573","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.11573"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mcg-nju/jomold","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mengyuanchen21/cvpr2023-cmpae","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/MCG-NJU/JoMoLD","reach":{"status":"ok"}}],"summary":{"ran":3,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1},"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"09c6cee14764da2c","entry":"Encoder","repo":"MCG-NJU/JoMoLD","repo_kind":"official","path":"nets/net_audiovisual.py","file_url":"https://github.com/MCG-NJU/JoMoLD/blob/HEAD/nets/net_audiovisual.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09c6cee14764da2c"}},{"code_sha256_prefix":"dc4267bfaf17ac28","entry":"HANLayer","repo":"MCG-NJU/JoMoLD","repo_kind":"official","path":"nets/net_audiovisual.py","file_url":"https://github.com/MCG-NJU/JoMoLD/blob/HEAD/nets/net_audiovisual.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc4267bfaf17ac28"}},{"code_sha256_prefix":"4913d3d548ebcc79","entry":"MMIL_Net","repo":"mengyuanchen21/cvpr2023-cmpae","repo_kind":"listed","path":"nets/net_audiovisual.py","file_url":"https://github.com/mengyuanchen21/cvpr2023-cmpae/blob/HEAD/nets/net_audiovisual.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4913d3d548ebcc79"}},{"code_sha256_prefix":"73ab74fee9265b2c","entry":"exp_evidence","repo":"mengyuanchen21/cvpr2023-cmpae","repo_kind":"listed","path":"nets/net_audiovisual.py","file_url":"https://github.com/mengyuanchen21/cvpr2023-cmpae/blob/HEAD/nets/net_audiovisual.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73ab74fee9265b2c"}},{"code_sha256_prefix":"ac4ccd836b0b4d36","entry":"get_p_and_u_from_logit","repo":"mengyuanchen21/cvpr2023-cmpae","repo_kind":"listed","path":"nets/net_audiovisual.py","file_url":"https://github.com/mengyuanchen21/cvpr2023-cmpae/blob/HEAD/nets/net_audiovisual.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac4ccd836b0b4d36"}},{"code_sha256_prefix":"e7739be4fe052b6e","entry":"MMIL_Net","repo":"MCG-NJU/JoMoLD","repo_kind":"official","path":"nets/net_audiovisual.py","file_url":"https://github.com/MCG-NJU/JoMoLD/blob/HEAD/nets/net_audiovisual.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7739be4fe052b6e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}