{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multi-granularity-correspondence-learning-1","title":"Multi-granularity Correspondence Learning from Long-term Noisy Videos","arxiv_id":"2401.16702","date":"2024-01-30","proceeding":null,"authors":["Yijie Lin","Jie Zhang","Zhenyu Huang","Jia Liu","Zujie Wen","Xi Peng"],"abstract":"Existing video-language studies mainly focus on learning short video clips, leaving long-term temporal dependencies rarely explored due to over-high computational cost of modeling long videos. To address this issue, one feasible solution is learning the correspondence between video clips and captions, which however inevitably encounters the multi-granularity noisy correspondence (MNC) problem. To be specific, MNC refers to the clip-caption misalignment (coarse-grained) and frame-word misalignment (fine-grained), hindering temporal learning and video understanding. In this paper, we propose NOise Robust Temporal Optimal traNsport (Norton) that addresses MNC in a unified optimal transport (OT) framework. In brief, Norton employs video-paragraph and clip-caption contrastive losses to capture long-term dependencies based on OT. To address coarse-grained misalignment in video-paragraph contrast, Norton filters out the irrelevant clips and captions through an alignable prompt bucket and realigns asynchronous clip-caption pairs based on transport distance. To address the fine-grained misalignment, Norton incorporates a soft-maximum operator to identify crucial words and key frames. Additionally, Norton exploits the potential faulty negative samples in clip-caption contrast by rectifying the alignment target with OT assignment to ensure precise temporal modeling. Extensive experiments on video retrieval, videoQA, and action segmentation verify the effectiveness of our method. Code is available at https://lin-yijie.github.io/projects/Norton.","url_abs":"https://arxiv.org/abs/2401.16702v1","url_pdf":"https://arxiv.org/pdf/2401.16702v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multi-granularity-correspondence-learning-1","repo_url":"https://github.com/XLearning-SCU/2024-ICLR-Norton","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-segmentation","task_name":"Action Segmentation"},{"task_slug":"long-video-retrieval-background-removed","task_name":"Long Video Retrieval (Background Removed)"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-segmentation-on-coin","task":"Action Segmentation","dataset":"COIN","model":"Norton","rank_in_archive_order":3,"of":9,"metrics":{"Frame accuracy":"69.8"},"uses_additional_data":true},{"leaderboard":"/sota/long-video-retrieval-background-removed-on","task":"Long Video Retrieval (Background Removed)","dataset":"YouCook2","model":"Norton","rank_in_archive_order":1,"of":6,"metrics":{"Cap. Avg. R@1":"75.5","Cap. Avg. R@10":"97.7","Cap. Avg. R@5":"95.0","DTW R@1":"88.7","DTW R@10":"99.5","DTW R@5":"98.8","OTAM R@1":"88.9","OTAM R@10":"99.5","OTAM R@5":"98.4"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-msrvtt-mc","task":"Video Question Answering","dataset":"MSRVTT-MC","model":"Norton","rank_in_archive_order":6,"of":7,"metrics":{"Accuracy":"92.7"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msr-vtt","task":"Zero-Shot Video Retrieval","dataset":"MSR-VTT","model":"Norton","rank_in_archive_order":35,"of":41,"metrics":{"text-to-video R@1":"10.7","text-to-video R@5":"24.1"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-retrieval-on-youcook2","task":"Zero-Shot Video Retrieval","dataset":"YouCook2","model":"Norton","rank_in_archive_order":2,"of":9,"metrics":{"text-to-video R@1":"24.2","text-to-video R@10":"64.1","text-to-video R@5":"51.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.16702","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.16702"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/XLearning-SCU/2024-ICLR-Norton","reach":null}],"summary":{"ran":2,"unverified":1},"by_repo_kind":{"listed":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3c604915121f5194","entry":"Loss","repo":"XLearning-SCU/2024-ICLR-Norton","repo_kind":"listed","path":"mmpt/losses/nce.py","file_url":"https://github.com/XLearning-SCU/2024-ICLR-Norton/blob/HEAD/mmpt/losses/nce.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c604915121f5194"}},{"code_sha256_prefix":"fa8f4f2b0da67ea8","entry":"SequenceContrastLoss","repo":"XLearning-SCU/2024-ICLR-Norton","repo_kind":"listed","path":"mmpt/losses/nce.py","file_url":"https://github.com/XLearning-SCU/2024-ICLR-Norton/blob/HEAD/mmpt/losses/nce.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa8f4f2b0da67ea8"}},{"code_sha256_prefix":"14142a69635283c4","entry":"MMContraLoss","repo":"XLearning-SCU/2024-ICLR-Norton","repo_kind":"listed","path":"mmpt/losses/nce.py","file_url":"https://github.com/XLearning-SCU/2024-ICLR-Norton/blob/HEAD/mmpt/losses/nce.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"14142a69635283c4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}