{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-grounded-vision-language","title":"Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos","arxiv_id":"2303.06378","date":"2023-03-11","proceeding":null,"authors":["Teng Wang","Jinrui Zhang","Feng Zheng","Wenhao Jiang","Ran Cheng","Ping Luo"],"abstract":"Joint video-language learning has received increasing attention in recent years. However, existing works mainly focus on single or multiple trimmed video clips (events), which makes human-annotated event boundaries necessary during inference. To break away from the ties, we propose a grounded vision-language learning framework for untrimmed videos, which automatically detects informative events and effectively excavates the alignments between multi-sentence descriptions and corresponding event segments. Instead of coarse-level video-language alignments, we present two dual pretext tasks to encourage fine-grained segment-level alignments, i.e., text-to-event grounding (TEG) and event-to-text generation (ETG). TEG learns to adaptively ground the possible event proposals given a set of sentences by estimating the cross-modal distance in a joint semantic space. Meanwhile, ETG aims to reconstruct (generate) the matched texts given event proposals, encouraging the event representation to retain meaningful semantic information. To encourage accurate label assignment between the event set and the text set, we propose a novel semantic-aware cost to mitigate the sub-optimal matching results caused by ambiguous boundary annotations. Our framework is easily extensible to tasks covering visually-grounded language understanding and generation. We achieve state-of-the-art dense video captioning performance on ActivityNet Captions, YouCook2 and YouMakeup, and competitive performance on several other language generation and understanding tasks. Our method also achieved 1st place in both the MTVG and MDVC tasks of the PIC 4th Challenge. Our code is publicly available at https://github.com/zjr2000/GVL.","url_abs":"https://arxiv.org/abs/2303.06378v2","url_pdf":"https://arxiv.org/pdf/2303.06378v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-grounded-vision-language","repo_url":"https://github.com/zjr2000/gvl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"dense-video-captioning","task_name":"Dense Video Captioning"},{"task_slug":"natural-language-moment-retrieval","task_name":"Natural Language Moment Retrieval"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"video-captioning","task_name":"Video Captioning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/dense-video-captioning-on-activitynet","task":"Dense Video Captioning","dataset":"ActivityNet Captions","model":"GVL","rank_in_archive_order":4,"of":12,"metrics":{"CIDEr":"33.33","METEOR":"10.03","SODA":"7.11"},"uses_additional_data":false},{"leaderboard":"/sota/dense-video-captioning-on-youcook2","task":"Dense Video Captioning","dataset":"YouCook2","model":"GVL","rank_in_archive_order":5,"of":7,"metrics":{"CIDEr":"26.52","METEOR":"5.01","SODA":"4.91"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-moment-retrieval-on","task":"Natural Language Moment Retrieval","dataset":"ActivityNet Captions","model":"GVL (paragraph-level)","rank_in_archive_order":1,"of":8,"metrics":{"R@1,IoU=0.5":"60.67","R@1,IoU=0.7":"38.55"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-moment-retrieval-on","task":"Natural Language Moment Retrieval","dataset":"ActivityNet Captions","model":"GVL","rank_in_archive_order":3,"of":8,"metrics":{"R@1,IoU=0.5":"49.18","R@1,IoU=0.7":"29.69"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-moment-retrieval-on-tacos","task":"Natural Language Moment Retrieval","dataset":"TACoS","model":"GVL (paragraph-level)","rank_in_archive_order":10,"of":13,"metrics":{"R@1,IoU=0.3":"48.29","R@1,IoU=0.5":"36.07"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-moment-retrieval-on-tacos","task":"Natural Language Moment Retrieval","dataset":"TACoS","model":"GVL","rank_in_archive_order":11,"of":13,"metrics":{"R@1,IoU=0.3":"45.92","R@1,IoU=0.5":"34.57"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.06378","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.06378"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zjr2000/gvl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"unverified":11},"by_repo_kind":{"official":{"samples":12,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"81aab88321de62c2","entry":"interpolated_prec_rec","repo":"zjr2000/gvl","repo_kind":"official","path":"densevid_eval3/eval_tal_utils.py","file_url":"https://github.com/zjr2000/gvl/blob/HEAD/densevid_eval3/eval_tal_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81aab88321de62c2"}},{"code_sha256_prefix":"53e64888999b6cb7","entry":"calculate_avg_proposal_num","repo":"zjr2000/gvl","repo_kind":"official","path":"eval_utils.py","file_url":"https://github.com/zjr2000/gvl/blob/HEAD/eval_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53e64888999b6cb7"}},{"code_sha256_prefix":"cc29ecc7589efdff","entry":"collate_fn","repo":"zjr2000/gvl","repo_kind":"official","path":"video_dataset.py","file_url":"https://github.com/zjr2000/gvl/blob/HEAD/video_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc29ecc7589efdff"}},{"code_sha256_prefix":"85e25c640a683e5a","entry":"collate_fn","repo":"zjr2000/gvl","repo_kind":"official","path":"video_dataset_with_data_aug.py","file_url":"https://github.com/zjr2000/gvl/blob/HEAD/video_dataset_with_data_aug.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85e25c640a683e5a"}},{"code_sha256_prefix":"0f585e754688f3eb","entry":"get_blocked_videos","repo":"zjr2000/gvl","repo_kind":"official","path":"densevid_eval3/eval_tal_utils.py","file_url":"https://github.com/zjr2000/gvl/blob/HEAD/densevid_eval3/eval_tal_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f585e754688f3eb"}},{"code_sha256_prefix":"fe5aa6f8364bdae4","entry":"get_iou","repo":"zjr2000/gvl","repo_kind":"official","path":"densevid_eval3/eval_grounding.py","file_url":"https://github.com/zjr2000/gvl/blob/HEAD/densevid_eval3/eval_grounding.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe5aa6f8364bdae4"}},{"code_sha256_prefix":"931c2ee1f4c9cd81","entry":"get_miou","repo":"zjr2000/gvl","repo_kind":"official","path":"densevid_eval3/eval_grounding.py","file_url":"https://github.com/zjr2000/gvl/blob/HEAD/densevid_eval3/eval_grounding.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"931c2ee1f4c9cd81"}},{"code_sha256_prefix":"c6a05eda2a29c456","entry":"get_recall_at_k","repo":"zjr2000/gvl","repo_kind":"official","path":"densevid_eval3/eval_grounding.py","file_url":"https://github.com/zjr2000/gvl/blob/HEAD/densevid_eval3/eval_grounding.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6a05eda2a29c456"}},{"code_sha256_prefix":"004b519990491671","entry":"iou","repo":"zjr2000/gvl","repo_kind":"official","path":"video_dataset.py","file_url":"https://github.com/zjr2000/gvl/blob/HEAD/video_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"004b519990491671"}},{"code_sha256_prefix":"a45f62024eb3d2c5","entry":"remove_nonascii","repo":"zjr2000/gvl","repo_kind":"official","path":"densevid_eval3/evaluate2018.py","file_url":"https://github.com/zjr2000/gvl/blob/HEAD/densevid_eval3/evaluate2018.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a45f62024eb3d2c5"}},{"code_sha256_prefix":"930f6b6986cf84bf","entry":"segment_iou","repo":"zjr2000/gvl","repo_kind":"official","path":"densevid_eval3/eval_tal_utils.py","file_url":"https://github.com/zjr2000/gvl/blob/HEAD/densevid_eval3/eval_tal_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"930f6b6986cf84bf"}},{"code_sha256_prefix":"956db474f5613b75","entry":"sort_events","repo":"zjr2000/gvl","repo_kind":"official","path":"video_dataset.py","file_url":"https://github.com/zjr2000/gvl/blob/HEAD/video_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"956db474f5613b75"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}