{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/video-guided-curriculum-learning-for-spoken","title":"Video-Guided Curriculum Learning for Spoken Video Grounding","arxiv_id":"2209.00277","date":"2022-09-01","proceeding":null,"authors":["Yan Xia","Zhou Zhao","Shangwei Ye","Yang Zhao","Haoyuan Li","Yi Ren"],"abstract":"In this paper, we introduce a new task, spoken video grounding (SVG), which aims to localize the desired video fragments from spoken language descriptions. Compared with using text, employing audio requires the model to directly exploit the useful phonemes and syllables related to the video from raw speech. Moreover, we randomly add environmental noises to this speech audio, further increasing the difficulty of this task and better simulating real applications. To rectify the discriminative phonemes and extract video-related information from noisy audio, we develop a novel video-guided curriculum learning (VGCL) during the audio pre-training process, which can make use of the vital visual perceptions to help understand the spoken language and suppress the external noise. Considering during inference the model can not obtain ground truth video segments, we design a curriculum strategy that gradually shifts the input video from the ground truth to the entire video content during pre-training. Finally, the model can learn how to extract critical visual information from the entire video clip to help understand the spoken language. In addition, we collect the first large-scale spoken video grounding dataset based on ActivityNet, which is named as ActivityNet Speech dataset. Extensive experiments demonstrate our proposed video-guided curriculum learning can facilitate the pre-training process to obtain a mutual audio encoder, significantly promoting the performance of spoken video grounding tasks. Moreover, we prove that in the case of noisy sound, our model outperforms the method that grounding video with ASR transcripts, further demonstrating the effectiveness of our curriculum strategy.","url_abs":"https://arxiv.org/abs/2209.00277v1","url_pdf":"https://arxiv.org/pdf/2209.00277v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"video-guided-curriculum-learning-for-spoken","repo_url":"https://github.com/marmot-xy/spoken-video-grounding","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"video-grounding","task_name":"Video Grounding"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2209.00277","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.00277"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/marmot-xy/spoken-video-grounding","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8f266b9d616de9b9","entry":"fill_with_neg_inf","repo":"marmot-xy/spoken-video-grounding","repo_kind":"official","path":"modules/attention_layers.py","file_url":"https://github.com/marmot-xy/spoken-video-grounding/blob/HEAD/modules/attention_layers.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f266b9d616de9b9"}},{"code_sha256_prefix":"9ba9ba4e02b0453d","entry":"analog_normal_dist","repo":"marmot-xy/spoken-video-grounding","repo_kind":"official","path":"model/loss.py","file_url":"https://github.com/marmot-xy/spoken-video-grounding/blob/HEAD/model/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ba9ba4e02b0453d"}},{"code_sha256_prefix":"63678ba35090fef3","entry":"calc_cls_loss","repo":"marmot-xy/spoken-video-grounding","repo_kind":"official","path":"model/loss.py","file_url":"https://github.com/marmot-xy/spoken-video-grounding/blob/HEAD/model/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63678ba35090fef3"}},{"code_sha256_prefix":"b287aaa4184d5e80","entry":"construct_target_cls","repo":"marmot-xy/spoken-video-grounding","repo_kind":"official","path":"model/loss.py","file_url":"https://github.com/marmot-xy/spoken-video-grounding/blob/HEAD/model/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b287aaa4184d5e80"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}