{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/prior-knowledge-integration-via-llm-encoding","title":"Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval","arxiv_id":"2407.15051","date":"2024-07-21","proceeding":null,"authors":["Yiyang Jiang","WengYu Zhang","Xulu Zhang","XiaoYong Wei","Chang Wen Chen","Qing Li"],"abstract":"In this paper, we investigate the feasibility of leveraging large language models (LLMs) for integrating general knowledge and incorporating pseudo-events as priors for temporal content distribution in video moment retrieval (VMR) models. The motivation behind this study arises from the limitations of using LLMs as decoders for generating discrete textual descriptions, which hinders their direct application to continuous outputs like salience scores and inter-frame embeddings that capture inter-frame relations. To overcome these limitations, we propose utilizing LLM encoders instead of decoders. Through a feasibility study, we demonstrate that LLM encoders effectively refine inter-concept relations in multimodal embeddings, even without being trained on textual embeddings. We also show that the refinement capability of LLM encoders can be transferred to other embeddings, such as BLIP and T5, as long as these embeddings exhibit similar inter-concept similarity patterns to CLIP embeddings. We present a general framework for integrating LLM encoders into existing VMR architectures, specifically within the fusion module. Through experimental validation, we demonstrate the effectiveness of our proposed methods by achieving state-of-the-art performance in VMR. The source code can be accessed at https://github.com/fletcherjiang/LLMEPET.","url_abs":"https://arxiv.org/abs/2407.15051v3","url_pdf":"https://arxiv.org/pdf/2407.15051v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prior-knowledge-integration-via-llm-encoding","repo_url":"https://github.com/fletcherjiang/llmepet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"general-knowledge","task_name":"General Knowledge"},{"task_slug":"highlight-detection","task_name":"Highlight Detection"},{"task_slug":"moment-retrieval","task_name":"Moment Retrieval"},{"task_slug":"natural-language-moment-retrieval","task_name":"Natural Language Moment Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"video-grounding","task_name":"Video Grounding"}],"methods":[{"method_slug":"adafactor","method_name":"Adafactor"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"blip","method_name":"BLIP"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"glu","method_name":"Gated Linear Unit"},{"method_slug":"inverse-square-root-schedule","method_name":"Inverse Square Root Schedule"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sentencepiece","method_name":"SentencePiece"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"t5","method_name":"T5"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/highlight-detection-on-qvhighlights","task":"Highlight Detection","dataset":"QVHighlights","model":"LLMEPET","rank_in_archive_order":11,"of":21,"metrics":{"Hit@1":"65.69","mAP":"40.33"},"uses_additional_data":false},{"leaderboard":"/sota/highlight-detection-on-youtube-highlights","task":"Highlight Detection","dataset":"YouTube Highlights","model":"LLMEPET","rank_in_archive_order":6,"of":7,"metrics":{"mAP":"75.3"},"uses_additional_data":false},{"leaderboard":"/sota/moment-retrieval-on-charades-sta","task":"Moment Retrieval","dataset":"Charades-STA","model":"LLMEPET","rank_in_archive_order":16,"of":25,"metrics":{"R@1 IoU=0.5":"58.31","R@1 IoU=0.7":"36.49"},"uses_additional_data":false},{"leaderboard":"/sota/moment-retrieval-on-qvhighlights","task":"Moment Retrieval","dataset":"QVHighlights","model":"LLMEPET","rank_in_archive_order":15,"of":32,"metrics":{"R@1 IoU=0.5":"66.73","R@1 IoU=0.7":"49.94","mAP":"44.05","mAP@0.5":"65.76","mAP@0.75":"43.91"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-moment-retrieval-on-tacos","task":"Natural Language Moment Retrieval","dataset":"TACoS","model":"LLMEPET","rank_in_archive_order":7,"of":13,"metrics":{"R@1,IoU=0.3":"52.73","R@1,IoU=0.5":"40.12","R@1,IoU=0.7":"22.78","mIoU":"36.55"},"uses_additional_data":false},{"leaderboard":"/sota/video-grounding-on-qvhighlights","task":"Video Grounding","dataset":"QVHighlights","model":"LLMEPET","rank_in_archive_order":3,"of":7,"metrics":{"R@1,IoU=0.5":"66.73","R@1,IoU=0.7":"49.94"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.15051","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.15051"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fletcherjiang/llmepet","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran_fixture":2,"ran":1,"ran_draft_wrong":1,"ran_violates":1,"unverified":6},"by_repo_kind":{"official":{"samples":11,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b47d48e431b34acd","entry":"apply_rotary_emb","repo":"fletcherjiang/llmepet","repo_kind":"official","path":"llm_epet/llama.py","file_url":"https://github.com/fletcherjiang/llmepet/blob/HEAD/llm_epet/llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b47d48e431b34acd"}},{"code_sha256_prefix":"79c51f8fc390ab83","entry":"build_position_encoding","repo":"fletcherjiang/llmepet","repo_kind":"official","path":"llm_epet/position_encoding.py","file_url":"https://github.com/fletcherjiang/llmepet/blob/HEAD/llm_epet/position_encoding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"79c51f8fc390ab83"}},{"code_sha256_prefix":"6897880b6ce14c94","entry":"inverse_sigmoid","repo":"fletcherjiang/llmepet","repo_kind":"official","path":"llm_epet/model.py","file_url":"https://github.com/fletcherjiang/llmepet/blob/HEAD/llm_epet/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"6897880b6ce14c94"}},{"code_sha256_prefix":"14a84c2cbfebc413","entry":"precompute_freqs_cis","repo":"fletcherjiang/llmepet","repo_kind":"official","path":"llm_epet/llama.py","file_url":"https://github.com/fletcherjiang/llmepet/blob/HEAD/llm_epet/llama.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"14a84c2cbfebc413"}},{"code_sha256_prefix":"70bf6ebaafd266c4","entry":"reshape_for_broadcast","repo":"fletcherjiang/llmepet","repo_kind":"official","path":"llm_epet/llama.py","file_url":"https://github.com/fletcherjiang/llmepet/blob/HEAD/llm_epet/llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"70bf6ebaafd266c4"}},{"code_sha256_prefix":"081c8abd57d082fd","entry":"accuracy","repo":"fletcherjiang/llmepet","repo_kind":"official","path":"llm_epet/misc.py","file_url":"https://github.com/fletcherjiang/llmepet/blob/HEAD/llm_epet/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"081c8abd57d082fd"}},{"code_sha256_prefix":"edb929dece68582e","entry":"count_parameters","repo":"fletcherjiang/llmepet","repo_kind":"official","path":"utils/model_utils.py","file_url":"https://github.com/fletcherjiang/llmepet/blob/HEAD/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"edb929dece68582e"}},{"code_sha256_prefix":"6d07870656ce939e","entry":"element_wise_list_equal","repo":"fletcherjiang/llmepet","repo_kind":"official","path":"llm_epet/model.py","file_url":"https://github.com/fletcherjiang/llmepet/blob/HEAD/llm_epet/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"6d07870656ce939e"}},{"code_sha256_prefix":"f2157348a4ae2f51","entry":"find_nth","repo":"fletcherjiang/llmepet","repo_kind":"official","path":"llm_epet/model.py","file_url":"https://github.com/fletcherjiang/llmepet/blob/HEAD/llm_epet/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f2157348a4ae2f51"}},{"code_sha256_prefix":"a82091666b9db8cf","entry":"multi_head_attention_forward","repo":"fletcherjiang/llmepet","repo_kind":"official","path":"llm_epet/attention.py","file_url":"https://github.com/fletcherjiang/llmepet/blob/HEAD/llm_epet/attention.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"a82091666b9db8cf"}},{"code_sha256_prefix":"52dcd63a2ee1f51a","entry":"multi_head_attention_forward","repo":"fletcherjiang/llmepet","repo_kind":"official","path":"llm_epet/crossattention.py","file_url":"https://github.com/fletcherjiang/llmepet/blob/HEAD/llm_epet/crossattention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"52dcd63a2ee1f51a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}