{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/naq-leveraging-narrations-as-queries-to","title":"NaQ: Leveraging Narrations as Queries to Supervise Episodic Memory","arxiv_id":"2301.00746","date":"2023-01-02","proceeding":"CVPR 2023 1","authors":["Santhosh Kumar Ramakrishnan","Ziad Al-Halah","Kristen Grauman"],"abstract":"Searching long egocentric videos with natural language queries (NLQ) has compelling applications in augmented reality and robotics, where a fluid index into everything that a person (agent) has seen before could augment human memory and surface relevant information on demand. However, the structured nature of the learning problem (free-form text query inputs, localized video temporal window outputs) and its needle-in-a-haystack nature makes it both technically challenging and expensive to supervise. We introduce Narrations-as-Queries (NaQ), a data augmentation strategy that transforms standard video-text narrations into training data for a video query localization model. Validating our idea on the Ego4D benchmark, we find it has tremendous impact in practice. NaQ improves multiple top models by substantial margins (even doubling their accuracy), and yields the very best results to date on the Ego4D NLQ challenge, soundly outperforming all challenge winners in the CVPR and ECCV 2022 competitions and topping the current public leaderboard. Beyond achieving the state-of-the-art for NLQ, we also demonstrate unique properties of our approach such as the ability to perform zero-shot and few-shot NLQ, and improved performance on queries about long-tail object categories. Code and models: {\\small\\url{http://vision.cs.utexas.edu/projects/naq}}.","url_abs":"https://arxiv.org/abs/2301.00746v2","url_pdf":"https://arxiv.org/pdf/2301.00746v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"naq-leveraging-narrations-as-queries-to","repo_url":"https://github.com/srama2512/NaQ","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"natural-language-queries","task_name":"Natural Language Queries"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2301.00746","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2301.00746"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/srama2512/NaQ","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran_honours":2,"ran":2,"unverified":10},"by_repo_kind":{"official":{"samples":15,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"srama2512/NaQ","repo_kind":"official","path":"ReLER/ms_cm/bert_layers.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/ReLER/ms_cm/bert_layers.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"3c06d11eadebd504","entry":"gelu","repo":"srama2512/NaQ","repo_kind":"official","path":"ReLER/ms_cm/bert_layers.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/ReLER/ms_cm/bert_layers.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c06d11eadebd504"}},{"code_sha256_prefix":"a8ccf61e0ee23809","entry":"load_json","repo":"srama2512/NaQ","repo_kind":"official","path":"ReLER/ms_cm/vslnet_utils.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/ReLER/ms_cm/vslnet_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8ccf61e0ee23809"}},{"code_sha256_prefix":"259b0af842be3b00","entry":"load_pickle","repo":"srama2512/NaQ","repo_kind":"official","path":"ReLER/ms_cm/vslnet_utils.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/ReLER/ms_cm/vslnet_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"259b0af842be3b00"}},{"code_sha256_prefix":"60df4d5a1eee1b92","entry":"mask_logits","repo":"srama2512/NaQ","repo_kind":"official","path":"ReLER/ms_cm/model.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/ReLER/ms_cm/model.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60df4d5a1eee1b92"}},{"code_sha256_prefix":"9312610a229b4ebf","entry":"build_model","repo":"srama2512/NaQ","repo_kind":"official","path":"ReLER/ms_cm/clip_model.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/ReLER/ms_cm/clip_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9312610a229b4ebf"}},{"code_sha256_prefix":"edb929dece68582e","entry":"count_parameters","repo":"srama2512/NaQ","repo_kind":"official","path":"ReLER/utils/model_utils.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/ReLER/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"edb929dece68582e"}},{"code_sha256_prefix":"094900ebac5a22d2","entry":"filter_glove_embedding","repo":"srama2512/NaQ","repo_kind":"official","path":"VSLNet/utils/data_gen.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/VSLNet/utils/data_gen.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"094900ebac5a22d2"}},{"code_sha256_prefix":"e46b6ba1c089129f","entry":"get_em_clips","repo":"srama2512/NaQ","repo_kind":"official","path":"utils/create_naq_dataset.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/utils/create_naq_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e46b6ba1c089129f"}},{"code_sha256_prefix":"623b22c7208a83a3","entry":"load_glove","repo":"srama2512/NaQ","repo_kind":"official","path":"VSLNet/utils/data_gen.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/VSLNet/utils/data_gen.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"623b22c7208a83a3"}},{"code_sha256_prefix":"21e3f8885442c5f3","entry":"load_json","repo":"srama2512/NaQ","repo_kind":"official","path":"utils/create_naq_dataset.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/utils/create_naq_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21e3f8885442c5f3"}},{"code_sha256_prefix":"f319d88d5afe29f2","entry":"load_lines","repo":"srama2512/NaQ","repo_kind":"official","path":"ReLER/ms_cm/vslnet_utils.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/ReLER/ms_cm/vslnet_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f319d88d5afe29f2"}},{"code_sha256_prefix":"32e7ce1faa537e14","entry":"load_narrations_data","repo":"srama2512/NaQ","repo_kind":"official","path":"utils/create_naq_dataset.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/utils/create_naq_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32e7ce1faa537e14"}},{"code_sha256_prefix":"4789ccc84c2e515c","entry":"pad_text_tenor_list","repo":"srama2512/NaQ","repo_kind":"official","path":"ReLER/ms_cm/model.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/ReLER/ms_cm/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4789ccc84c2e515c"}},{"code_sha256_prefix":"76e886ecae69991a","entry":"vocab_emb_gen","repo":"srama2512/NaQ","repo_kind":"official","path":"VSLNet/utils/data_gen.py","file_url":"https://github.com/srama2512/NaQ/blob/HEAD/VSLNet/utils/data_gen.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76e886ecae69991a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}