{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/glance-and-focus-memory-prompting-for-multi-1","title":"Glance and Focus: Memory Prompting for Multi-Event Video Question Answering","arxiv_id":"2401.01529","date":"2024-01-03","proceeding":"NeurIPS 2023 11","authors":["Ziyi Bai","Ruiping Wang","Xilin Chen"],"abstract":"Video Question Answering (VideoQA) has emerged as a vital tool to evaluate agents' ability to understand human daily behaviors. Despite the recent success of large vision language models in many multi-modal tasks, complex situation reasoning over videos involving multiple human-object interaction events still remains challenging. In contrast, humans can easily tackle it by using a series of episode memories as anchors to quickly locate question-related key moments for reasoning. To mimic this effective reasoning strategy, we propose the Glance-Focus model. One simple way is to apply an action detection model to predict a set of actions as key memories. However, these actions within a closed set vocabulary are hard to generalize to various video domains. Instead of that, we train an Encoder-Decoder to generate a set of dynamic event memories at the glancing stage. Apart from using supervised bipartite matching to obtain the event memories, we further design an unsupervised memory generation method to get rid of dependence on event annotations. Next, at the focusing stage, these event memories act as a bridge to establish the correlation between the questions with high-level event concepts and low-level lengthy video content. Given the question, the model first focuses on the generated key event memory, then focuses on the most relevant moment for reasoning through our designed multi-level cross-attention mechanism. We conduct extensive experiments on four Multi-Event VideoQA benchmarks including STAR, EgoTaskQA, AGQA, and NExT-QA. Our proposed model achieves state-of-the-art results, surpassing current large models in various challenging reasoning tasks. The code and models are available at https://github.com/ByZ0e/Glance-Focus.","url_abs":"https://arxiv.org/abs/2401.01529v1","url_pdf":"https://arxiv.org/pdf/2401.01529v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"glance-and-focus-memory-prompting-for-multi-1","repo_url":"https://github.com/byz0e/glance-focus","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"action-detection","task_name":"Action Detection"},{"task_slug":"human-object-interaction-detection","task_name":"Human-Object Interaction Detection"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/question-answering-on-egotaskqa","task":"Question Answering","dataset":"EgoTaskQA","model":"GF(sup)","rank_in_archive_order":2,"of":4,"metrics":{"Direct":"44.27"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-egotaskqa","task":"Question Answering","dataset":"EgoTaskQA","model":"GF(uns)","rank_in_archive_order":3,"of":4,"metrics":{"Direct":"43.06"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-agqa-2-0-balanced","task":"Video Question Answering","dataset":"AGQA 2.0 balanced","model":"GF (sup) - Faster RCNN","rank_in_archive_order":1,"of":8,"metrics":{"Average Accuracy":"55.08"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-agqa-2-0-balanced","task":"Video Question Answering","dataset":"AGQA 2.0 balanced","model":"GF (uns) - S3D","rank_in_archive_order":3,"of":8,"metrics":{"Average Accuracy":"53.33"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-agqa-2-0-balanced","task":"Video Question Answering","dataset":"AGQA 2.0 balanced","model":"AIO - ViT","rank_in_archive_order":7,"of":8,"metrics":{"Average Accuracy":"48.59"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-next-qa","task":"Video Question Answering","dataset":"NExT-QA","model":"GF","rank_in_archive_order":38,"of":47,"metrics":{"Accuracy":"58.83"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-situated","task":"Video Question Answering","dataset":"STAR Benchmark","model":"GF(sup)","rank_in_archive_order":5,"of":17,"metrics":{"Average Accuracy":"53.94"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-situated","task":"Video Question Answering","dataset":"STAR Benchmark","model":"GF(uns)","rank_in_archive_order":6,"of":17,"metrics":{"Average Accuracy":"53.86"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.01529","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.01529"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ByZ0e/Glance-Focus","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/byz0e/glance-focus","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"ran_violates":1,"unverified":9},"by_repo_kind":{"official":{"samples":18,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d725dcf4cf20da7f","entry":"FCNet","repo":"byz0e/glance-focus","repo_kind":"official","path":"model/glance_focus.py","file_url":"https://github.com/byz0e/glance-focus/blob/HEAD/model/glance_focus.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d725dcf4cf20da7f"}},{"code_sha256_prefix":"440bd21060030be0","entry":"GF","repo":"byz0e/glance-focus","repo_kind":"official","path":"model/glance_focus.py","file_url":"https://github.com/byz0e/glance-focus/blob/HEAD/model/glance_focus.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"440bd21060030be0"}},{"code_sha256_prefix":"c61a34a3c9e0df91","entry":"chunk_list","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"dataset/agqa.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/dataset/agqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c61a34a3c9e0df91"}},{"code_sha256_prefix":"6941a4a46f03fa07","entry":"collate_func","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"dataset/egotaskqa.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/dataset/egotaskqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6941a4a46f03fa07"}},{"code_sha256_prefix":"cef10190a6f8f7e9","entry":"flat_list_of_lists","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"dataset/nextqa.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/dataset/nextqa.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cef10190a6f8f7e9"}},{"code_sha256_prefix":"b7639c2e8b83837b","entry":"span_cxw_to_xx","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"utils/span_utils.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/utils/span_utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b7639c2e8b83837b"}},{"code_sha256_prefix":"93d1fd5afccb1ea1","entry":"span_xx_to_cxw","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"dataset/egotaskqa.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/dataset/egotaskqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93d1fd5afccb1ea1"}},{"code_sha256_prefix":"c31fe80a1b2ba813","entry":"tokenize","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"dataset/nextqa.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/dataset/nextqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c31fe80a1b2ba813"}},{"code_sha256_prefix":"bab1acae729c21cf","entry":"videoqa_collate_fn","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"dataset/nextqa.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/dataset/nextqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bab1acae729c21cf"}},{"code_sha256_prefix":"e66b5e8b24dfea32","entry":"build_transformer","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"model/transformer_gf.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/model/transformer_gf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e66b5e8b24dfea32"}},{"code_sha256_prefix":"1eee3af921bc5d88","entry":"load_imgs","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"utils/basic_utils.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/utils/basic_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1eee3af921bc5d88"}},{"code_sha256_prefix":"bace1acf5a748097","entry":"load_json","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"utils/basic_utils.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/utils/basic_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bace1acf5a748097"}},{"code_sha256_prefix":"7de9d66696a307eb","entry":"load_pickle","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"utils/basic_utils.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/utils/basic_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7de9d66696a307eb"}},{"code_sha256_prefix":"6381e6df15302a0e","entry":"mk_input_group","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"dataset/agqa.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/dataset/agqa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6381e6df15302a0e"}},{"code_sha256_prefix":"bffb542b7e13c786","entry":"repeat_tensor_rows","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"dataset/agqa.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/dataset/agqa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bffb542b7e13c786"}},{"code_sha256_prefix":"0b0c5cf51bc1e61f","entry":"sorter","repo":"byz0e/glance-focus","repo_kind":"official","path":"train_glance_focus_sup.py","file_url":"https://github.com/byz0e/glance-focus/blob/HEAD/train_glance_focus_sup.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b0c5cf51bc1e61f"}},{"code_sha256_prefix":"7114f025044af9b4","entry":"span_xx_to_cxw","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"utils/span_utils.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/utils/span_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7114f025044af9b4"}},{"code_sha256_prefix":"e4ae5582a2533cdb","entry":"temporal_iou","repo":"ByZ0e/Glance-Focus","repo_kind":"official","path":"utils/span_utils.py","file_url":"https://github.com/ByZ0e/Glance-Focus/blob/HEAD/utils/span_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4ae5582a2533cdb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}