{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dense-caption-matching-and-frame-selection","title":"Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA","arxiv_id":"2005.06409","date":"2020-05-13","proceeding":"ACL 2020 6","authors":["Hyounghun Kim","Zineng Tang","Mohit Bansal"],"abstract":"Videos convey rich information. Dynamic spatio-temporal relationships between people/objects, and diverse multimodal events are present in a video clip. Hence, it is important to develop automated models that can accurately extract such information from videos. Answering questions on videos is one of the tasks which can evaluate such AI abilities. In this paper, we propose a video question answering model which effectively integrates multi-modal input sources and finds the temporally relevant information to answer questions. Specifically, we first employ dense image captions to help identify objects and their detailed salient regions and actions, and hence give the model useful extra information (in explicit textual format to allow easier matching) for answering questions. Moreover, our model is also comprised of dual-level attention (word/object and frame level), multi-head self/cross-integration for different sources (video and dense captions), and gates which pass more relevant information to the classifier. Finally, we also cast the frame selection problem as a multi-label classification task and introduce two loss functions, In-andOut Frame Score Margin (IOFSM) and Balanced Binary Cross-Entropy (BBCE), to better supervise the model with human importance annotations. We evaluate our model on the challenging TVQA dataset, where each of our model components provides significant gains, and our overall model outperforms the state-of-the-art by a large margin (74.09% versus 70.52%). We also present several word, object, and frame level visualization studies. Our code is publicly available at: https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020","url_abs":"https://arxiv.org/abs/2005.06409v1","url_pdf":"https://arxiv.org/pdf/2005.06409v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dense-caption-matching-and-frame-selection","repo_url":"https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"multi-label-classification-2","task_name":"MUlTI-LABEL-ClASSIFICATION"},{"task_slug":"multi-label-classification","task_name":"Multi-Label Classification"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"temporal-localization","task_name":"Temporal Localization"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2005.06409","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2005.06409"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020","reach":null}],"summary":{"ran":6,"ran_draft_wrong":1,"ran_honours":2,"unverified":3},"by_repo_kind":{"official":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"eabc5a84173598c5","entry":"DepthwiseSeparableConv","repo":"hyounghk/VideoQADenseCapFrameGate-ACL2020","repo_kind":"official","path":"qanet/tvqanet.py","file_url":"https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020/blob/HEAD/qanet/tvqanet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eabc5a84173598c5"}},{"code_sha256_prefix":"92c3decd59ca89ec","entry":"LinearWrapper","repo":"hyounghk/VideoQADenseCapFrameGate-ACL2020","repo_kind":"official","path":"qanet/tvqanet.py","file_url":"https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020/blob/HEAD/qanet/tvqanet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92c3decd59ca89ec"}},{"code_sha256_prefix":"27eddc30c65e3b95","entry":"MultiHeadedAttention","repo":"hyounghk/VideoQADenseCapFrameGate-ACL2020","repo_kind":"official","path":"qanet/tvqanet.py","file_url":"https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020/blob/HEAD/qanet/tvqanet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27eddc30c65e3b95"}},{"code_sha256_prefix":"b9683b87afb37df5","entry":"PositionEncoding","repo":"hyounghk/VideoQADenseCapFrameGate-ACL2020","repo_kind":"official","path":"qanet/tvqanet.py","file_url":"https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020/blob/HEAD/qanet/tvqanet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b9683b87afb37df5"}},{"code_sha256_prefix":"d606630a26a361c2","entry":"StructuredAttention_bi","repo":"hyounghk/VideoQADenseCapFrameGate-ACL2020","repo_kind":"official","path":"qanet/tvqanet.py","file_url":"https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020/blob/HEAD/qanet/tvqanet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d606630a26a361c2"}},{"code_sha256_prefix":"ca5ccdc1e66312e7","entry":"StructuredAttention_frame","repo":"hyounghk/VideoQADenseCapFrameGate-ACL2020","repo_kind":"official","path":"qanet/tvqanet.py","file_url":"https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020/blob/HEAD/qanet/tvqanet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca5ccdc1e66312e7"}},{"code_sha256_prefix":"792073de4c443fa8","entry":"clones","repo":"hyounghk/VideoQADenseCapFrameGate-ACL2020","repo_kind":"official","path":"qanet/tvqanet.py","file_url":"https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020/blob/HEAD/qanet/tvqanet.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"792073de4c443fa8"}},{"code_sha256_prefix":"1c17153fb5488283","entry":"mask_logits","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"1c17153fb5488283"}},{"code_sha256_prefix":"ef908c1d80306f2d","entry":"mask_logits_sum","repo":"hyounghk/VideoQADenseCapFrameGate-ACL2020","repo_kind":"official","path":"qanet/tvqanet.py","file_url":"https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020/blob/HEAD/qanet/tvqanet.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef908c1d80306f2d"}},{"code_sha256_prefix":"141e2be2575e53f1","entry":"EncoderBlock","repo":"hyounghk/VideoQADenseCapFrameGate-ACL2020","repo_kind":"official","path":"qanet/tvqanet.py","file_url":"https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020/blob/HEAD/qanet/tvqanet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"141e2be2575e53f1"}},{"code_sha256_prefix":"7bbe54b9dd2d671a","entry":"StackedEncoder","repo":"hyounghk/VideoQADenseCapFrameGate-ACL2020","repo_kind":"official","path":"qanet/tvqanet.py","file_url":"https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020/blob/HEAD/qanet/tvqanet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7bbe54b9dd2d671a"}},{"code_sha256_prefix":"5a91cb89645d7259","entry":"TVQANet","repo":"hyounghk/VideoQADenseCapFrameGate-ACL2020","repo_kind":"official","path":"qanet/tvqanet.py","file_url":"https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020/blob/HEAD/qanet/tvqanet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a91cb89645d7259"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}