{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/open-vocabulary-video-question-answering-a","title":"Open-vocabulary Video Question Answering: A New Benchmark for Evaluating the Generalizability of Video Question Answering Models","arxiv_id":"2308.09363","date":"2023-08-18","proceeding":"ICCV 2023 1","authors":["Dohwan Ko","Ji Soo Lee","Miso Choi","Jaewon Chu","Jihwan Park","Hyunwoo J. Kim"],"abstract":"Video Question Answering (VideoQA) is a challenging task that entails complex multi-modal reasoning. In contrast to multiple-choice VideoQA which aims to predict the answer given several options, the goal of open-ended VideoQA is to answer questions without restricting candidate answers. However, the majority of previous VideoQA models formulate open-ended VideoQA as a classification task to classify the video-question pairs into a fixed answer set, i.e., closed-vocabulary, which contains only frequent answers (e.g., top-1000 answers). This leads the model to be biased toward only frequent answers and fail to generalize on out-of-vocabulary answers. We hence propose a new benchmark, Open-vocabulary Video Question Answering (OVQA), to measure the generalizability of VideoQA models by considering rare and unseen answers. In addition, in order to improve the model's generalization power, we introduce a novel GNN-based soft verbalizer that enhances the prediction on rare and unseen answers by aggregating the information from their similar words. For evaluation, we introduce new baselines by modifying the existing (closed-vocabulary) open-ended VideoQA models and improve their performances by further taking into account rare and unseen answers. Our ablation studies and qualitative analyses demonstrate that our GNN-based soft verbalizer further improves the model performance, especially on rare and unseen answers. We hope that our benchmark OVQA can serve as a guide for evaluating the generalizability of VideoQA models and inspire future research. Code is available at https://github.com/mlvlab/OVQA.","url_abs":"https://arxiv.org/abs/2308.09363v1","url_pdf":"https://arxiv.org/pdf/2308.09363v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"open-vocabulary-video-question-answering-a","repo_url":"https://github.com/mlvlab/ovqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"tgif-frame","task_name":"TGIF-Frame"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":null,"task_name":"Visu"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"fail","method_name":"fail"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-question-answering-on-activitynet-qa","task":"Video Question Answering","dataset":"ActivityNet-QA","model":"FrozenBiLM+","rank_in_archive_order":18,"of":36,"metrics":{"Accuracy":"44.8"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-activitynet-qa","task":"Video Question Answering","dataset":"ActivityNet-QA","model":"All-in-one+","rank_in_archive_order":24,"of":36,"metrics":{"Accuracy":"40.0"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-activitynet-qa","task":"Video Question Answering","dataset":"ActivityNet-QA","model":"VIOLET+","rank_in_archive_order":25,"of":36,"metrics":{"Accuracy":"39.7"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-msrvtt-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSRVTT-QA","model":"FrozenBiLM+","rank_in_archive_order":8,"of":34,"metrics":{"Accuracy":"0.470"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-msrvtt-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSRVTT-QA","model":"JustAsk+","rank_in_archive_order":25,"of":34,"metrics":{"Accuracy":"0.418"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-msrvtt-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSRVTT-QA","model":"All-in-one+","rank_in_archive_order":26,"of":34,"metrics":{"Accuracy":"0.395"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-msvd-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSVD-QA","model":"FrozenBiLM+","rank_in_archive_order":10,"of":36,"metrics":{"Accuracy":"0.558"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-msvd-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSVD-QA","model":"VIOLET+","rank_in_archive_order":22,"of":36,"metrics":{"Accuracy":"0.495"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-msvd-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSVD-QA","model":"JustAsk+","rank_in_archive_order":26,"of":36,"metrics":{"Accuracy":"0.477"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-msvd-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSVD-QA","model":"All-in-one+","rank_in_archive_order":30,"of":36,"metrics":{"Accuracy":"0.438"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.09363","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.09363"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/mlvlab/OVQA","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":5},"by_repo_kind":{"official":{"samples":8,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"fa6132bc58853abe","entry":"get_mask","repo":"mlvlab/OVQA","repo_kind":"official","path":"util/misc.py","file_url":"https://github.com/mlvlab/OVQA/blob/HEAD/util/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa6132bc58853abe"}},{"code_sha256_prefix":"e42c9680fb0f8054","entry":"mask_tokens","repo":"mlvlab/OVQA","repo_kind":"official","path":"util/misc.py","file_url":"https://github.com/mlvlab/OVQA/blob/HEAD/util/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e42c9680fb0f8054"}},{"code_sha256_prefix":"95c71826ab5dd916","entry":"videoqa_collate_fn","repo":"mlvlab/OVQA","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/mlvlab/OVQA/blob/HEAD/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"95c71826ab5dd916"}},{"code_sha256_prefix":"2b95e75e05ea5abf","entry":"all_gather","repo":"mlvlab/OVQA","repo_kind":"official","path":"util/dist.py","file_url":"https://github.com/mlvlab/OVQA/blob/HEAD/util/dist.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b95e75e05ea5abf"}},{"code_sha256_prefix":"b1852809457a4b1f","entry":"build_relative_position","repo":"mlvlab/OVQA","repo_kind":"official","path":"model/deberta.py","file_url":"https://github.com/mlvlab/OVQA/blob/HEAD/model/deberta.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1852809457a4b1f"}},{"code_sha256_prefix":"05732f1a5126672d","entry":"get_mask","repo":"mlvlab/OVQA","repo_kind":"official","path":"model/deberta.py","file_url":"https://github.com/mlvlab/OVQA/blob/HEAD/model/deberta.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"05732f1a5126672d"}},{"code_sha256_prefix":"29371b9c270911e1","entry":"make_log_bucket_position","repo":"mlvlab/OVQA","repo_kind":"official","path":"model/deberta.py","file_url":"https://github.com/mlvlab/OVQA/blob/HEAD/model/deberta.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"29371b9c270911e1"}},{"code_sha256_prefix":"a38ed1aa8d59134e","entry":"reduce_dict","repo":"mlvlab/OVQA","repo_kind":"official","path":"util/dist.py","file_url":"https://github.com/mlvlab/OVQA/blob/HEAD/util/dist.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a38ed1aa8d59134e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}