{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gerea-question-aware-prompt-captions-for","title":"GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering","arxiv_id":"2402.02503","date":"2024-02-04","proceeding":null,"authors":["Ziyu Ma","Shutao Li","Bin Sun","Jianfei Cai","Zuxiang Long","Fuyan Ma"],"abstract":"Knowledge-based visual question answering (VQA) requires world knowledge beyond the image for accurate answer. Recently, instead of extra knowledge bases, a large language model (LLM) like GPT-3 is activated as an implicit knowledge engine to jointly acquire and reason the necessary knowledge for answering by converting images into textual information (e.g., captions and answer candidates). However, such conversion may introduce irrelevant information, which causes the LLM to misinterpret images and ignore visual details crucial for accurate knowledge. We argue that multimodal large language model (MLLM) is a better implicit knowledge engine than the LLM for its superior capability of visual understanding. Despite this, how to activate the capacity of MLLM as the implicit knowledge engine has not been explored yet. Therefore, we propose GeReA, a generate-reason framework that prompts a MLLM like InstructBLIP with question relevant vision and language information to generate knowledge-relevant descriptions and reasons those descriptions for knowledge-based VQA. Specifically, the question-relevant image regions and question-specific manual prompts are encoded in the MLLM to generate the knowledge relevant descriptions, referred to as question-aware prompt captions. After that, the question-aware prompt captions, image-question pair, and similar samples are sent into the multi-modal reasoning model to learn a joint knowledge-image-question representation for answer prediction. GeReA unlocks the use of MLLM as the implicit knowledge engine, surpassing all previous state-of-the-art methods on OK-VQA and A-OKVQA datasets, with test accuracies of 66.5% and 63.3% respectively. Our code will be released at https://github.com/Upper9527/GeReA.","url_abs":"https://arxiv.org/abs/2402.02503v1","url_pdf":"https://arxiv.org/pdf/2402.02503v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gerea-question-aware-prompt-captions-for","repo_url":"https://github.com/upper9527/gerea","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"multimodal-large-language-model","task_name":"Multimodal Large Language Model"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.02503","atlas_url":"https://app.syntology.ai/?focus=2402.02503","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.02503"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/upper9527/gerea","reach":{"status":"ok"}}],"summary":{"ran":11,"unverified":7},"by_repo_kind":{"official":{"samples":18,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"e3ead4161d5e3d4a","entry":"calc_banned_bad_words_ids","repo":"upper9527/gerea","repo_kind":"official","path":"src/generation_utils.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/generation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3ead4161d5e3d4a"}},{"code_sha256_prefix":"8d77347cfa15b834","entry":"calc_banned_ngram_tokens","repo":"upper9527/gerea","repo_kind":"official","path":"src/generation_utils.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/generation_utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d77347cfa15b834"}},{"code_sha256_prefix":"637b51b8df78832a","entry":"check_answer","repo":"upper9527/gerea","repo_kind":"official","path":"src/evaluation.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/evaluation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"637b51b8df78832a"}},{"code_sha256_prefix":"bc0dae797248182b","entry":"encode_passages","repo":"upper9527/gerea","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bc0dae797248182b"}},{"code_sha256_prefix":"e3a0e6ac26d46ae2","entry":"get_checkpoint_path","repo":"upper9527/gerea","repo_kind":"official","path":"src/util.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3a0e6ac26d46ae2"}},{"code_sha256_prefix":"fa4ecbff4b9a354b","entry":"has_answer","repo":"upper9527/gerea","repo_kind":"official","path":"src/evaluation.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/evaluation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa4ecbff4b9a354b"}},{"code_sha256_prefix":"1700ba94a8342336","entry":"init_logger","repo":"upper9527/gerea","repo_kind":"official","path":"src/util.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1700ba94a8342336"}},{"code_sha256_prefix":"3762227466b20f21","entry":"okvqa_ems","repo":"upper9527/gerea","repo_kind":"official","path":"leaderboard_evaluation.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/leaderboard_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3762227466b20f21"}},{"code_sha256_prefix":"e2f8b9900fd662b1","entry":"select_examples_NQ","repo":"upper9527/gerea","repo_kind":"official","path":"src/preprocess.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/preprocess.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2f8b9900fd662b1"}},{"code_sha256_prefix":"e544cd4b39e5ec61","entry":"select_examples_TQA","repo":"upper9527/gerea","repo_kind":"official","path":"src/preprocess.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/preprocess.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e544cd4b39e5ec61"}},{"code_sha256_prefix":"0dc219ab6716a127","entry":"top_k_top_p_filtering","repo":"upper9527/gerea","repo_kind":"official","path":"src/generation_utils.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/generation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0dc219ab6716a127"}},{"code_sha256_prefix":"7a4988952dd3a127","entry":"calculate_matches","repo":"upper9527/gerea","repo_kind":"official","path":"src/evaluation.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a4988952dd3a127"}},{"code_sha256_prefix":"108d116eb5bdf952","entry":"exact_match_score","repo":"upper9527/gerea","repo_kind":"official","path":"leaderboard_evaluation.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/leaderboard_evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"108d116eb5bdf952"}},{"code_sha256_prefix":"4c076a246978c67e","entry":"get_options","repo":"upper9527/gerea","repo_kind":"official","path":"src/options.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/options.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c076a246978c67e"}},{"code_sha256_prefix":"a16b92bf7d32bad5","entry":"load","repo":"upper9527/gerea","repo_kind":"official","path":"src/util.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a16b92bf7d32bad5"}},{"code_sha256_prefix":"67eac65279649d0a","entry":"load_data","repo":"upper9527/gerea","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"67eac65279649d0a"}},{"code_sha256_prefix":"52fdf83141474e49","entry":"load_tf_weights_in_t5","repo":"upper9527/gerea","repo_kind":"official","path":"src/modeling_t5.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/src/modeling_t5.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"52fdf83141474e49"}},{"code_sha256_prefix":"b92b07bb3574992a","entry":"normalize_answer","repo":"upper9527/gerea","repo_kind":"official","path":"leaderboard_evaluation.py","file_url":"https://github.com/upper9527/gerea/blob/HEAD/leaderboard_evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b92b07bb3574992a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}