{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/prompting-large-language-models-with-answer","title":"Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering","arxiv_id":"2303.01903","date":"2023-03-03","proceeding":"CVPR 2023 1","authors":["Zhou Yu","Xuecheng Ouyang","Zhenwei Shao","Meng Wang","Jun Yu"],"abstract":"Knowledge-based visual question answering (VQA) requires external knowledge beyond the image to answer the question. Early studies retrieve required knowledge from explicit knowledge bases (KBs), which often introduces irrelevant information to the question, hence restricting the performance of their models. Recent works have resorted to using a powerful large language model (LLM) as an implicit knowledge engine to acquire the necessary knowledge for answering. Despite the encouraging results achieved by these methods, we argue that they have not fully activated the capacity of the \\emph{blind} LLM as the provided textual input is insufficient to depict the required visual information to answer the question. In this paper, we present Prophet -- a conceptually simple, flexible, and general framework designed to prompt LLM with answer heuristics for knowledge-based VQA. Specifically, we first train a vanilla VQA model on a specific knowledge-based VQA dataset without external knowledge. After that, we extract two types of complementary answer heuristics from the VQA model: answer candidates and answer-aware examples. The two types of answer heuristics are jointly encoded into a formatted prompt to facilitate the LLM's understanding of both the image and question, thus generating a more accurate answer. By incorporating the state-of-the-art LLM GPT-3, Prophet significantly outperforms existing state-of-the-art methods on four challenging knowledge-based VQA datasets. Prophet is general that can be instantiated with the combinations of different VQA models (i.e., both discriminative and generative ones) and different LLMs (i.e., both commercial and open-source ones). Moreover, Prophet can also be integrated with modern large multimodal models in different stages, which is named Prophet++, to further improve the capabilities on knowledge-based VQA tasks.","url_abs":"https://arxiv.org/abs/2303.01903v4","url_pdf":"https://arxiv.org/pdf/2303.01903v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prompting-large-language-models-with-answer","repo_url":"https://github.com/milvlg/prophet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-a-okvqa","task":"Visual Question Answering (VQA)","dataset":"A-OKVQA","model":"Prophet","rank_in_archive_order":3,"of":15,"metrics":{"DA VQA Score":"58.5","MC Accuracy":"75.1"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-ok-vqa","task":"Visual Question Answering (VQA)","dataset":"OK-VQA","model":"Prophet","rank_in_archive_order":5,"of":37,"metrics":{"Accuracy":"62.5"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2303.01903","atlas_url":"https://app.syntology.ai/?focus=2303.01903","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.01903"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/milvlg/prophet","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"47ce2e0ae5037265","entry":"aok_score","repo":"milvlg/prophet","repo_kind":"official","path":"prophet/stage2/utils/data_utils.py","file_url":"https://github.com/milvlg/prophet/blob/HEAD/prophet/stage2/utils/data_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"47ce2e0ae5037265"}},{"code_sha256_prefix":"f8f937b414d7e0a9","entry":"apply_rotary_pos_emb","repo":"milvlg/prophet","repo_kind":"official","path":"prophet/stage1/model/rope2d.py","file_url":"https://github.com/milvlg/prophet/blob/HEAD/prophet/stage1/model/rope2d.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f8f937b414d7e0a9"}},{"code_sha256_prefix":"0ae3d3fa436c0fdc","entry":"flatten","repo":"milvlg/prophet","repo_kind":"official","path":"prophet/stage1/model/net_utils.py","file_url":"https://github.com/milvlg/prophet/blob/HEAD/prophet/stage1/model/net_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ae3d3fa436c0fdc"}},{"code_sha256_prefix":"6aa5b8bdcc4dfde8","entry":"ok_score","repo":"milvlg/prophet","repo_kind":"official","path":"prophet/stage2/utils/data_utils.py","file_url":"https://github.com/milvlg/prophet/blob/HEAD/prophet/stage2/utils/data_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6aa5b8bdcc4dfde8"}},{"code_sha256_prefix":"8ad5d5cfe019a6ae","entry":"rotate_every_two","repo":"milvlg/prophet","repo_kind":"official","path":"prophet/stage1/model/rope2d.py","file_url":"https://github.com/milvlg/prophet/blob/HEAD/prophet/stage1/model/rope2d.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8ad5d5cfe019a6ae"}},{"code_sha256_prefix":"bde236b337b5013f","entry":"unflatten","repo":"milvlg/prophet","repo_kind":"official","path":"prophet/stage1/model/net_utils.py","file_url":"https://github.com/milvlg/prophet/blob/HEAD/prophet/stage1/model/net_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bde236b337b5013f"}},{"code_sha256_prefix":"4120d55fb48dd6b0","entry":"get_optim","repo":"milvlg/prophet","repo_kind":"official","path":"prophet/stage1/utils/optim.py","file_url":"https://github.com/milvlg/prophet/blob/HEAD/prophet/stage1/utils/optim.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4120d55fb48dd6b0"}},{"code_sha256_prefix":"7b57f7d22b08cde1","entry":"get_optim_for_finetune","repo":"milvlg/prophet","repo_kind":"official","path":"prophet/stage1/utils/optim.py","file_url":"https://github.com/milvlg/prophet/blob/HEAD/prophet/stage1/utils/optim.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b57f7d22b08cde1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}