{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mukea-multimodal-knowledge-extraction-and","title":"MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering","arxiv_id":"2203.09138","date":"2022-03-17","proceeding":"CVPR 2022 1","authors":["Yang Ding","Jing Yu","Bang Liu","Yue Hu","Mingxin Cui","Qi Wu"],"abstract":"Knowledge-based visual question answering requires the ability of associating external knowledge for open-ended cross-modal scene understanding. One limitation of existing solutions is that they capture relevant knowledge from text-only knowledge bases, which merely contain facts expressed by first-order predicates or language descriptions while lacking complex but indispensable multimodal knowledge for visual understanding. How to construct vision-relevant and explainable multimodal knowledge for the VQA scenario has been less studied. In this paper, we propose MuKEA to represent multimodal knowledge by an explicit triplet to correlate visual objects and fact answers with implicit relations. To bridge the heterogeneous gap, we propose three objective losses to learn the triplet representations from complementary views: embedding structure, topological relation and semantic space. By adopting a pre-training and fine-tuning learning strategy, both basic and domain-specific multimodal knowledge are progressively accumulated for answer prediction. We outperform the state-of-the-art by 3.35% and 6.08% respectively on two challenging knowledge-required datasets: OK-VQA and KRVQA. Experimental results prove the complementary benefits of the multimodal knowledge with existing knowledge bases and the advantages of our end-to-end framework over the existing pipeline methods. The code is available at https://github.com/AndersonStra/MuKEA.","url_abs":"https://arxiv.org/abs/2203.09138v1","url_pdf":"https://arxiv.org/pdf/2203.09138v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mukea-multimodal-knowledge-extraction-and","repo_url":"https://github.com/andersonstra/mukea","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"implicit-relations","task_name":"Implicit Relations"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":null,"task_name":"Triplet"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.09138","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.09138"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/andersonstra/mukea","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/AndersonStra/MuKEA","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":1,"unverified":8},"by_repo_kind":{"official":{"samples":10,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0d41bbe52278fde1","entry":"gumbel_softmax_sample","repo":"AndersonStra/MuKEA","repo_kind":"official","path":"okvqa/gumbel_softmax.py","file_url":"https://github.com/AndersonStra/MuKEA/blob/HEAD/okvqa/gumbel_softmax.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d41bbe52278fde1"}},{"code_sha256_prefix":"7c913c414af7196f","entry":"sample_gumbel","repo":"AndersonStra/MuKEA","repo_kind":"official","path":"okvqa/gumbel_softmax.py","file_url":"https://github.com/AndersonStra/MuKEA/blob/HEAD/okvqa/gumbel_softmax.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c913c414af7196f"}},{"code_sha256_prefix":"d342fd2a97947b8f","entry":"distanceL2","repo":"AndersonStra/MuKEA","repo_kind":"official","path":"contrastive_loss.py","file_url":"https://github.com/AndersonStra/MuKEA/blob/HEAD/contrastive_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d342fd2a97947b8f"}},{"code_sha256_prefix":"6a49bab714ec6122","entry":"dot_sim","repo":"AndersonStra/MuKEA","repo_kind":"official","path":"contrastive_loss.py","file_url":"https://github.com/AndersonStra/MuKEA/blob/HEAD/contrastive_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a49bab714ec6122"}},{"code_sha256_prefix":"8181c4e9f8e6986a","entry":"gumbel_softmax","repo":"AndersonStra/MuKEA","repo_kind":"official","path":"okvqa/gumbel_softmax.py","file_url":"https://github.com/AndersonStra/MuKEA/blob/HEAD/okvqa/gumbel_softmax.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8181c4e9f8e6986a"}},{"code_sha256_prefix":"4f72de8caeab7fb0","entry":"l2_sim","repo":"AndersonStra/MuKEA","repo_kind":"official","path":"contrastive_loss.py","file_url":"https://github.com/AndersonStra/MuKEA/blob/HEAD/contrastive_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f72de8caeab7fb0"}},{"code_sha256_prefix":"6928e9031c786532","entry":"my_collate","repo":"AndersonStra/MuKEA","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/AndersonStra/MuKEA/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6928e9031c786532"}},{"code_sha256_prefix":"aa11211deb269101","entry":"my_collate_2","repo":"AndersonStra/MuKEA","repo_kind":"official","path":"inference.py","file_url":"https://github.com/AndersonStra/MuKEA/blob/HEAD/inference.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa11211deb269101"}},{"code_sha256_prefix":"fbc1ba4579bafa09","entry":"my_collate_pretrain","repo":"AndersonStra/MuKEA","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/AndersonStra/MuKEA/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbc1ba4579bafa09"}},{"code_sha256_prefix":"2eb06695bbb6d9cb","entry":"plural","repo":"AndersonStra/MuKEA","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/AndersonStra/MuKEA/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2eb06695bbb6d9cb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}