{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/knowledge-graphs-meet-multi-modal-learning-a","title":"Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey","arxiv_id":"2402.05391","date":"2024-02-08","proceeding":null,"authors":["Zhuo Chen","Yichi Zhang","Yin Fang","Yuxia Geng","Lingbing Guo","Xiang Chen","Qian Li","Wen Zhang","Jiaoyan Chen","Yushan Zhu","Jiaqi Li","Xiaoze Liu","Jeff Z. Pan","Ningyu Zhang","Huajun Chen"],"abstract":"Knowledge Graphs (KGs) play a pivotal role in advancing various AI applications, with the semantic web community's exploration into multi-modal dimensions unlocking new avenues for innovation. In this survey, we carefully review over 300 articles, focusing on KG-aware research in two principal aspects: KG-driven Multi-Modal (KG4MM) learning, where KGs support multi-modal tasks, and Multi-Modal Knowledge Graph (MM4KG), which extends KG studies into the MMKG realm. We begin by defining KGs and MMKGs, then explore their construction progress. Our review includes two primary task categories: KG-aware multi-modal learning tasks, such as Image Classification and Visual Question Answering, and intrinsic MMKG tasks like Multi-modal Knowledge Graph Completion and Entity Alignment, highlighting specific research trajectories. For most of these tasks, we provide definitions, evaluation benchmarks, and additionally outline essential insights for conducting relevant research. Finally, we discuss current challenges and identify emerging trends, such as progress in Large Language Modeling and Multi-modal Pre-training strategies. This survey aims to serve as a comprehensive reference for researchers already involved in or considering delving into KG and multi-modal learning research, offering insights into the evolving landscape of MMKG research and supporting future work.","url_abs":"https://arxiv.org/abs/2402.05391v4","url_pdf":"https://arxiv.org/pdf/2402.05391v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"knowledge-graphs-meet-multi-modal-learning-a","repo_url":"https://github.com/zjukg/kg-mm-survey","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"knowledge-graphs-meet-multi-modal-learning-a","repo_url":"https://github.com/China-UK-ZSL/ZS-F-VQA","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"knowledge-graphs-meet-multi-modal-learning-a","repo_url":"https://github.com/hackerchenzhuo/LaKo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"knowledge-graphs-meet-multi-modal-learning-a","repo_url":"https://github.com/zjukg/maco","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"knowledge-graphs-meet-multi-modal-learning-a","repo_url":"https://github.com/zjukg/mygo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"knowledge-graphs-meet-multi-modal-learning-a","repo_url":"https://github.com/zjukg/snag","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"articles","task_name":"Articles"},{"task_slug":"entity-alignment","task_name":"Entity Alignment"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"knowledge-graph-completion","task_name":"Knowledge Graph Completion"},{"task_slug":"knowledge-graphs","task_name":"Knowledge Graphs"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"multi-modal-knowledge-graph","task_name":"Multi-modal Knowledge Graph"},{"task_slug":"survey","task_name":"Survey"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.05391","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.05391"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zjukg/kg-mm-survey","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zjukg/snag","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/China-UK-ZSL/ZS-F-VQA","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hackerchenzhuo/LaKo","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zjukg/mygo","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zjukg/maco","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_draft_wrong":1,"ran_fixture":1,"unverified":9},"by_repo_kind":{"listed":{"samples":18,"ran":9,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ac1c2010e4fa00c6","entry":"add_embeddings","repo":"hackerchenzhuo/LaKo","repo_kind":"listed","path":"fact_retrieval.py","file_url":"https://github.com/hackerchenzhuo/LaKo/blob/HEAD/fact_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac1c2010e4fa00c6"}},{"code_sha256_prefix":"745b7eb38c9d1b29","entry":"apply_attention","repo":"China-UK-ZSL/ZS-F-VQA","repo_kind":"listed","path":"code/model/attention.py","file_url":"https://github.com/China-UK-ZSL/ZS-F-VQA/blob/HEAD/code/model/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"745b7eb38c9d1b29"}},{"code_sha256_prefix":"9b0a787b92a87023","entry":"cosine_sim","repo":"zjukg/snag","repo_kind":"listed","path":"SNAG_MMEA/model/EVA_tools.py","file_url":"https://github.com/zjukg/snag/blob/HEAD/SNAG_MMEA/model/EVA_tools.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b0a787b92a87023"}},{"code_sha256_prefix":"6bef6bdd6eb44070","entry":"cross_attention_forward","repo":"hackerchenzhuo/LaKo","repo_kind":"listed","path":"src/model.py","file_url":"https://github.com/hackerchenzhuo/LaKo/blob/HEAD/src/model.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6bef6bdd6eb44070"}},{"code_sha256_prefix":"bc0dae797248182b","entry":"encode_passages","repo":"hackerchenzhuo/LaKo","repo_kind":"listed","path":"src/data.py","file_url":"https://github.com/hackerchenzhuo/LaKo/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc0dae797248182b"}},{"code_sha256_prefix":"0ec5a25ad1d89cc8","entry":"load_data","repo":"hackerchenzhuo/LaKo","repo_kind":"listed","path":"src/data.py","file_url":"https://github.com/hackerchenzhuo/LaKo/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec5a25ad1d89cc8"}},{"code_sha256_prefix":"e2f8b9900fd662b1","entry":"select_examples_NQ","repo":"hackerchenzhuo/LaKo","repo_kind":"listed","path":"src/preprocess.py","file_url":"https://github.com/hackerchenzhuo/LaKo/blob/HEAD/src/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2f8b9900fd662b1"}},{"code_sha256_prefix":"e544cd4b39e5ec61","entry":"select_examples_TQA","repo":"hackerchenzhuo/LaKo","repo_kind":"listed","path":"src/preprocess.py","file_url":"https://github.com/hackerchenzhuo/LaKo/blob/HEAD/src/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e544cd4b39e5ec61"}},{"code_sha256_prefix":"b70265fc26e335dc","entry":"tile_2d_over_nd","repo":"China-UK-ZSL/ZS-F-VQA","repo_kind":"listed","path":"code/model/attention.py","file_url":"https://github.com/China-UK-ZSL/ZS-F-VQA/blob/HEAD/code/model/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b70265fc26e335dc"}},{"code_sha256_prefix":"e7d0985e55efb9e7","entry":"attention","repo":"zjukg/snag","repo_kind":"listed","path":"SNAG_MMEA/model/EVA_tools.py","file_url":"https://github.com/zjukg/snag/blob/HEAD/SNAG_MMEA/model/EVA_tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7d0985e55efb9e7"}},{"code_sha256_prefix":"7a4988952dd3a127","entry":"calculate_matches","repo":"hackerchenzhuo/LaKo","repo_kind":"listed","path":"src/evaluation.py","file_url":"https://github.com/hackerchenzhuo/LaKo/blob/HEAD/src/evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a4988952dd3a127"}},{"code_sha256_prefix":"637b51b8df78832a","entry":"check_answer","repo":"hackerchenzhuo/LaKo","repo_kind":"listed","path":"src/evaluation.py","file_url":"https://github.com/hackerchenzhuo/LaKo/blob/HEAD/src/evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"637b51b8df78832a"}},{"code_sha256_prefix":"b107f7a819444b4f","entry":"generate_neg_triples_fast","repo":"zjukg/snag","repo_kind":"listed","path":"SNAG_MMEA/model/MSNEA_tools.py","file_url":"https://github.com/zjukg/snag/blob/HEAD/SNAG_MMEA/model/MSNEA_tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b107f7a819444b4f"}},{"code_sha256_prefix":"681342822f804578","entry":"generate_pos_triples","repo":"zjukg/snag","repo_kind":"listed","path":"SNAG_MMEA/model/MSNEA_tools.py","file_url":"https://github.com/zjukg/snag/blob/HEAD/SNAG_MMEA/model/MSNEA_tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"681342822f804578"}},{"code_sha256_prefix":"bda2af2880f66879","entry":"generate_relation_triple_batch","repo":"zjukg/snag","repo_kind":"listed","path":"SNAG_MMEA/model/MSNEA_tools.py","file_url":"https://github.com/zjukg/snag/blob/HEAD/SNAG_MMEA/model/MSNEA_tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bda2af2880f66879"}},{"code_sha256_prefix":"b4e32355db15fe9f","entry":"get_options","repo":"hackerchenzhuo/LaKo","repo_kind":"listed","path":"src/options.py","file_url":"https://github.com/hackerchenzhuo/LaKo/blob/HEAD/src/options.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b4e32355db15fe9f"}},{"code_sha256_prefix":"fa4ecbff4b9a354b","entry":"has_answer","repo":"hackerchenzhuo/LaKo","repo_kind":"listed","path":"src/evaluation.py","file_url":"https://github.com/hackerchenzhuo/LaKo/blob/HEAD/src/evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa4ecbff4b9a354b"}},{"code_sha256_prefix":"433bcbe46a942d69","entry":"l2norm","repo":"zjukg/snag","repo_kind":"listed","path":"SNAG_MMEA/model/EVA_tools.py","file_url":"https://github.com/zjukg/snag/blob/HEAD/SNAG_MMEA/model/EVA_tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"433bcbe46a942d69"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}