{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/re-improving-multi-hop-question-answering","title":"[Re] Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings","arxiv_id":null,"date":"2021-01-31","proceeding":"RC 2020","authors":["Jishnu Jaykumar P","Ashish Sardana"],"abstract":"Scope of Reproducibility\n\nOur work consists of four parts:\n1. Reproducing the results from [1].\n2. Exploring the effect of various knowledge graph embedding models in the Knowledge Graph Embedding module.\n3. Exploring the effect of various transformer models in the Question Embedding\nmodule.\n4. Verifying the importance of the Relation Matching (RM) module.\nBased on the code shared by the authors, we have reproduced the results for EmbedKGQA[1]. We have not performed relation matching deliberately to validate point-4.\n\nMethodology\n\nWe have used the code provided by [1] with some customization for reproducibility. In addition to making the codebase more modular and easy to navigate, we have made changes to incorporate different transformers in the question embedding module. QuestionAnswering models were trained from scratch as no pre-trained models were available for our particular dataset. The code for this work is available on GitHub (See page footer for the link).\n\nResults\n\nWe were able to reproduce the Hits@1 to be within ±2.4% of the reported value (in most cases). Anomalies were observed in 2 cases.\n1. In MetaQA-KG-Full (3-hop) dataset.\n2. WebQSP-KG-Full dataset.\nFrom our experiments on the QA model, we have found that a recent transformer architecture, SBERT[2] produced better accuracy than the original paper. Replacing RoBERTa[3] with SBERT[2] increased the absolute accuracy by ≈3.4% and ≈0.6% in the half KG and the full KG case respectively. (KG: Knowledge Graph, ”≈”: Approximately)\n\nWhat was easy\n\nAs the code was open-sourced, we didnʼt have to implement the paper giving us the liberty to customize the codebase to focus on the authorʼs claim validation, perform extended experiments and explore shared as well as new models. In addition to this, pretrained KG embedding models were shared which helped in the reproduction experiment.\n\nWhat was difficult\n\nThe lack of comprehensive documentation along with missing comments defining functions/classes/attributes etc. made it laborious to review the code and modify it. In addition to large training times for question answering models, the knowledge graph embeddings also required a significant amount of computing resources.\n\nCommunication with original authors\n\nWe had a couple of virtual meetings with Apoorv Saxena, the primary author of EmbedKGQA[1].","url_abs":"http://rescience.github.io/bibliography/P_2021.html","url_pdf":"https://zenodo.org/record/4834942/files/article.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"re-improving-multi-hop-question-answering","repo_url":"https://github.com/jishnujayakumar/MLRC2020-EmbedKGQA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"answer-selection","task_name":"Answer Selection"},{"task_slug":"graph-embedding","task_name":"Graph Embedding"},{"task_slug":"knowledge-graph-embedding","task_name":"Knowledge Graph Embedding"},{"task_slug":"knowledge-graph-embeddings","task_name":"Knowledge Graph Embeddings"},{"task_slug":"knowledge-graphs","task_name":"Knowledge Graphs"},{"task_slug":"link-prediction","task_name":"Link Prediction"},{"task_slug":"multi-hop-question-answering","task_name":"Multi-hop Question Answering"},{"task_slug":"navigate","task_name":"Navigate"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}