{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mask-and-reason-pre-training-knowledge-graph","title":"Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries","arxiv_id":"2208.07638","date":"2022-08-16","proceeding":null,"authors":["Xiao Liu","Shiyu Zhao","Kai Su","Yukuo Cen","Jiezhong Qiu","Mengdi Zhang","Wei Wu","Yuxiao Dong","Jie Tang"],"abstract":"Knowledge graph (KG) embeddings have been a mainstream approach for reasoning over incomplete KGs. However, limited by their inherently shallow and static architectures, they can hardly deal with the rising focus on complex logical queries, which comprise logical operators, imputed edges, multiple source entities, and unknown intermediate entities. In this work, we present the Knowledge Graph Transformer (kgTransformer) with masked pre-training and fine-tuning strategies. We design a KG triple transformation method to enable Transformer to handle KGs, which is further strengthened by the Mixture-of-Experts (MoE) sparse activation. We then formulate the complex logical queries as masked prediction and introduce a two-stage masked pre-training strategy to improve transferability and generalizability. Extensive experiments on two benchmarks demonstrate that kgTransformer can consistently outperform both KG embedding-based baselines and advanced encoders on nine in-domain and out-of-domain reasoning tasks. Additionally, kgTransformer can reason with explainability via providing the full reasoning paths to interpret given answers.","url_abs":"https://arxiv.org/abs/2208.07638v1","url_pdf":"https://arxiv.org/pdf/2208.07638v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mask-and-reason-pre-training-knowledge-graph","repo_url":"https://github.com/thudm/kgtransformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"graph-transformer","method_name":"Graph Transformer"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"lapeigen","method_name":"LapEigen"},{"method_slug":"laplacian-pe","method_name":"Laplacian PE"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2208.07638","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.07638"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thudm/kgtransformer","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":10},"by_repo_kind":{"official":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b5dc748a91e5e704","entry":"det_softmax","repo":"thudm/kgtransformer","repo_kind":"official","path":"deter_util.py","file_url":"https://github.com/thudm/kgtransformer/blob/HEAD/deter_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5dc748a91e5e704"}},{"code_sha256_prefix":"acf50f7fc9d5fea7","entry":"deter_scatter_add_","repo":"thudm/kgtransformer","repo_kind":"official","path":"deter_util.py","file_url":"https://github.com/thudm/kgtransformer/blob/HEAD/deter_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"acf50f7fc9d5fea7"}},{"code_sha256_prefix":"116134e273b34b63","entry":"get_adj_np","repo":"thudm/kgtransformer","repo_kind":"official","path":"graph_util.py","file_url":"https://github.com/thudm/kgtransformer/blob/HEAD/graph_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"116134e273b34b63"}},{"code_sha256_prefix":"314fee3dec4f4876","entry":"get_directed_lap_matrix_np","repo":"thudm/kgtransformer","repo_kind":"official","path":"graph_util.py","file_url":"https://github.com/thudm/kgtransformer/blob/HEAD/graph_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"314fee3dec4f4876"}},{"code_sha256_prefix":"0f476445cb4e78aa","entry":"ladies_sampler_np","repo":"thudm/kgtransformer","repo_kind":"official","path":"sampler.py","file_url":"https://github.com/thudm/kgtransformer/blob/HEAD/sampler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f476445cb4e78aa"}},{"code_sha256_prefix":"dd1cd67c9cfc143f","entry":"loss_cross_entropy_multi_ans","repo":"thudm/kgtransformer","repo_kind":"official","path":"metric.py","file_url":"https://github.com/thudm/kgtransformer/blob/HEAD/metric.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd1cd67c9cfc143f"}},{"code_sha256_prefix":"70f267c347fddaaa","entry":"loss_label_smoothing_multi_ans","repo":"thudm/kgtransformer","repo_kind":"official","path":"metric.py","file_url":"https://github.com/thudm/kgtransformer/blob/HEAD/metric.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"70f267c347fddaaa"}},{"code_sha256_prefix":"8b0d003d9e5f8311","entry":"read_pkl","repo":"thudm/kgtransformer","repo_kind":"official","path":"tasks/base.py","file_url":"https://github.com/thudm/kgtransformer/blob/HEAD/tasks/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b0d003d9e5f8311"}},{"code_sha256_prefix":"98b43295d1265f8f","entry":"read_txt_triples","repo":"thudm/kgtransformer","repo_kind":"official","path":"tasks/base.py","file_url":"https://github.com/thudm/kgtransformer/blob/HEAD/tasks/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"98b43295d1265f8f"}},{"code_sha256_prefix":"b7a726a8d5be145b","entry":"row_normalize_np","repo":"thudm/kgtransformer","repo_kind":"official","path":"graph_util.py","file_url":"https://github.com/thudm/kgtransformer/blob/HEAD/graph_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b7a726a8d5be145b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}