Knowledge Probing

Benchmarks archive 2025-07-28

No benchmark for this task in the archive.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

5 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Most implemented papers archive 2025-07-28

23 shown of 23 papers with code (46 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• GPT Understands, Too
  18 Mar 2021 10 repositories listed Syntology ran 3 of 3 samples · 0 unverified · 2 pointer-only (licence)
  Prompting a pretrained language model with natural language patterns has been proved effective for natural language understanding (NLU).
• Can Language Models Solve Graph Problems in Natural Language?
  17 May 2023 2 repositories listed Syntology ran 2 of 22 samples · 20 unverified
  We then propose Build-a-Graph Prompting and Algorithmic Prompting, two instruction-based approaches to enhance LLMs in solving natural language graph problems.
• LambdaKG: A Library for Pre-trained Language Model-Based Knowledge Graph Embeddings
  1 Oct 2022 2 repositories listed
  Knowledge Graphs (KGs) often have two characteristics: heterogeneous graph structure and text-rich entity/relation information.
• Correlation and Navigation in the Vocabulary Key Representation Space of Language Models
  3 Oct 2024 1 repository listed Syntology ran 2 of 3 samples · 1 unverified · 3 pointer-only (licence)
  In this paper, we study the effect of the key distribution on the NTP distribution, with a focus on whether the similarity between keys will trigger spurious correlations in NTP.
• BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models
  5 Apr 2024 1 repository listed Syntology ran 2 of 2 samples · 0 unverified
  We release the BEAR datasets and an open-source framework that implements the probing approach to the research community to facilitate the evaluation and development of LMs.
• Unveiling LLMs: The Evolution of Latent Representations in a Dynamic Knowledge Graph
  4 Apr 2024 1 repository listed Syntology ran 5 of 5 samples · 0 unverified · 5 pointer-only (licence)
  We propose an end-to-end framework to decode factual knowledge embedded in token representations from a vector space to a set of ground predicates, showing its layer-wise evolution using a dynamic knowledge graph.
• Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge
  8 Mar 2024 1 repository listed
  Acquiring factual knowledge for language models (LMs) in low-resource languages poses a serious challenge, thus resorting to cross-lingual transfer in multilingual LMs (ML-LMs).
• PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain
  22 Oct 2023 1 repository listed
  Biomedical language understanding benchmarks are the driving forces for artificial intelligence applications with large language model (LLM) back-ends.
• Assessing the Reliability of Large Language Model Knowledge
  15 Oct 2023 1 repository listed
  Large language models (LLMs) have been treated as knowledge bases due to their strong performance in knowledge probing tasks.
• Using Large Language Models for Knowledge Engineering (LLMKE): A Case Study on Wikidata
  15 Sep 2023 1 repository listed
  In this work, we explore the use of Large Language Models (LLMs) for knowledge engineering tasks in the context of the ISWC 2023 LM-KBC Challenge.
• LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development
  12 May 2023 1 repository listed
  To this end, we release a multinational English legal corpus (LeXFiles) and a legal knowledge probing benchmark (LegalLAMA) to facilitate training and detailed analysis of legal-oriented PLMs.
• Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding
  21 Mar 2023 1 repository listed
  We show that SOTA multimodally trained text encoders outperform unimodally trained text encoders on the VLU tasks while being underperformed by them on the NLU tasks, lending new context to previously mixed results…
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
  20 Dec 2022 1 repository listed Syntology ran 1 of 4 samples · 3 unverified
  Despite their impressive performance on diverse tasks, large language models (LMs) still struggle with tasks requiring rich world knowledge, implying the limitations of relying solely on their parameters to encode a…
• Injecting Domain Knowledge in Language Models for Task-Oriented Dialogue Systems
  15 Dec 2022 1 repository listed
  Pre-trained language models (PLM) have advanced the state-of-the-art across NLP applications, but lack domain-specific knowledge that does not naturally occur in pre-training data.
• Galactica: A Large Language Model for Science
  16 Nov 2022 1 repository listed Syntology ran 0 of 2 samples · 2 unverified
  We believe these results demonstrate the potential for language models as a new interface for science.
• COPEN: Probing Conceptual Knowledge in Pre-trained Language Models
  8 Nov 2022 1 repository listed Syntology ran 0 of 7 samples · 7 unverified
  We believe this is a critical bottleneck for realizing human-like cognition in PLMs.
• Calibrating Factual Knowledge in Pretrained Language Models
  7 Oct 2022 1 repository listed Syntology ran 1 of 1 samples · 0 unverified
  However, we find that facts stored in the PLMs are not always correct.
• LM-CORE: Language Models with Contextually Relevant External Knowledge
  12 Aug 2022 1 repository listed
  Large transformer-based pre-trained language models have achieved impressive performance on a variety of knowledge-intensive tasks and can capture factual knowledge in their parameters.
• mGPT: Few-Shot Learners Go Multilingual
  15 Apr 2022 1 repository listed
  Recent studies report that autoregressive language models can successfully solve many NLP tasks via zero- and few-shot learning paradigms, which opens up new possibilities for using the pre-trained language models.
• DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understanding
  2 Dec 2021 1 repository listed
  Knowledge-Enhanced Pre-trained Language Models (KEPLMs) are pre-trained models with relation triples injecting from knowledge graphs to improve language understanding abilities.
• Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Models
  15 Oct 2021 1 repository listed
  To catalyse the research in this direction, we release a well-curated biomedical knowledge probing benchmark, MedLAMA, which is constructed based on the Unified Medical Language System (UMLS) Metathesaurus.
• An Empirical Study on Few-shot Knowledge Probing for Pretrained Language Models
  6 Sep 2021 1 repository listed
  Prompt-based knowledge probing for 1-hop relations has been used to measure how much world knowledge is stored in pretrained language models.
• CoLAKE: Contextualized Language and Knowledge Embedding
  1 Oct 2020 1 repository listed Syntology ran 0 of 7 samples · 7 unverified
  With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing models consider shallow, static, and separately pre-trained entity embeddings, which limits the…

Syntology lines on 10 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.