Browse State-of-the-Art › Knowledge Probing
Knowledge Probing
23 papers with code · 0 benchmarks · 5 datasets archive 2025-07-28
Benchmarks archive 2025-07-28
No benchmark for this task in the archive.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
5 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 46 · with a code link: 23 · where Syntology ran a sample: 10 (7 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (10 of 46 tagged: 7 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 46 of 46, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
18 Mar 2021 10 repositories listed Syntology official (archive's flag): 1 ran · 3 ran (of which 1 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 2 pointer-only (licence)Prompting a pretrained language model with natural language patterns has been proved effective for natural language understanding (NLU).
-
17 May 2023 2 repositories listed Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 2 where Syntology's instrument failed) · 11 unverified (of 22 harvested samples)We then propose Build-a-Graph Prompting and Algorithmic Prompting, two instruction-based approaches to enhance LLMs in solving natural language graph problems.
-
1 Oct 2022 2 repositories listedKnowledge Graphs (KGs) often have two characteristics: heterogeneous graph structure and text-rich entity/relation information.
-
3 Oct 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 3 pointer-only (licence)In this paper, we study the effect of the key distribution on the NTP distribution, with a focus on whether the similarity between keys will trigger spurious correlations in NTP.
-
5 Apr 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)We release the BEAR datasets and an open-source framework that implements the probing approach to the research community to facilitate the evaluation and development of LMs.
-
4 Apr 2024 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 5 pointer-only (licence)We propose an end-to-end framework to decode factual knowledge embedded in token representations from a vector space to a set of ground predicates, showing its layer-wise evolution using a dynamic knowledge graph.
-
8 Mar 2024 1 repository listedAcquiring factual knowledge for language models (LMs) in low-resource languages poses a serious challenge, thus resorting to cross-lingual transfer in multilingual LMs (ML-LMs).
-
22 Oct 2023 1 repository listedBiomedical language understanding benchmarks are the driving forces for artificial intelligence applications with large language model (LLM) back-ends.
-
15 Oct 2023 1 repository listedLarge language models (LLMs) have been treated as knowledge bases due to their strong performance in knowledge probing tasks.
-
15 Sep 2023 1 repository listedIn this work, we explore the use of Large Language Models (LLMs) for knowledge engineering tasks in the context of the ISWC 2023 LM-KBC Challenge.
-
12 May 2023 1 repository listedTo this end, we release a multinational English legal corpus (LeXFiles) and a legal knowledge probing benchmark (LegalLAMA) to facilitate training and detailed analysis of legal-oriented PLMs.
-
21 Mar 2023 1 repository listedWe show that SOTA multimodally trained text encoders outperform unimodally trained text encoders on the VLU tasks while being underperformed by them on the NLU tasks, lending new context to previously mixed results…
-
20 Dec 2022 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples)Despite their impressive performance on diverse tasks, large language models (LMs) still struggle with tasks requiring rich world knowledge, implying the limitations of relying solely on their parameters to encode a…
-
15 Dec 2022 1 repository listedPre-trained language models (PLM) have advanced the state-of-the-art across NLP applications, but lack domain-specific knowledge that does not naturally occur in pre-training data.
-
16 Nov 2022 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)We believe these results demonstrate the potential for language models as a new interface for science.
-
8 Nov 2022 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified (of 7 harvested samples)We believe this is a critical bottleneck for realizing human-like cognition in PLMs.
-
7 Oct 2022 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)However, we find that facts stored in the PLMs are not always correct.
-
12 Aug 2022 1 repository listedLarge transformer-based pre-trained language models have achieved impressive performance on a variety of knowledge-intensive tasks and can capture factual knowledge in their parameters.
-
15 Apr 2022 1 repository listedRecent studies report that autoregressive language models can successfully solve many NLP tasks via zero- and few-shot learning paradigms, which opens up new possibilities for using the pre-trained language models.
-
DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understanding2 Dec 2021 1 repository listedKnowledge-Enhanced Pre-trained Language Models (KEPLMs) are pre-trained models with relation triples injecting from knowledge graphs to improve language understanding abilities.
-
15 Oct 2021 1 repository listedTo catalyse the research in this direction, we release a well-curated biomedical knowledge probing benchmark, MedLAMA, which is constructed based on the Unified Medical Language System (UMLS) Metathesaurus.
-
6 Sep 2021 1 repository listedPrompt-based knowledge probing for 1-hop relations has been used to measure how much world knowledge is stored in pretrained language models.
-
1 Oct 2020 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples)With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing models consider shallow, static, and separately pre-trained entity embeddings, which limits the…
-
Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probing27 May 2025 0 repositories listedThe reliability of large language models (LLMs) is greatly compromised by their tendency to hallucinate, underscoring the need for precise identification of knowledge gaps within LLMs.
-
"Let's Argue Both Sides": Argument Generation Can Force Small Models to Utilize Previously Inaccessible Reasoning Capabilities16 Oct 2024 0 repositories listedLarge Language Models (LLMs), despite achieving state-of-the-art results in a number of evaluation tasks, struggle to maintain their performance when logical reasoning is strictly required to correctly infer a…
-
LM-PUB-QUIZ: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models28 Aug 2024 0 repositories listedKnowledge probing evaluates the extent to which a language model (LM) has acquired relational knowledge during its pre-training phase.
-
Knowledge Probing for Graph Representation Learning7 Aug 2024 0 repositories listedGraph learning methods have been extensively applied in diverse application areas.
-
What Matters in Memorizing and Recalling Facts? Multifaceted Benchmarks for Knowledge Probing in Language Models18 Jun 2024 0 repositories listedLanguage models often struggle with handling factual knowledge, exhibiting factual hallucination issue.
-
Chaos with Keywords: Exposing Large Language Models Sycophantic Hallucination to Misleading Keywords and Evaluating Defense Strategies6 Jun 2024 0 repositories listedThis study explores the sycophantic tendencies of Large Language Models (LLMs), where these models tend to provide answers that match what users want to hear, even if they are not entirely correct.
-
TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Models17 Mar 2024 0 repositories listedKEPLMs are pre-trained models that utilize external knowledge to enhance language understanding.
-
Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigation27 Jan 2024 0 repositories listed
-
Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?7 Nov 2023 0 repositories listed
-
Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models25 Oct 2023 0 repositories listed
-
Kiki or Bouba? Sound Symbolism in Vision-and-Language Models25 Oct 2023 0 repositories listed
-
Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names?14 Sep 2023 0 repositories listed
-
Knowledge-augmented Frame Semantic Parsing with Hybrid Prompt-tuning25 Mar 2023 0 repositories listed
-
Context Variance Evaluation of Pretrained Language Models for Prompt-based Biomedical Knowledge Probing18 Nov 2022 0 repositories listed
-
Pre-training Language Models with Deterministic Factual Knowledge20 Oct 2022 0 repositories listed
-
PromptGen: Automatically Generate Prompts using Generative Models1 Jul 2022 0 repositories listed
-
Prompt-Learning for Fine-Grained Entity Typing16 Nov 2021 0 repositories listed
-
Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Models16 Nov 2021 0 repositories listed
-
Towards Fully Self-Supervised Learning of Knowledge from Unstructured Text16 Nov 2021 0 repositories listed
-
Integrating Transformers and Knowledge Graphs for Twitter Stance Detection1 Nov 2021 0 repositories listed
-
Prompt-Learning for Fine-Grained Entity Typing24 Aug 2021 0 repositories listed
-
Pretrain Knowledge-Aware Language Models1 Jan 2021 0 repositories listed
-
Knowledge-Aware Language Model Pretraining29 Jun 2020 0 repositories listed
Syntology lines on 10 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections