Browse State-of-the-Art › TruthfulQA
TruthfulQA
37 papers with code · 0 benchmarks · 3 datasets archive 2025-07-28
Benchmarks archive 2025-07-28
No benchmark for this task in the archive.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
3 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 80 · with a code link: 37 · where Syntology ran a sample: 20 (14 with a run with no instrument failure, 6 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (20 of 80 tagged: 14 with a run with no instrument failure, 6 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 80 of 80, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
7 Sep 2023 4 repositories listed Syntology official (archive's flag): 5 ran · 5 ran (of which 2 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 7 harvested samples) · 7 pointer-only (licence)Despite their impressive capabilities, large language models (LLMs) are prone to hallucinations, i. e., generating content that deviates from facts seen during pretraining.
-
13 May 2024 3 repositories listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)We present the workflow of Online Iterative Reinforcement Learning from Human Feedback (RLHF) in this technical report, which is widely reported to outperform its offline counterpart by a large margin in the recent…
-
8 Sep 2021 3 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)We crafted questions that some humans would answer falsely due to a false belief or misconception.
-
31 May 2024 2 repositories listedTo calibrate both implicit and explicit confidence markers, we introduce a pragmatic, listener-aware finetuning method (LACIE) that models the listener, considering not only whether an answer is right, but whether it…
-
16 Jan 2024 2 repositories listed Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified (of 10 harvested samples) · 10 pointer-only (licence)Despite the general capabilities of large pretrained language models, they consistently benefit from further adaptation to better achieve desired behaviors.
-
25 Dec 2023 2 repositories listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 6 unverified (of 13 harvested samples) · 6 pointer-only (licence)Experimental results on both discrimination-based and generation-based hallucination evaluation benchmarks, such as TruthfulQA and FActScore, demonstrate that our proposed ICD methods can effectively enhance the…
-
18 Aug 2023 2 repositories listedIn this work, we propose a new safety evaluation benchmark RED-EVAL that carries out red-teaming.
-
11 Jun 2025 1 repository listedFinally, we show that our method can improve the training of frontier LMs: we use our method to train an unsupervised reward model and use reinforcement learning to train a Claude 3.
-
18 May 2025 1 repository listedIn this paper, we propose a method for identifying representations of truthfulness at the neuron level.
-
4 Mar 2025 1 repository listedLarge Language Models (LLMs) are increasingly being used in real-world applications.
-
13 Feb 2025 1 repository listed Syntology official: not harvested · 0 ran · 1 unverified (of 1 harvested sample)We introduce a professionally translated extension of the TruthfulQA benchmark designed to evaluate truthfulness in Basque, Catalan, Galician, and Spanish.
-
6 Feb 2025 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample) · 1 pointer-only (licence)First, we develop an agent-fusion framework for encouraging multiple LLM based agents to collaborate in producing the final inference output for each LLM query.
-
5 Jan 2025 1 repository listedIn this work, we introduce CHAIR (Classifier of Hallucination As ImproveR), a supervised framework for detecting hallucinations by analyzing internal logits from each layer of every token.
-
3 Jan 2025 1 repository listedLarge Language Models (LLMs) have become essential tools across various domains due to their impressive capabilities in understanding and generating human-like text.
-
11 Oct 2024 1 repository listed Syntology 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)NoVo demonstrates exceptional generalization to 20 diverse datasets, with significant gains in over 90% of them, far exceeding all current representation editing and reading methods.
-
7 Oct 2024 1 repository listedLLM-based NLP systems typically work by embedding their input data into prompt templates which contain instructions and/or in-context examples, creating queries which are submitted to a LLM, and then parsing the LLM…
-
2 Oct 2024 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 3 harvested samples) · 3 pointer-only (licence)Self-consistency-based approaches, which involve repeatedly sampling multiple outputs and selecting the most consistent one as the final response, prove to be remarkably effective in improving the factual accuracy of…
-
4 Jul 2024 1 repository listed Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified (of 9 harvested samples) · 9 pointer-only (licence)Large Language Models (LLMs) vary in their abilities on a range of tasks.
-
25 Jun 2024 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples) · 6 pointer-only (licence)As large language models achieve impressive scores on traditional benchmarks, an increasing number of researchers are becoming concerned about benchmark data leakage during pre-training, commonly known as the data…
-
21 Jun 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)To demonstrate the generality and transferability of our method beyond jailbreaks, we show that our KTS model can be steered to reduce bias towards user-suggested answers on TruthfulQA.
-
19 Jun 2024 1 repository listedBuilt on top of ActCab, we further propose CoDec, a confidence-guided decoding strategy to elicit truthful answers with high confidence from LMs.
-
24 May 2024 1 repository listedThis paper presents a dual-pronged approach to enhance the ethical and safe behavior of large language models (LLMs) by addressing the issues of harmful responses and copyrighted content.
-
23 May 2024 1 repository listed Syntology official (archive's flag): 16 ran · 16 ran (of which 0 constructed an object rather than computing a result; 11 with no instrument failure: 0 honoured, 1 violated, 10 with no contract checked; 5 where Syntology's instrument failed) · 4 unverified (of 20 harvested samples) · 20 pointer-only (licence)We observe that IM is especially beneficial when trained on datasets with lengthy instructions paired with brief outputs, or under the Superficial Alignment Hypothesis (SAH) where a small amount of training examples are…
-
14 Apr 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)In this paper, we set out to clarify these capabilities under a more stringent evaluation setting in which we disallow any kind of external feedback.
-
6 Apr 2024 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 1 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples) · 7 pointer-only (licence)Despite tremendous advancements in large language models (LLMs) over recent years, a notably urgent challenge for their practical deployment is the phenomenon of hallucination, where the model fabricates facts and…
-
27 Mar 2024 1 repository listedIn this work, we explore LLM's internal representation space to identify attention heads that contain the most truthful and accurate information.
-
3 Mar 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples) · 4 pointer-only (licence)Large language models (LLMs) frequently hallucinate and produce factual errors, yet our understanding of why they make these errors remains limited.
-
27 Feb 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 2 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; every one of the 2 samples that ran constructed an object rather than computing a result (of 3 harvested samples) · 3 pointer-only (licence)In this paper, we propose TruthX, an inference-time intervention method to activate the truthfulness of LLM by identifying and editing the features within LLM's internal representations that govern the truthfulness.
-
21 Feb 2024 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)To this extent, we construct the Moral Consistency Corpus (MCC), containing 50K moral questions, responses to them by LLMs, and the RoTs that these models followed.
-
29 Dec 2023 1 repository listedWe introduce Truth Forest, a method that enhances truthfulness in LLMs by uncovering hidden truth representations using multi-dimensional orthogonal probes.
-
14 Oct 2023 1 repository listed
-
2 Oct 2023 1 repository listed Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 8 harvested samples)
-
13 Sep 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
13 Sep 2023 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 1 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 6 harvested samples) · 2 pointer-only (licence)
-
6 Jun 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample)
-
24 May 2023 1 repository listed
-
10 Nov 2022 1 repository listed
-
Model Unlearning via Sparse Autoencoder Subspace Guided Projections30 May 2025 0 repositories listed
-
Shadows in the Attention: Contextual Perturbation and Representation Drift in the Dynamics of Hallucination in LLMs22 May 2025 0 repositories listed
-
Elastic Weight Consolidation for Full-Parameter Continual Pre-Training of Gemma29 May 2025 0 repositories listed
-
DYNAMAX: Dynamic computing for Transformers and Mamba based architectures29 Apr 2025 0 repositories listed
-
Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer17 Apr 2025 0 repositories listed
-
Sample, Don't Search: Rethinking Test-Time Alignment for Language Models4 Apr 2025 0 repositories listed
-
Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency4 Apr 2025 0 repositories listed
-
More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment3 Apr 2025 0 repositories listed
-
When Persuasion Overrides Truth in Multi-Agent LLM Debates: Introducing a Confidence-Weighted Persuasion Override Rate (CW-POR)1 Apr 2025 0 repositories listed
-
Obliviate: Efficient Unmemorization for Protecting Intellectual Property in Large Language Models20 Feb 2025 0 repositories listed
-
Cost-Saving LLM Cascades with Early Abstention13 Feb 2025 0 repositories listed
-
Selective Self-to-Supervised Fine-Tuning for Generalization in Large Language Models12 Feb 2025 0 repositories listed
-
TruthFlow: Truthful LLM Generation via Representation Flow Correction6 Feb 2025 0 repositories listed
-
Monty Hall and Optimized Conformal Prediction to Improve Decision-Making with LLMs31 Dec 2024 0 repositories listed
-
Mitigating Adversarial Attacks in LLMs through Defensive Suffix Generation18 Dec 2024 0 repositories listed
-
Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languages1 Dec 2024 0 repositories listed
-
Layer Importance and Hallucination Analysis in Large Language Models via Enhanced Activation Variance-Sparsity15 Nov 2024 0 repositories listed
-
Maintaining Informative Coherence: Migrating Hallucinations in Large Language Models via Absorbing Markov Chains27 Oct 2024 0 repositories listed
-
A Debate-Driven Experiment on LLM Hallucinations and Accuracy25 Oct 2024 0 repositories listed
-
Evaluating Consistencies in LLM responses through a Semantic Clustering of Question Answering20 Oct 2024 0 repositories listed
-
Iter-AHMCL: Alleviate Hallucination for Large Language Model via Iterative Model-level Contrastive Learning16 Oct 2024 0 repositories listed
-
SkillAggregation: Reference-free LLM-Dependent Aggregation14 Oct 2024 0 repositories listed
-
Benchmark Inflation: Revealing LLM Performance Gaps Using Retro-Holdouts11 Oct 2024 0 repositories listed
-
Towards Multilingual LLM Evaluation for European Languages11 Oct 2024 0 repositories listed
-
Efficiently Deploying LLMs with Controlled Risk3 Oct 2024 0 repositories listed
-
Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs30 Sep 2024 0 repositories listed
-
Selective Self-Rehearsal: A Fine-Tuning Approach to Improve Generalization in Large Language Models7 Sep 2024 0 repositories listed
-
Lower Layer Matters: Alleviating Hallucination via Multi-Layer Fusion Contrastive Decoding with Truthfulness Refocused16 Aug 2024 0 repositories listed
-
LokiLM: Technical Report10 Jul 2024 0 repositories listed
-
Multi-Reference Preference Optimization for Large Language Models26 May 2024 0 repositories listed
-
Harmonic LLMs are Trustworthy30 Apr 2024 0 repositories listed
-
Student Data Paradox and Curious Case of Single Student-Tutor Model: Regressive Side Effects of Training LLMs for Personalized Learning23 Apr 2024 0 repositories listed
-
PRobELM: Plausibility Ranking Evaluation for Language Models4 Apr 2024 0 repositories listed
-
LLMAuditor: A Framework for Auditing Large Language Models Using Human-in-the-Loop14 Feb 2024 0 repositories listed
-
Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation14 Feb 2024 0 repositories listed
-
GRATH: Gradual Self-Truthifying for Large Language Models22 Jan 2024 0 repositories listed
-
Reducing LLM Hallucinations using Epistemic Neural Networks25 Dec 2023 0 repositories listed
-
Self-Evaluation Improves Selective Generation in Large Language Models14 Dec 2023 0 repositories listed
-
Uncertainty-aware Language Modeling for Selective Question Answering26 Nov 2023 0 repositories listed
-
Investigating Data Contamination in Modern Benchmarks for Large Language Models16 Nov 2023 0 repositories listed
-
On The Truthfulness of 'Surprisingly Likely' Responses of Large Language Models13 Nov 2023 0 repositories listed
-
Semantic Consistency for Assuring Reliability of Large Language Models17 Aug 2023 0 repositories listed
-
Teaching language models to support answers with verified quotes21 Mar 2022 0 repositories listed
Syntology lines on 22 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections