Bias Detection

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

5 leaderboard tables shown for this task, 5 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

11 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

1 subtask in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 80 papers with code (199 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• StereoSet: Measuring stereotypical bias in pretrained language models
  20 Apr 2020 3 repositories listed
  Since pretrained language models are trained on large real world data, they are known to capture stereotypical biases.
• Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models
  16 Sep 2023 2 repositories listed
  LLMs are increasingly powerful and widely used to assist users in a variety of tasks.
• Bipol: Multi-axes Evaluation of Bias with Explainability in Benchmark Datasets
  28 Jan 2023 2 repositories listed
  Hence, we also contribute a new, large Swedish bias-labelled dataset (of 2 million samples), translated from the English version and train the SotA mT5 model on it.
• Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases
  6 Jun 2020 2 repositories listed Syntology ran 6 of 10 samples · 4 unverified · 6 pointer-only (licence)
  Furthermore, we develop two methods, Intersectional Bias Detection (IBD) and Emergent Intersectional Bias Detection (EIBD), to automatically identify the intersectional biases and emergent intersectional biases from…
• Automated Dependence Plots
  2 Dec 2019 2 repositories listed
  To address these drawbacks, we formalize a method for automating the selection of interesting PDPs and extend PDPs beyond showing single features to show the model response along arbitrary directions, for example in raw…
• How Neural Networks Organize Concepts: Introducing Concept Trajectory Analysis for Deep Learning Interpretability
  1 Jun 2025 1 repository listed
  We present Concept Trajectory Analysis (CTA), an interpretability method that tracks how neural networks organize concepts by following their paths through clustered activation spaces across layers.
• To Bias or Not to Bias: Detecting bias in News with bias-detector
  19 May 2025 1 repository listed
  Media bias detection is a critical task in ensuring fair and balanced information dissemination, yet it remains challenging due to the subjectivity of bias and the scarcity of high-quality annotated data.
• Can Global XAI Methods Reveal Injected Bias in LLMs? SHAP vs Rule Extraction vs RuleSHAP
  16 May 2025 1 repository listed
  Generative AI systems can help spread information but also misinformation and biases, potentially undermining the UN Sustainable Development Goals (SDGs).
• Explainable AI in Spatial Analysis
  1 May 2025 1 repository listed
  A key objective in spatial analysis is to model spatial relationships and infer spatial processes to generate knowledge from spatial data, which has been largely based on spatial statistical methods.
• Robust Bias Detection in MLMs and its Application to Human Trait Ratings
  21 Feb 2025 1 repository listed
  There is general agreement for the remaining 3 personality dimensions: both sides observe at most small differences across gender.
• ViLBias: A Comprehensive Framework for Bias Detection through Linguistic and Visual Cues , presenting Annotation Strategies, Evaluation, and Key Challenges
  22 Dec 2024 1 repository listed
  The integration of Large Language Models (LLMs) and Vision-Language Models (VLMs) opens new avenues for addressing complex challenges in multimodal content analysis, particularly in biased news detection.
• MT-LENS: An all-in-one Toolkit for Better Machine Translation Evaluation
  16 Dec 2024 1 repository listed
  We introduce MT-LENS, a framework designed to evaluate Machine Translation (MT) systems across a variety of tasks, including translation quality, gender bias detection, added toxicity, and robustness to misspellings.
• The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection
  17 Nov 2024 1 repository listed
  Our classifier, fine-tuned on this dataset, surpasses all of the annotator LLMs by 5-9 percent in Matthews Correlation Coefficient (MCC) and performs close to or outperforms the model trained on human-labeled data when…
• Mitigating Bias in Queer Representation within Large Language Models: A Collaborative Agent Approach
  12 Nov 2024 1 repository listed Syntology ran 1 of 1 samples · 0 unverified
  Large Language Models (LLMs) often perpetuate biases in pronoun usage, leading to misrepresentation or exclusion of queer individuals.
• debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias
  17 Oct 2024 1 repository listed
  This study displays the problems with the current benchmarks for measuring demographic bias in Vision Language Models and introduces both a more effective dataset for measuring bias and a novel and interpretable…
• GUS-Net: Social Bias Classification in Text with Generalizations, Unfairness, and Stereotypes
  10 Oct 2024 1 repository listed
  The detection of bias in natural language processing (NLP) is a critical challenge, particularly with the increasing use of large language models (LLMs) in various domains.
• TinyEmo: Scaling down Emotional Reasoning via Metric Projection
  9 Oct 2024 1 repository listed
  Our approach features: (1) a synthetic emotional instruct dataset for both pre-training and fine-tuning stages, (2) a Metric Projector that delegates classification from the language model allowing for more efficient…
• Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions
  3 Oct 2024 1 repository listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  To that end, our study investigates the presence of implicit gender biases in multi-agent LLM interactions and proposes two strategies to mitigate these biases.
• Counterfactual Token Generation in Large Language Models
  25 Sep 2024 1 repository listed Syntology ran 10 of 10 samples · 0 unverified · 10 pointer-only (licence)
  Our model allows any large language model to perform counterfactual token generation at almost no cost in comparison with vanilla token generation, it is embarrassingly simple to implement, and it does not require any…
• Explainable AI for computational pathology identifies model limitations and tissue biomarkers
  4 Sep 2024 1 repository listed
  In IDH mutation classification, HIPPO more robustly identified the pathology regions responsible for false negatives compared to attention, suggesting its potential to outperform attention in explaining model decisions.
• GradBias: Unveiling Word Influence on Bias in Text-to-Image Generative Models
  29 Aug 2024 1 repository listed
  OpenBias detects and quantifies biases, while GradBias determines the contribution of individual prompt words on biases.
• The BIAS Detection Framework: Bias Detection in Word Embeddings and Language Models for European Languages
  26 Jul 2024 1 repository listed
  The project BIAS: Mitigating Diversity Biases of AI in the Labor Market is a four-year project funded by the European commission and supported by the Swiss State Secretariat for Education, Research and Innovation (SERI).
• Social Bias in Large Language Models For Bangla: An Empirical Study on Gender and Religious Bias
  3 Jul 2024 1 repository listed
  The rapid growth of Large Language Models (LLMs) has put forward the study of biases as a crucial field.
• A Study of Nationality Bias in Names and Perplexity using Off-the-Shelf Affect-related Tweet Classifiers
  1 Jul 2024 1 repository listed
  In this paper, we apply a method to quantify biases associated with named entities from various countries.
• Gender Bias Detection in Court Decisions: A Brazilian Case Study
  1 Jun 2024 1 repository listed
  Data derived from the realm of the social sciences is often produced in digital text form, which motivates its use as a source for natural language processing methods.
• DispaRisk: Auditing Fairness Through Usable Information
  20 May 2024 1 repository listed
  Machine Learning algorithms (ML) impact virtually every aspect of human lives and have found use across diverse sectors including healthcare, finance, and education.
• SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation
  16 May 2024 1 repository listed Syntology ran 0 of 1 samples · 1 unverified
  It is often desirable to distill the capabilities of large language models (LLMs) into smaller student models due to compute and memory constraints.
• Benchmarking Llama2, Mistral, Gemma and GPT for Factuality, Toxicity, Bias and Propensity for Hallucinations
  15 Apr 2024 1 repository listed Syntology ran 7 of 8 samples · 1 unverified
  In this research, we used OpenAI GPT as point of comparison since it excels at all levels of safety.
• OpenBias: Open-set Bias Detection in Text-to-Image Generative Models
  11 Apr 2024 1 repository listed Syntology ran 9 of 10 samples · 1 unverified · 10 pointer-only (licence)
  In this paper, we tackle the challenge of open-set bias detection in text-to-image generative models presenting OpenBias, a new pipeline that identifies and quantifies the severity of biases agnostically, without access…
• DeNetDM: Debiasing by Network Depth Modulation
  28 Mar 2024 1 repository listed Syntology ran 6 of 7 samples · 1 unverified
  Neural networks trained on biased datasets tend to inadvertently learn spurious correlations, hindering generalization.

Syntology lines on 8 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.