Methods › Natural Language Processing › Autoencoding Transformers › BERT
BERT
Introduced by Jacob Devlin et al. in BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
BERT, or Bidirectional Encoder Representations from Transformers, improves upon standard Transformers by removing the unidirectionality constraint by using a masked language model (MLM) pre-training objective. The masked language model randomly masks some of the tokens from the input, and the objective is to predict the original vocabulary id of the masked word based only on its context. Unlike left-to-right language model pre-training, the MLM objective enables the representation to fuse the left and the right context, which allows us to pre-train a deep bidirectional Transformer. In addition to the masked language model, BERT uses a next sentence prediction task that jointly pre-trains text-pair representations.
There are two steps in BERT: pre-training and fine-tuning. During pre-training, the model is trained on unlabeled data over different pre-training tasks. For fine-tuning, the BERT model is first initialized with the pre-trained parameters, and all of the parameters are fine-tuned using labeled data from the downstream tasks. Each downstream task has separate fine-tuned models, even though they are initialized with the same pre-trained parameters.
In Syntology repository known, no verified implementation (declared_code; repositories: 5 listed of 533 in the graph, strongest link: official). This answer is about arXiv:1810.04805, the paper the archive credits with this method.
Origin paper of this answer, arXiv:1810.04805 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 300 ran (of which 75 constructed an object rather than computing a result; 234 with no instrument failure: 18 honoured, 5 violated, 211 with no contract checked; 66 where Syntology's instrument failed) · 359 unverified (of 659 harvested samples) · 164 pointer-only (licence). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 6,938 · with a code link: 2,862 · where Syntology ran a sample: 640 (505 with a run with no instrument failure, 135 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (640 of 6,938 tagged: 505 with a run with no instrument failure, 135 where every run was a failure of Syntology's instrument)
Page 1 of 70: papers 1 to 100 of 6,938, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Developing Visual Augmented Q&A System using Scalable Vision Embedding Retrieval & Late Interaction Re-ranker 16 Jul 2025 · 1 repository · arXiv:2507.12378
-
Addressing Data Imbalance in Transformer-Based Multi-Label Emotion Detection with Weighted Loss 15 Jul 2025 · 1 repository · arXiv:2507.11384
-
Leveraging RAG-LLMs for Urban Mobility Simulation and Analysis 14 Jul 2025 · 0 repositories · arXiv:2507.10382
-
SentiDrop: A Multi Modal Machine Learning model for Predicting Dropout in Distance Learning 14 Jul 2025 · 0 repositories · arXiv:2507.10421
-
Orchestrator-Agent Trust: A Modular Agentic AI Visual Classification System with Trust-Aware Orchestration and RAG-Based Reasoning 9 Jul 2025 · 1 repository · arXiv:2507.10571
-
The Dark Side of LLMs Agent-based Attacks for Complete Computer Takeover 9 Jul 2025 · 0 repositories · arXiv:2507.06850
-
SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression 8 Jul 2025 · 0 repositories · arXiv:2507.05633
-
AI Generated Text Detection Using Instruction Fine-tuned Large Language and Transformer-Based Models 7 Jul 2025 · 0 repositories · arXiv:2507.05157
-
CyberRAG: An agentic RAG cyber attack classification and reporting tool 3 Jul 2025 · 0 repositories · arXiv:2507.02424
-
Knowledge Protocol Engineering: A New Paradigm for AI in Domain-Specific Knowledge Work 3 Jul 2025 · 0 repositories · arXiv:2507.02760
-
Robustness of Misinformation Classification Systems to Adversarial Examples Through BeamAttack 30 Jun 2025 · 1 repository · arXiv:2506.23661
-
Knowledge Augmented Finetuning Matters in both RAG and Agent Based Dialog Systems 28 Jun 2025 · 0 repositories · arXiv:2506.22852
-
ARAG: Agentic Retrieval Augmented Generation for Personalized Recommendation 27 Jun 2025 · 0 repositories · arXiv:2506.21931
-
EraRAG: Efficient and Incremental Retrieval Augmented Generation for Growing Corpora 26 Jun 2025 · 1 repository · arXiv:2506.20963
-
Leveraging LLM-Assisted Query Understanding for Live Retrieval-Augmented Generation 26 Jun 2025 · 0 repositories · arXiv:2506.21384
-
PsyLite Technical Report 26 Jun 2025 · 1 repository · arXiv:2506.21536
-
Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality 26 Jun 2025 · 1 repository · arXiv:2506.20978Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
AI Assistants to Enhance and Exploit the PETSc Knowledge Base 25 Jun 2025 · 0 repositories · arXiv:2506.20608
-
CCRS: A Zero-Shot LLM-as-a-Judge Framework for Comprehensive RAG Evaluation 25 Jun 2025 · 0 repositories · arXiv:2506.20128
-
Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation 25 Jun 2025 · 0 repositories · arXiv:2506.20869
-
Knowledge-Aware Diverse Reranking for Cross-Source Question Answering 25 Jun 2025 · 0 repositories · arXiv:2506.20476
-
Memento: Note-Taking for Your Future Self 25 Jun 2025 · 0 repositories · arXiv:2506.20642
-
Accurate and Energy Efficient: Local Retrieval-Augmented Generation Models Outperform Commercial Large Language Models in Medical Tasks 24 Jun 2025 · 0 repositories · arXiv:2506.20009
-
Controlled Retrieval-augmented Context Evaluation for Long-form RAG 24 Jun 2025 · 0 repositories · arXiv:2506.20051
-
Inference Scaled GraphRAG: Improving Multi Hop Question Answering on Knowledge Graphs 24 Jun 2025 · 0 repositories · arXiv:2506.19967
-
KunLunBaizeRAG: Reinforcement Learning Driven Inference Performance Leap for Large Language Models 24 Jun 2025 · 0 repositories · arXiv:2506.19466
-
Unlocking Insights Addressing Alcohol Inference Mismatch through Database-Narrative Alignment 24 Jun 2025 · 0 repositories · arXiv:2506.19342
-
An Audio-centric Multi-task Learning Framework for Streaming Ads Targeting on Spotify 23 Jun 2025 · 0 repositories · arXiv:2506.18735
-
Semantic similarity estimation for domain specific data using BERT and other techniques 23 Jun 2025 · 0 repositories · arXiv:2506.18602
-
T-CPDL: A Temporal Causal Probabilistic Description Logic for Developing Logic-RAG Agent 23 Jun 2025 · 0 repositories · arXiv:2506.18559
-
Multimodal Political Bias Identification and Neutralization 20 Jun 2025 · 0 repositories · arXiv:2506.17372
-
Do We Talk to Robots Like Therapists, and Do They Respond Accordingly? Language Alignment in AI Emotional Support 19 Jun 2025 · 0 repositories · arXiv:2506.16473
-
REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing 19 Jun 2025 · 0 repositories · arXiv:2506.16444
-
cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree 18 Jun 2025 · 1 repository · arXiv:2506.15655
-
I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution 18 Jun 2025 · 0 repositories · arXiv:2506.17323
-
Identifying economic narratives in large text corpora -- An integrated approach using Large Language Models 18 Jun 2025 · 0 repositories · arXiv:2506.15041
-
AviationLLM: An LLM-based Knowledge System for Aviation Training 17 Jun 2025 · 0 repositories · arXiv:2506.14336
-
Lightweight Relevance Grader in RAG 17 Jun 2025 · 1 repository · arXiv:2506.14084
-
RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition 17 Jun 2025 · 0 repositories · arXiv:2506.14412
-
SCISSOR: Mitigating Semantic Bias through Cluster-Aware Siamese Networks for Robust Classification 17 Jun 2025 · 1 repository · arXiv:2506.14587Syntology official: harvested, nothing ran · 0 ran · 3 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences 16 Jun 2025 · 2 repositories · arXiv:2506.13996
-
Evaluating Large Language Models for Phishing Detection, Self-Consistency, Faithfulness, and Explainability 16 Jun 2025 · 1 repository · arXiv:2506.13746
-
LTRR: Learning To Rank Retrievers for LLMs 16 Jun 2025 · 1 repository · arXiv:2506.13743
-
Tree-Based Text Retrieval via Hierarchical Clustering in RAGFrameworks: Application on Taiwanese Regulations 16 Jun 2025 · 1 repository · arXiv:2506.13607
-
Rethinking Hate Speech Detection on Social Media: Can LLMs Replace Traditional Models? 15 Jun 2025 · 0 repositories · arXiv:2506.12744
-
FlexRAG: A Flexible and Comprehensive Framework for Retrieval-Augmented Generation 14 Jun 2025 · 1 repository · arXiv:2506.12494
-
Bias Amplification in RAG: Poisoning Knowledge Retrieval to Steer LLMs 13 Jun 2025 · 0 repositories · arXiv:2506.11415
-
Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation 13 Jun 2025 · 0 repositories · arXiv:2506.17277
-
Dr. GPT Will See You Now, but Should It? Exploring the Benefits and Harms of Large Language Models in Medical Diagnosis using Crowdsourced Clinical Cases 13 Jun 2025 · 0 repositories · arXiv:2506.13805
-
Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements 12 Jun 2025 · 0 repositories · arXiv:2506.10330
-
CIIR@LiveRAG 2025: Optimizing Multi-Agent Retrieval Augmented Generation through Self-Training 12 Jun 2025 · 1 repository · arXiv:2506.10844
-
Constructing and Evaluating Declarative RAG Pipelines in PyTerrier 12 Jun 2025 · 1 repository · arXiv:2506.10802
-
Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challenges 12 Jun 2025 · 0 repositories · arXiv:2506.10408
-
TableRAG: A Retrieval Augmented Generation Framework for Heterogeneous Document Reasoning 12 Jun 2025 · 1 repository · arXiv:2506.10380
-
Auto-Compressing Networks 11 Jun 2025 · 0 repositories · arXiv:2506.09714
-
Learning Efficient and Generalizable Graph Retriever for Knowledge-Graph Question Answering 11 Jun 2025 · 1 repository · arXiv:2506.09645
-
LLM-Powered CPI Prediction Inference with Online Text Time Series 11 Jun 2025 · 1 repository · arXiv:2506.09516
-
Measuring Corporate Human Capital Disclosures: Lexicon, Data, Code, and Research Opportunities 11 Jun 2025 · 0 repositories · arXiv:2506.10155
-
You Are What You Say: Exploiting Linguistic Content for VoicePrivacy Attacks 11 Jun 2025 · 0 repositories · arXiv:2506.09521
-
Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environment 10 Jun 2025 · 1 repository · arXiv:2506.10030
-
CC-RAG: Structured Multi-Hop Reasoning via Theme-Based Causal Graphs 10 Jun 2025 · 0 repositories · arXiv:2506.08364
-
DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs 10 Jun 2025 · 1 repository · arXiv:2506.08500
-
FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation 10 Jun 2025 · 1 repository · arXiv:2506.08938
-
FedRAG: A Framework for Fine-Tuning Retrieval-Augmented Generation Systems 10 Jun 2025 · 1 repository · arXiv:2506.09200
-
The Predictive Brain: Neural Correlates of Word Expectancy Align with Large Language Model Prediction Probabilities 10 Jun 2025 · 0 repositories · arXiv:2506.08511
-
XGraphRAG: Interactive Visual Analysis for Graph-based Retrieval-Augmented Generation 10 Jun 2025 · 1 repository · arXiv:2506.13782
-
Benchmarking Foundation Speech and Language Models for Alzheimer's Disease and Related Dementia Detection from Spontaneous Speech 9 Jun 2025 · 0 repositories · arXiv:2506.11119
-
Hierarchical Lexical Graph for Enhanced Multi-Hop Retrieval 9 Jun 2025 · 1 repository · arXiv:2506.08074
-
Knowledge Compression via Question Generation: Enhancing Multihop Document Retrieval without Fine-tuning 9 Jun 2025 · 0 repositories · arXiv:2506.13778
-
LlamaRec-LKG-RAG: A Single-Pass, Learnable Knowledge Graph-RAG Framework for LLM-Based Ranking 9 Jun 2025 · 1 repository · arXiv:2506.07449
-
Multilingual Hate Speech Detection in Social Media Using Translation-Based Approaches with Large Language Models 9 Jun 2025 · 0 repositories · arXiv:2506.08147
-
SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understanding 9 Jun 2025 · 0 repositories · arXiv:2506.07600
-
BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactions 6 Jun 2025 · 0 repositories · arXiv:2506.05766
-
Joint-GCG: Unified Gradient-Based Poisoning Attacks on Retrieval-Augmented Generation Systems 6 Jun 2025 · 1 repository · arXiv:2506.06151
-
When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation 6 Jun 2025 · 1 repository · arXiv:2506.05690Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples)
-
Design of intelligent proofreading system for English translation based on CNN and BERT 5 Jun 2025 · 0 repositories · arXiv:2506.04811
-
Dynamic Context Tuning for Retrieval-Augmented Generation: Enhancing Multi-Turn Planning and Tool Adaptation 5 Jun 2025 · 0 repositories · arXiv:2506.11092
-
ECoRAG: Evidentiality-guided Compression for Long Context RAG 5 Jun 2025 · 1 repository · arXiv:2506.05167
-
Grapheme-Coherent Phonemic and Prosodic Annotation of Speech by Implicit and Explicit Grapheme Conditioning 5 Jun 2025 · 0 repositories · arXiv:2506.04527
-
Knowledgeable-r1: Policy Optimization for Knowledge Exploration in Retrieval-Augmented Generation 5 Jun 2025 · 1 repository · arXiv:2506.05154Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 5 harvested samples)
-
LotusFilter: Fast Diverse Nearest Neighbor Search via a Learned Cutoff Table 5 Jun 2025 · 1 repository · arXiv:2506.04790Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples)
-
Mathematical Reasoning for Unmanned Aerial Vehicles: A RAG-Based Approach for Complex Arithmetic Reasoning 5 Jun 2025 · 1 repository · arXiv:2506.04998
-
Micro-Act: Mitigate Knowledge Conflict in Question Answering via Actionable Self-Reasoning 5 Jun 2025 · 1 repository · arXiv:2506.05278
-
On Automating Security Policies with Contemporary LLMs 5 Jun 2025 · 0 repositories · arXiv:2506.04838
-
Predicting ICU In-Hospital Mortality Using Adaptive Transformer Layer Fusion 5 Jun 2025 · 1 repository · arXiv:2506.04924
-
The NTNU System at the S&I Challenge 2025 SLA Open Track 5 Jun 2025 · 0 repositories · arXiv:2506.05121
-
Magic Mushroom: A Customizable Benchmark for Fine-grained Analysis of Retrieval Noise Erosion in RAG Systems 4 Jun 2025 · 0 repositories · arXiv:2506.03901
-
R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning 4 Jun 2025 · 1 repository · arXiv:2506.04185
-
TracLLM: A Generic Framework for Attributing Long Context LLMs 4 Jun 2025 · 1 repository · arXiv:2506.04202
-
Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds 3 Jun 2025 · 0 repositories · arXiv:2506.03100
-
CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAG 3 Jun 2025 · 0 repositories · arXiv:2506.02544
-
Enhancing Automatic PT Tagging for MEDLINE Citations Using Transformer-Based Models 3 Jun 2025 · 0 repositories · arXiv:2506.03321
-
Rethinking the effects of data contamination in Code Intelligence 3 Jun 2025 · 0 repositories · arXiv:2506.02791
-
Hybrid AI for Responsive Multi-Turn Online Conversations with Novel Dynamic Routing and Feedback Adaptation 2 Jun 2025 · 0 repositories · arXiv:2506.02097
-
LLMs as World Models: Data-Driven and Human-Centered Pre-Event Simulation for Disaster Impact Assessment 2 Jun 2025 · 0 repositories · arXiv:2506.06355
-
Retrieval-Augmented Generation of Ontologies from Relational Databases 2 Jun 2025 · 0 repositories · arXiv:2506.01232
-
A Graph-Retrieval-Augmented Generation Framework Enhances Decision-Making in the Circular Economy 1 Jun 2025 · 0 repositories · arXiv:2506.04252
-
RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented Generation Systems 1 Jun 2025 · 1 repository · arXiv:2506.00789
-
FinBERT2: A Specialized Bidirectional Encoder for Bridging the Gap in Finance-Specific Deployment of Large Language Models 31 May 2025 · 0 repositories · arXiv:2506.06335
-
Adversarial Threat Vectors and Risk Mitigation for Retrieval-Augmented Generation Systems 30 May 2025 · 0 repositories · arXiv:2506.00281
Tasks archive 2025-07-28
20 shown of 1,187 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| RAG | 1,288 |
| Retrieval | 1,260 |
| Language Modelling | 1,119 |
| Retrieval-augmented Generation | 1,092 |
| Language Modeling | 903 |
| Question Answering | 751 |
| Sentence | 730 |
| Sentiment Analysis | 406 |
| Text Classification | 387 |
| text-classification | 342 |
| Transfer Learning | 305 |
| Classification | 261 |
| Word Embeddings | 258 |
| Information Retrieval | 243 |
| Named Entity Recognition | 242 |
| Named Entity Recognition (NER) | 231 |
| named-entity-recognition | 231 |
| Natural Language Understanding | 229 |
| Large Language Model | 221 |
| Articles | 188 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections