Methods › Natural Language Processing › Autoencoding Transformers › BERT
BERT
Introduced by Jacob Devlin et al. in BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
BERT, or Bidirectional Encoder Representations from Transformers, improves upon standard Transformers by removing the unidirectionality constraint by using a masked language model (MLM) pre-training objective. The masked language model randomly masks some of the tokens from the input, and the objective is to predict the original vocabulary id of the masked word based only on its context. Unlike left-to-right language model pre-training, the MLM objective enables the representation to fuse the left and the right context, which allows us to pre-train a deep bidirectional Transformer. In addition to the masked language model, BERT uses a next sentence prediction task that jointly pre-trains text-pair representations.
There are two steps in BERT: pre-training and fine-tuning. During pre-training, the model is trained on unlabeled data over different pre-training tasks. For fine-tuning, the BERT model is first initialized with the pre-trained parameters, and all of the parameters are fine-tuned using labeled data from the downstream tasks. Each downstream task has separate fine-tuned models, even though they are initialized with the same pre-trained parameters.
Papers archive 2025-07-28
30 shown of 6,938, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Developing Visual Augmented Q&A System using Scalable Vision Embedding Retrieval & Late Interaction Re-ranker 16 Jul 2025 · 1 repository · arXiv:2507.12378
-
Addressing Data Imbalance in Transformer-Based Multi-Label Emotion Detection with Weighted Loss 15 Jul 2025 · 1 repository · arXiv:2507.11384
-
Leveraging RAG-LLMs for Urban Mobility Simulation and Analysis 14 Jul 2025 · 0 repositories · arXiv:2507.10382
-
SentiDrop: A Multi Modal Machine Learning model for Predicting Dropout in Distance Learning 14 Jul 2025 · 0 repositories · arXiv:2507.10421
-
Orchestrator-Agent Trust: A Modular Agentic AI Visual Classification System with Trust-Aware Orchestration and RAG-Based Reasoning 9 Jul 2025 · 1 repository · arXiv:2507.10571
-
The Dark Side of LLMs Agent-based Attacks for Complete Computer Takeover 9 Jul 2025 · 0 repositories · arXiv:2507.06850
-
SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression 8 Jul 2025 · 0 repositories · arXiv:2507.05633
-
AI Generated Text Detection Using Instruction Fine-tuned Large Language and Transformer-Based Models 7 Jul 2025 · 0 repositories · arXiv:2507.05157
-
CyberRAG: An agentic RAG cyber attack classification and reporting tool 3 Jul 2025 · 0 repositories · arXiv:2507.02424
-
Knowledge Protocol Engineering: A New Paradigm for AI in Domain-Specific Knowledge Work 3 Jul 2025 · 0 repositories · arXiv:2507.02760
-
Robustness of Misinformation Classification Systems to Adversarial Examples Through BeamAttack 30 Jun 2025 · 1 repository · arXiv:2506.23661
-
Knowledge Augmented Finetuning Matters in both RAG and Agent Based Dialog Systems 28 Jun 2025 · 0 repositories · arXiv:2506.22852
-
ARAG: Agentic Retrieval Augmented Generation for Personalized Recommendation 27 Jun 2025 · 0 repositories · arXiv:2506.21931
-
EraRAG: Efficient and Incremental Retrieval Augmented Generation for Growing Corpora 26 Jun 2025 · 1 repository · arXiv:2506.20963
-
Leveraging LLM-Assisted Query Understanding for Live Retrieval-Augmented Generation 26 Jun 2025 · 0 repositories · arXiv:2506.21384
-
PsyLite Technical Report 26 Jun 2025 · 1 repository · arXiv:2506.21536
-
Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality 26 Jun 2025 · 1 repository · arXiv:2506.20978Syntology ran 0 of 3 samples · 3 unverified · 3 pointer-only (licence)
-
AI Assistants to Enhance and Exploit the PETSc Knowledge Base 25 Jun 2025 · 0 repositories · arXiv:2506.20608
-
CCRS: A Zero-Shot LLM-as-a-Judge Framework for Comprehensive RAG Evaluation 25 Jun 2025 · 0 repositories · arXiv:2506.20128
-
Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation 25 Jun 2025 · 0 repositories · arXiv:2506.20869
-
Knowledge-Aware Diverse Reranking for Cross-Source Question Answering 25 Jun 2025 · 0 repositories · arXiv:2506.20476
-
Memento: Note-Taking for Your Future Self 25 Jun 2025 · 0 repositories · arXiv:2506.20642
-
Accurate and Energy Efficient: Local Retrieval-Augmented Generation Models Outperform Commercial Large Language Models in Medical Tasks 24 Jun 2025 · 0 repositories · arXiv:2506.20009
-
Controlled Retrieval-augmented Context Evaluation for Long-form RAG 24 Jun 2025 · 0 repositories · arXiv:2506.20051
-
Inference Scaled GraphRAG: Improving Multi Hop Question Answering on Knowledge Graphs 24 Jun 2025 · 0 repositories · arXiv:2506.19967
-
KunLunBaizeRAG: Reinforcement Learning Driven Inference Performance Leap for Large Language Models 24 Jun 2025 · 0 repositories · arXiv:2506.19466
-
Unlocking Insights Addressing Alcohol Inference Mismatch through Database-Narrative Alignment 24 Jun 2025 · 0 repositories · arXiv:2506.19342
-
An Audio-centric Multi-task Learning Framework for Streaming Ads Targeting on Spotify 23 Jun 2025 · 0 repositories · arXiv:2506.18735
-
Semantic similarity estimation for domain specific data using BERT and other techniques 23 Jun 2025 · 0 repositories · arXiv:2506.18602
-
T-CPDL: A Temporal Causal Probabilistic Description Logic for Developing Logic-RAG Agent 23 Jun 2025 · 0 repositories · arXiv:2506.18559
Tasks archive 2025-07-28
20 shown of 1,187 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| RAG | 1,288 |
| Retrieval | 1,260 |
| Language Modelling | 1,119 |
| Retrieval-augmented Generation | 1,092 |
| Language Modeling | 903 |
| Question Answering | 751 |
| Sentence | 730 |
| Sentiment Analysis | 406 |
| Text Classification | 387 |
| text-classification | 342 |
| Transfer Learning | 305 |
| Classification | 261 |
| Word Embeddings | 258 |
| Information Retrieval | 243 |
| Named Entity Recognition | 242 |
| Named Entity Recognition (NER) | 231 |
| named-entity-recognition | 231 |
| Natural Language Understanding | 229 |
| Large Language Model | 221 |
| Articles | 188 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections