Methods › Natural Language Processing › Subword Segmentation › WordPiece
WordPiece
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
WordPiece is a subword segmentation algorithm used in natural language processing. The vocabulary is initialized with individual characters in the language, then the most frequent combinations of symbols in the vocabulary are iteratively added to the vocabulary. The process is:
- Initialize the word unit inventory with all the characters in the text.
- Build a language model on the training data using the inventory from 1.
- Generate a new word unit by combining two units out of the current word inventory to increment the word unit inventory by one. Choose the new word unit out of all the possible ones that increases the likelihood on the training data the most when added to the model.
- Goto 2 until a predefined limit of word units is reached or the likelihood increase falls below a certain threshold.
Text: Source
Image: WordPiece as used in BERT
Papers archive 2025-07-28
30 shown of 7,063, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements 12 Jun 2025 · 0 repositories · arXiv:2506.10330
-
ALBERT: Advanced Localization and Bidirectional Encoder Representations from Transformers for Automotive Damage Evaluation 12 Jun 2025 · 0 repositories · arXiv:2506.10524
-
CIIR@LiveRAG 2025: Optimizing Multi-Agent Retrieval Augmented Generation through Self-Training 12 Jun 2025 · 1 repository · arXiv:2506.10844
-
Constructing and Evaluating Declarative RAG Pipelines in PyTerrier 12 Jun 2025 · 1 repository · arXiv:2506.10802
-
Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challenges 12 Jun 2025 · 0 repositories · arXiv:2506.10408
-
TableRAG: A Retrieval Augmented Generation Framework for Heterogeneous Document Reasoning 12 Jun 2025 · 1 repository · arXiv:2506.10380
-
Auto-Compressing Networks 11 Jun 2025 · 0 repositories · arXiv:2506.09714
-
Learning Efficient and Generalizable Graph Retriever for Knowledge-Graph Question Answering 11 Jun 2025 · 1 repository · arXiv:2506.09645
-
LLM-Powered CPI Prediction Inference with Online Text Time Series 11 Jun 2025 · 1 repository · arXiv:2506.09516
-
Measuring Corporate Human Capital Disclosures: Lexicon, Data, Code, and Research Opportunities 11 Jun 2025 · 0 repositories · arXiv:2506.10155
-
You Are What You Say: Exploiting Linguistic Content for VoicePrivacy Attacks 11 Jun 2025 · 0 repositories · arXiv:2506.09521
-
Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environment 10 Jun 2025 · 1 repository · arXiv:2506.10030
-
CC-RAG: Structured Multi-Hop Reasoning via Theme-Based Causal Graphs 10 Jun 2025 · 0 repositories · arXiv:2506.08364
-
DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs 10 Jun 2025 · 1 repository · arXiv:2506.08500
-
FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation 10 Jun 2025 · 1 repository · arXiv:2506.08938
-
FedRAG: A Framework for Fine-Tuning Retrieval-Augmented Generation Systems 10 Jun 2025 · 1 repository · arXiv:2506.09200
-
The Predictive Brain: Neural Correlates of Word Expectancy Align with Large Language Model Prediction Probabilities 10 Jun 2025 · 0 repositories · arXiv:2506.08511
-
Hierarchical Lexical Graph for Enhanced Multi-Hop Retrieval 9 Jun 2025 · 1 repository · arXiv:2506.08074
-
LlamaRec-LKG-RAG: A Single-Pass, Learnable Knowledge Graph-RAG Framework for LLM-Based Ranking 9 Jun 2025 · 1 repository · arXiv:2506.07449
-
Multilingual Hate Speech Detection in Social Media Using Translation-Based Approaches with Large Language Models 9 Jun 2025 · 0 repositories · arXiv:2506.08147
-
SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understanding 9 Jun 2025 · 0 repositories · arXiv:2506.07600
-
Joint-GCG: Unified Gradient-Based Poisoning Attacks on Retrieval-Augmented Generation Systems 6 Jun 2025 · 1 repository · arXiv:2506.06151
-
When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation 6 Jun 2025 · 1 repository · arXiv:2506.05690Syntology ran 0 of 5 samples · 5 unverified
-
Design of intelligent proofreading system for English translation based on CNN and BERT 5 Jun 2025 · 0 repositories · arXiv:2506.04811
-
Dynamic Context Tuning for Retrieval-Augmented Generation: Enhancing Multi-Turn Planning and Tool Adaptation 5 Jun 2025 · 0 repositories · arXiv:2506.11092
-
ECoRAG: Evidentiality-guided Compression for Long Context RAG 5 Jun 2025 · 1 repository · arXiv:2506.05167
-
Grapheme-Coherent Phonemic and Prosodic Annotation of Speech by Implicit and Explicit Grapheme Conditioning 5 Jun 2025 · 0 repositories · arXiv:2506.04527
-
Knowledgeable-r1: Policy Optimization for Knowledge Exploration in Retrieval-Augmented Generation 5 Jun 2025 · 1 repository · arXiv:2506.05154Syntology ran 3 of 5 samples · 2 unverified
-
Mathematical Reasoning for Unmanned Aerial Vehicles: A RAG-Based Approach for Complex Arithmetic Reasoning 5 Jun 2025 · 1 repository · arXiv:2506.04998
-
Micro-Act: Mitigate Knowledge Conflict in Question Answering via Actionable Self-Reasoning 5 Jun 2025 · 1 repository · arXiv:2506.05278
Tasks archive 2025-07-28
20 shown of 1,195 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Retrieval | 1,242 |
| RAG | 1,241 |
| Language Modelling | 1,157 |
| Retrieval-augmented Generation | 1,057 |
| Language Modeling | 935 |
| Question Answering | 758 |
| Sentence | 747 |
| Sentiment Analysis | 411 |
| Text Classification | 393 |
| text-classification | 347 |
| Transfer Learning | 314 |
| Classification | 263 |
| Word Embeddings | 260 |
| Named Entity Recognition | 248 |
| Information Retrieval | 244 |
| Named Entity Recognition (NER) | 237 |
| named-entity-recognition | 237 |
| Natural Language Understanding | 233 |
| Large Language Model | 213 |
| Articles | 193 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections