{"url":"/method/wordpiece","slug":"wordpiece","name":"WordPiece","full_name":"WordPiece","full_name_withheld":false,"description_markdown":"**WordPiece** is a subword segmentation algorithm used in natural language processing.  The vocabulary is initialized with individual characters in the language, then the most frequent combinations of symbols in the vocabulary are iteratively added to the vocabulary. The process is:\r\n\r\n1. Initialize the word unit inventory with all the characters in the text.\r\n2. Build a language model on the training data using the inventory from 1.\r\n3. Generate a new word unit by combining two units out of the current word inventory to increment the word unit inventory by one. Choose the new word unit out of all the possible ones that increases the likelihood on the training data the most when added to the model.\r\n4. Goto 2 until a predefined limit of word units is reached or the likelihood increase falls below a certain threshold.\r\n\r\nText: [Source](https://stackoverflow.com/questions/55382596/how-is-wordpiece-tokenization-helpful-to-effectively-deal-with-rare-words-proble/55416944#55416944)\r\n\r\nImage: WordPiece as used in [BERT](https://paperswithcode.com/method/bert)","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"http://arxiv.org/abs/1609.08144v2","title":"Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Subword Segmentation","url":"/methods/category/subword-segmentation","pwc_aliases":[]}],"n_papers_tagged":7063,"archive_num_papers":null,"papers_newest_first":[{"paper":null,"title":"Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements","date":"2025-06-12","arxiv_id":"2506.10330","n_code_links":0,"syntology":null},{"paper":null,"title":"ALBERT: Advanced Localization and Bidirectional Encoder Representations from Transformers for Automotive Damage Evaluation","date":"2025-06-12","arxiv_id":"2506.10524","n_code_links":0,"syntology":null},{"paper":"/paper/ciir-liverag-2025-optimizing-multi-agent","title":"CIIR@LiveRAG 2025: Optimizing Multi-Agent Retrieval Augmented Generation through Self-Training","date":"2025-06-12","arxiv_id":"2506.10844","n_code_links":1,"syntology":null},{"paper":"/paper/constructing-and-evaluating-declarative-rag","title":"Constructing and Evaluating Declarative RAG Pipelines in PyTerrier","date":"2025-06-12","arxiv_id":"2506.10802","n_code_links":1,"syntology":null},{"paper":null,"title":"Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challenges","date":"2025-06-12","arxiv_id":"2506.10408","n_code_links":0,"syntology":null},{"paper":"/paper/tablerag-a-retrieval-augmented-generation","title":"TableRAG: A Retrieval Augmented Generation Framework for Heterogeneous Document Reasoning","date":"2025-06-12","arxiv_id":"2506.10380","n_code_links":1,"syntology":null},{"paper":null,"title":"Auto-Compressing Networks","date":"2025-06-11","arxiv_id":"2506.09714","n_code_links":0,"syntology":null},{"paper":"/paper/learning-efficient-and-generalizable-graph","title":"Learning Efficient and Generalizable Graph Retriever for Knowledge-Graph Question Answering","date":"2025-06-11","arxiv_id":"2506.09645","n_code_links":1,"syntology":null},{"paper":"/paper/llm-powered-cpi-prediction-inference-with","title":"LLM-Powered CPI Prediction Inference with Online Text Time Series","date":"2025-06-11","arxiv_id":"2506.09516","n_code_links":1,"syntology":null},{"paper":null,"title":"Measuring Corporate Human Capital Disclosures: Lexicon, Data, Code, and Research Opportunities","date":"2025-06-11","arxiv_id":"2506.10155","n_code_links":0,"syntology":null},{"paper":null,"title":"You Are What You Say: Exploiting Linguistic Content for VoicePrivacy Attacks","date":"2025-06-11","arxiv_id":"2506.09521","n_code_links":0,"syntology":null},{"paper":"/paper/2506-10030","title":"Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environment","date":"2025-06-10","arxiv_id":"2506.10030","n_code_links":1,"syntology":null},{"paper":null,"title":"CC-RAG: Structured Multi-Hop Reasoning via Theme-Based Causal Graphs","date":"2025-06-10","arxiv_id":"2506.08364","n_code_links":0,"syntology":null},{"paper":"/paper/dragged-into-conflicts-detecting-and","title":"DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs","date":"2025-06-10","arxiv_id":"2506.08500","n_code_links":1,"syntology":null},{"paper":"/paper/faithfulrag-fact-level-conflict-modeling-for","title":"FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation","date":"2025-06-10","arxiv_id":"2506.08938","n_code_links":1,"syntology":null},{"paper":"/paper/fedrag-a-framework-for-fine-tuning-retrieval","title":"FedRAG: A Framework for Fine-Tuning Retrieval-Augmented Generation Systems","date":"2025-06-10","arxiv_id":"2506.09200","n_code_links":1,"syntology":null},{"paper":null,"title":"The Predictive Brain: Neural Correlates of Word Expectancy Align with Large Language Model Prediction Probabilities","date":"2025-06-10","arxiv_id":"2506.08511","n_code_links":0,"syntology":null},{"paper":"/paper/hierarchical-lexical-graph-for-enhanced-multi","title":"Hierarchical Lexical Graph for Enhanced Multi-Hop Retrieval","date":"2025-06-09","arxiv_id":"2506.08074","n_code_links":1,"syntology":null},{"paper":"/paper/llamarec-lkg-rag-a-single-pass-learnable","title":"LlamaRec-LKG-RAG: A Single-Pass, Learnable Knowledge Graph-RAG Framework for LLM-Based Ranking","date":"2025-06-09","arxiv_id":"2506.07449","n_code_links":1,"syntology":null},{"paper":null,"title":"Multilingual Hate Speech Detection in Social Media Using Translation-Based Approaches with Large Language Models","date":"2025-06-09","arxiv_id":"2506.08147","n_code_links":0,"syntology":null},{"paper":null,"title":"SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understanding","date":"2025-06-09","arxiv_id":"2506.07600","n_code_links":0,"syntology":null},{"paper":"/paper/joint-gcg-unified-gradient-based-poisoning","title":"Joint-GCG: Unified Gradient-Based Poisoning Attacks on Retrieval-Augmented Generation Systems","date":"2025-06-06","arxiv_id":"2506.06151","n_code_links":1,"syntology":null},{"paper":"/paper/when-to-use-graphs-in-rag-a-comprehensive","title":"When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation","date":"2025-06-06","arxiv_id":"2506.05690","n_code_links":1,"syntology":{"ran":0,"of":5,"unverified":5,"pointer_only":0}},{"paper":null,"title":"Design of intelligent proofreading system for English translation based on CNN and BERT","date":"2025-06-05","arxiv_id":"2506.04811","n_code_links":0,"syntology":null},{"paper":null,"title":"Dynamic Context Tuning for Retrieval-Augmented Generation: Enhancing Multi-Turn Planning and Tool Adaptation","date":"2025-06-05","arxiv_id":"2506.11092","n_code_links":0,"syntology":null},{"paper":"/paper/ecorag-evidentiality-guided-compression-for","title":"ECoRAG: Evidentiality-guided Compression for Long Context RAG","date":"2025-06-05","arxiv_id":"2506.05167","n_code_links":1,"syntology":null},{"paper":null,"title":"Grapheme-Coherent Phonemic and Prosodic Annotation of Speech by Implicit and Explicit Grapheme Conditioning","date":"2025-06-05","arxiv_id":"2506.04527","n_code_links":0,"syntology":null},{"paper":"/paper/knowledgeable-r1-policy-optimization-for","title":"Knowledgeable-r1: Policy Optimization for Knowledge Exploration in Retrieval-Augmented Generation","date":"2025-06-05","arxiv_id":"2506.05154","n_code_links":1,"syntology":{"ran":3,"of":5,"unverified":2,"pointer_only":0}},{"paper":"/paper/mathematical-reasoning-for-unmanned-aerial","title":"Mathematical Reasoning for Unmanned Aerial Vehicles: A RAG-Based Approach for Complex Arithmetic Reasoning","date":"2025-06-05","arxiv_id":"2506.04998","n_code_links":1,"syntology":null},{"paper":"/paper/micro-act-mitigate-knowledge-conflict-in","title":"Micro-Act: Mitigate Knowledge Conflict in Question Answering via Actionable Self-Reasoning","date":"2025-06-05","arxiv_id":"2506.05278","n_code_links":1,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/retrieval","name":"Retrieval","papers":1242},{"task":"/task/rag","name":"RAG","papers":1241},{"task":"/task/language-modelling","name":"Language Modelling","papers":1157},{"task":"/task/retrieval-augmented-generation","name":"Retrieval-augmented Generation","papers":1057},{"task":"/task/language-modeling","name":"Language Modeling","papers":935},{"task":"/task/question-answering","name":"Question Answering","papers":758},{"task":"/task/sentence","name":"Sentence","papers":747},{"task":"/task/sentiment-analysis","name":"Sentiment Analysis","papers":411},{"task":"/task/text-classification","name":"Text Classification","papers":393},{"task":"/task/text-classification-1","name":"text-classification","papers":347},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":314},{"task":"/task/classification-1","name":"Classification","papers":263},{"task":"/task/word-embeddings","name":"Word Embeddings","papers":260},{"task":"/task/named-entity-recognition-1","name":"Named Entity Recognition","papers":248},{"task":"/task/information-retrieval","name":"Information Retrieval","papers":244},{"task":"/task/named-entity-recognition-ner","name":"Named Entity Recognition (NER)","papers":237},{"task":"/task/named-entity-recognition","name":"named-entity-recognition","papers":237},{"task":"/task/natural-language-understanding","name":"Natural Language Understanding","papers":233},{"task":"/task/large-language-model","name":"Large Language Model","papers":213},{"task":"/task/articles","name":"Articles","papers":193}],"tasks_shown":20,"n_tasks":1195,"usage_by_year":[{"year":"2016","papers":1},{"year":"2017","papers":1},{"year":"2018","papers":6},{"year":"2019","papers":578},{"year":"2020","papers":1258},{"year":"2021","papers":1343},{"year":"2022","papers":847},{"year":"2023","papers":860},{"year":"2024","papers":1399},{"year":"2025","papers":770}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/wordpiece"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}