Methods › Natural Language Processing › Subword Segmentation › BPE
Byte Pair Encoding
BPE
Introduced by Rico Sennrich et al. in Neural Machine Translation of Rare Words with Subword Units
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations).
Lei Mao has a detailed blog post that explains how this works.
In Syntology about another paper, arXiv:2601.10024: no code under this exact name (no_code; paper_only). About another paper: this answer is about arXiv:2601.10024, while the archive credits arXiv:1508.07909, so it does not say whether this method has code.
Origin paper of this answer, arXiv:2601.10024 (its page on Syntology).
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 18,975 · with a code link: 8,675 · where Syntology ran a sample: 2,895 (2,353 with a run with no instrument failure, 542 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (2,895 of 18,975 tagged: 2,353 with a run with no instrument failure, 542 where every run was a failure of Syntology's instrument)
Page 1 of 190: papers 1 to 100 of 18,975, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
DASViT: Differentiable Architecture Search for Vision Transformer 17 Jul 2025 · 0 repositories · arXiv:2507.13079
-
Making Language Model a Hierarchical Classifier and Generator 17 Jul 2025 · 1 repository · arXiv:2507.12930
-
Best Practices for Large-Scale, Pixel-Wise Crop Mapping and Transfer Learning Workflows 16 Jul 2025 · 1 repository · arXiv:2507.12590
-
DVFL-Net: A Lightweight Distilled Video Focal Modulation Network for Spatio-Temporal Action Recognition 16 Jul 2025 · 1 repository · arXiv:2507.12426
-
Biological Processing Units: Leveraging an Insect Connectome to Pioneer Biofidelic Neural Architectures 15 Jul 2025 · 0 repositories · arXiv:2507.10951
-
Generative Click-through Rate Prediction with Applications to Search Advertising 15 Jul 2025 · 0 repositories · arXiv:2507.11246
-
Hashed Watermark as a Filter: Defeating Forging and Overwriting Attacks in Weight-based Neural Network Watermarking 15 Jul 2025 · 1 repository · arXiv:2507.11137
-
KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding 15 Jul 2025 · 1 repository · arXiv:2507.11273Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Langevin Flows for Modeling Neural Latent Dynamics 15 Jul 2025 · 1 repository · arXiv:2507.11531
-
Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI 13 Jul 2025 · 0 repositories · arXiv:2507.09702
-
Learning from Synthetic Labs: Language Models as Auction Participants 12 Jul 2025 · 0 repositories · arXiv:2507.09083
-
Comparative Analysis of Vision Transformers and Traditional Deep Learning Approaches for Automated Pneumonia Detection in Chest X-Rays 11 Jul 2025 · 0 repositories · arXiv:2507.10589
-
A Wireless Foundation Model for Multi-Task Prediction 8 Jul 2025 · 0 repositories · arXiv:2507.05938
-
Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solving 8 Jul 2025 · 1 repository · arXiv:2507.06229Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Geo-Registration of Terrestrial LiDAR Point Clouds with Satellite Images without GNSS 8 Jul 2025 · 0 repositories · arXiv:2507.05999
-
Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate 8 Jul 2025 · 1 repository · arXiv:2507.07129
-
SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression 8 Jul 2025 · 0 repositories · arXiv:2507.05633
-
Tile-Based ViT Inference with Visual-Cluster Priors for Zero-Shot Multi-Species Plant Identification 8 Jul 2025 · 1 repository · arXiv:2507.06093
-
AI Generated Text Detection Using Instruction Fine-tuned Large Language and Transformer-Based Models 7 Jul 2025 · 0 repositories · arXiv:2507.05157
-
Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations 7 Jul 2025 · 1 repository · arXiv:2507.04886
-
Estimating Interventional Distributions with Uncertain Causal Graphs through Meta-Learning 7 Jul 2025 · 0 repositories · arXiv:2507.05526
-
SV-DRR: High-Fidelity Novel View X-Ray Synthesis Using Diffusion Model 7 Jul 2025 · 1 repository · arXiv:2507.05148
-
Behaviour Space Analysis of LLM-driven Meta-heuristic Discovery 4 Jul 2025 · 0 repositories · arXiv:2507.03605
-
CyberRAG: An agentic RAG cyber attack classification and reporting tool 3 Jul 2025 · 0 repositories · arXiv:2507.02424
-
DeepGesture: A conversational gesture synthesis system based on emotions and semantics 3 Jul 2025 · 1 repository · arXiv:2507.03147
-
Fast and Simplex: 2-Simplicial Attention in Triton 3 Jul 2025 · 0 repositories · arXiv:2507.02754
-
Knowledge Protocol Engineering: A New Paradigm for AI in Domain-Specific Knowledge Work 3 Jul 2025 · 0 repositories · arXiv:2507.02760
-
AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation 2 Jul 2025 · 0 repositories · arXiv:2507.01961
-
Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer 2 Jul 2025 · 1 repository · arXiv:2507.02199Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 1 honoured, 2 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 11 harvested samples) · 11 pointer-only (licence)
-
A Unified Transformer-Based Framework with Pretraining For Whole Body Grasping Motion Generation 1 Jul 2025 · 1 repository · arXiv:2507.00676
-
Large Language Models Don't Make Sense of Word Problems. A Scoping Review from a Mathematics Education Perspective 30 Jun 2025 · 0 repositories · arXiv:2506.24006
-
Mamba-FETrack V2: Revisiting State Space Model for Frame-Event based Visual Object Tracking 30 Jun 2025 · 1 repository · arXiv:2506.23783
-
CycleVAR: Repurposing Autoregressive Model for Unsupervised One-Step Image Translation 29 Jun 2025 · 1 repository · arXiv:2506.23347
-
Agent-to-Agent Theory of Mind: Testing Interlocutor Awareness among Large Language Models 28 Jun 2025 · 0 repositories · arXiv:2506.22957
-
Attention to Burstiness: Low-Rank Bilinear Prompt Tuning 28 Jun 2025 · 1 repository · arXiv:2506.22908
-
Boosting Generative Adversarial Transferability with Self-supervised Vision Transformer Features 26 Jun 2025 · 1 repository · arXiv:2506.21046
-
Cat and Mouse -- Can Fake Text Generation Outpace Detector Systems? 26 Jun 2025 · 0 repositories · arXiv:2506.21274
-
Chain-of-Thought Enhanced Shallow Transformers for Wireless Symbol Detection 26 Jun 2025 · 0 repositories · arXiv:2506.21093
-
DBConformer: Dual-Branch Convolutional Transformer for EEG Decoding 26 Jun 2025 · 1 repository · arXiv:2506.21140
-
Early Stopping Tabular In-Context Learning 26 Jun 2025 · 0 repositories · arXiv:2506.21387
-
EraRAG: Efficient and Incremental Retrieval Augmented Generation for Growing Corpora 26 Jun 2025 · 1 repository · arXiv:2506.20963
-
Forecasting Geopolitical Events with a Sparse Temporal Fusion Transformer and Gaussian Process Hybrid: A Case Study in Middle Eastern and U.S. Conflict Dynamics 26 Jun 2025 · 0 repositories · arXiv:2506.20935
-
Large Language Models Acing Chartered Accountancy 26 Jun 2025 · 0 repositories · arXiv:2506.21031
-
Leveraging LLM-Assisted Query Understanding for Live Retrieval-Augmented Generation 26 Jun 2025 · 0 repositories · arXiv:2506.21384
-
M2SFormer: Multi-Spectral and Multi-Scale Attention with Edge-Aware Difficulty Guidance for Image Forgery Localization 26 Jun 2025 · 0 repositories · arXiv:2506.20922
-
Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models 26 Jun 2025 · 0 repositories · arXiv:2506.21119
-
PsyLite Technical Report 26 Jun 2025 · 1 repository · arXiv:2506.21536
-
Real-Time ESFP: Estimating, Smoothing, Filtering, and Pose-Mapping 26 Jun 2025 · 1 repository · arXiv:2506.21234
-
Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality 26 Jun 2025 · 1 repository · arXiv:2506.20978Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
Transformer-Based Spatial-Temporal Counterfactual Outcomes Estimation 26 Jun 2025 · 1 repository · arXiv:2506.21154Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Video Virtual Try-on with Conditional Diffusion Transformer Inpainter 26 Jun 2025 · 0 repositories · arXiv:2506.21270
-
A foundation model with multi-variate parallel attention to generate neuronal activity 25 Jun 2025 · 1 repository · arXiv:2506.20354
-
AdvMIM: Adversarial Masked Image Modeling for Semi-Supervised Medical Image Segmentation 25 Jun 2025 · 0 repositories · arXiv:2506.20563
-
AI Assistants to Enhance and Exploit the PETSc Knowledge Base 25 Jun 2025 · 0 repositories · arXiv:2506.20608
-
CCRS: A Zero-Shot LLM-as-a-Judge Framework for Comprehensive RAG Evaluation 25 Jun 2025 · 0 repositories · arXiv:2506.20128
-
Differential Transformer-driven 6G Physical Layer for Collaborative Perception Enhancement 25 Jun 2025 · 0 repositories · arXiv:2506.20597
-
Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation 25 Jun 2025 · 0 repositories · arXiv:2506.20869
-
Feature Hallucination for Self-supervised Action Recognition 25 Jun 2025 · 0 repositories · arXiv:2506.20342
-
ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset 25 Jun 2025 · 0 repositories · arXiv:2506.20093
-
Knowledge-Aware Diverse Reranking for Cross-Source Question Answering 25 Jun 2025 · 0 repositories · arXiv:2506.20476
-
Large Language Model-Driven Code Compliance Checking in Building Information Modeling 25 Jun 2025 · 0 repositories · arXiv:2506.20551
-
Med-Art: Diffusion Transformer for 2D Medical Text-to-Image Generation 25 Jun 2025 · 0 repositories · arXiv:2506.20449
-
Memento: Note-Taking for Your Future Self 25 Jun 2025 · 0 repositories · arXiv:2506.20642
-
MS-IQA: A Multi-Scale Feature Fusion Network for PET/CT Image Quality Assessment 25 Jun 2025 · 1 repository · arXiv:2506.20200
-
Probing AI Safety with Source Code 25 Jun 2025 · 1 repository · arXiv:2506.20471
-
X-SiT: Inherently Interpretable Surface Vision Transformers for Dementia Diagnosis 25 Jun 2025 · 0 repositories · arXiv:2506.20267
-
Accurate and Energy Efficient: Local Retrieval-Augmented Generation Models Outperform Commercial Large Language Models in Medical Tasks 24 Jun 2025 · 0 repositories · arXiv:2506.20009
-
Controlled Retrieval-augmented Context Evaluation for Long-form RAG 24 Jun 2025 · 0 repositories · arXiv:2506.20051
-
Inference Scaled GraphRAG: Improving Multi Hop Question Answering on Knowledge Graphs 24 Jun 2025 · 0 repositories · arXiv:2506.19967
-
KunLunBaizeRAG: Reinforcement Learning Driven Inference Performance Leap for Large Language Models 24 Jun 2025 · 0 repositories · arXiv:2506.19466
-
Lost in Translation? Converting RegExes for Log Parsing into Dynatrace Pattern Language 24 Jun 2025 · 0 repositories · arXiv:2506.19539
-
Toward the Explainability of Protein Language Models for Sequence Design 24 Jun 2025 · 0 repositories · arXiv:2506.19532
-
Vision Transformer-Based Time-Series Image Reconstruction for Cloud-Filling Applications 24 Jun 2025 · 0 repositories · arXiv:2506.19591
-
WebGuard++:Interpretable Malicious URL Detection via Bidirectional Fusion of HTML Subgraphs and Multi-Scale Convolutional BERT 24 Jun 2025 · 0 repositories · arXiv:2506.19356
-
A Study of Dynamic Stock Relationship Modeling and S&P500 Price Forecasting Based on Differential Graph Transformer 23 Jun 2025 · 0 repositories · arXiv:2506.18717
-
Deep CNN Face Matchers Inherently Support Revocable Biometric Templates 23 Jun 2025 · 0 repositories · arXiv:2506.18731
-
Fully Few-shot Class-incremental Audio Classification Using Multi-level Embedding Extractor and Ridge Regression Classifier 23 Jun 2025 · 1 repository · arXiv:2506.18406
-
MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners 23 Jun 2025 · 0 repositories · arXiv:2506.18729
-
Rethinking Decoder Design: Improving Biomarker Segmentation Using Depth-to-Space Restoration and Residual Linear Attention 23 Jun 2025 · 1 repository · arXiv:2506.18335Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Security Assessment of DeepSeek and GPT Series Models against Jailbreak Attacks 23 Jun 2025 · 0 repositories · arXiv:2506.18543
-
SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications 23 Jun 2025 · 0 repositories · arXiv:2506.18951
-
T-CPDL: A Temporal Causal Probabilistic Description Logic for Developing Logic-RAG Agent 23 Jun 2025 · 0 repositories · arXiv:2506.18559
-
Transformer World Model for Sample Efficient Multi-Agent Reinforcement Learning 23 Jun 2025 · 1 repository · arXiv:2506.18537
-
Leveraging Large Language Model for Intelligent Log Processing and Autonomous Debugging in Cloud AI Platforms 22 Jun 2025 · 0 repositories · arXiv:2506.17900
-
In-Context Learning Strategies Emerge Rationally 21 Jun 2025 · 0 repositories · arXiv:2506.17859
-
MTSIC: Multi-stage Transformer-based GAN for Spectral Infrared Image Colorization 21 Jun 2025 · 0 repositories · arXiv:2506.17540
-
IsoNet: Causal Analysis of Multimodal Transformers for Neuromuscular Gesture Classification 20 Jun 2025 · 0 repositories · arXiv:2506.16744
-
Leveraging LLMs to Assess Tutor Moves in Real-Life Dialogues: A Feasibility Study 20 Jun 2025 · 0 repositories · arXiv:2506.17410
-
Optimal Depth of Neural Networks 20 Jun 2025 · 0 repositories · arXiv:2506.16862
-
Trans²-CBCT: A Dual-Transformer Framework for Sparse-View CBCT Reconstruction 20 Jun 2025 · 0 repositories · arXiv:2506.17425
-
UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation 20 Jun 2025 · 1 repository · arXiv:2506.17202Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 8 where Syntology's instrument failed) · 0 unverified (of 8 harvested samples) · 8 pointer-only (licence)
-
VMRA-MaR: An Asymmetry-Aware Temporal Framework for Longitudinal Breast Cancer Risk Prediction 20 Jun 2025 · 1 repository · arXiv:2506.17412
-
CapsDT: Diffusion-Transformer for Capsule Robot Manipulation 19 Jun 2025 · 0 repositories · arXiv:2506.16263
-
Do We Talk to Robots Like Therapists, and Do They Respond Accordingly? Language Alignment in AI Emotional Support 19 Jun 2025 · 0 repositories · arXiv:2506.16473
-
Exoplanet Classification through Vision Transformers with Temporal Image Analysis 19 Jun 2025 · 0 repositories · arXiv:2506.16597
-
REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing 19 Jun 2025 · 0 repositories · arXiv:2506.16444
-
STAR-Pose: Efficient Low-Resolution Video Human Pose Estimation via Spatial-Temporal Adaptive Super-Resolution 19 Jun 2025 · 0 repositories · arXiv:2506.16061
-
cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree 18 Jun 2025 · 1 repository · arXiv:2506.15655
-
DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement 18 Jun 2025 · 2 repositories · arXiv:2506.15583Syntology official (archive's flag): 17 ran · 17 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 1 honoured, 1 violated, 7 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified (of 19 harvested samples) · 19 pointer-only (licence)
-
I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution 18 Jun 2025 · 0 repositories · arXiv:2506.17323
Tasks archive 2025-07-28
20 shown of 2,351 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Language Modelling | 2,160 |
| Language Modeling | 1,675 |
| Retrieval | 1,510 |
| RAG | 1,346 |
| Decoder | 1,232 |
| Retrieval-augmented Generation | 1,166 |
| Question Answering | 1,130 |
| Translation | 930 |
| Machine Translation | 856 |
| Large Language Model | 762 |
| Sentence | 712 |
| Text Generation | 689 |
| Semantic Segmentation | 681 |
| Object Detection | 555 |
| Image Classification | 534 |
| object-detection | 503 |
| Representation Learning | 472 |
| Segmentation | 462 |
| image-classification | 431 |
| Transfer Learning | 417 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections