Methods › Natural Language Processing › Transformers › RoBERTa
RoBERTa
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
RoBERTa is an extension of BERT with changes to the pretraining procedure. The modifications include:
- training the model longer, with bigger batches, over more data
- removing the next sentence prediction objective
- training on longer sequences
- dynamically changing the masking pattern applied to the training data. The authors also collect a large new dataset (CC-News) of comparable size to other privately used datasets, to better control for training set size effects
Papers archive 2025-07-28
30 shown of 913, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Rethinking the effects of data contamination in Code Intelligence 3 Jun 2025 · 0 repositories · arXiv:2506.02791
-
Evaluating AI capabilities in detecting conspiracy theories on YouTube 29 May 2025 · 1 repository · arXiv:2505.23570
-
Detection of Suicidal Risk on Social Media: A Hybrid Model 26 May 2025 · 0 repositories · arXiv:2505.23797
-
Hermes@DravidianLangTech 2025: Sentiment Analysis of Dravidian Languages using XLM-RoBERTa 25 May 2025 · 1 repository
-
Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval 25 May 2025 · 1 repository · arXiv:2505.19356
-
Multi-Scale Probabilistic Generation Theory: A Hierarchical Framework for Interpreting Large Language Models 23 May 2025 · 0 repositories · arXiv:2505.18244
-
Climate Research Domain BERTs: Pretraining, Adaptation, and Evaluation 19 May 2025 · 0 repositories
-
Suicide Risk Assessment Using Multimodal Speech Features: A Study on the SW1 Challenge Dataset 19 May 2025 · 0 repositories · arXiv:2505.13069
-
KGAlign: Joint Semantic-Structural Knowledge Encoding for Multimodal Fake News Detection 18 May 2025 · 1 repository · arXiv:2505.14714
-
Comparative sentiment analysis of public perception: Monkeypox vs. COVID-19 behavioral insights 12 May 2025 · 0 repositories · arXiv:2505.07430
-
The Sound of Populism: Distinct Linguistic Features Across Populist Variants 10 May 2025 · 0 repositories · arXiv:2505.07874
-
Leveraging Language Models for Automated Patient Record Linkage 21 Apr 2025 · 0 repositories · arXiv:2504.15261
-
LLMs as Data Annotators: How Close Are We to Human Performance 21 Apr 2025 · 0 repositories · arXiv:2504.15022
-
The Synthetic Imputation Approach: Generating Optimal Synthetic Texts For Underrepresented Categories In Supervised Classification Tasks 21 Apr 2025 · 0 repositories · arXiv:2504.15160
-
ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance 11 Apr 2025 · 0 repositories · arXiv:2504.08716
-
Beyond LLMs: A Linguistic Approach to Causal Graph Generation from Narrative Texts 10 Apr 2025 · 0 repositories · arXiv:2504.07459
-
Exploring Generative AI Techniques in Government: A Case Study 6 Apr 2025 · 0 repositories · arXiv:2504.10497
-
A thorough benchmark of automatic text classification: From traditional approaches to large language models 2 Apr 2025 · 1 repository · arXiv:2504.01930
-
Context-Aware Toxicity Detection in Multiplayer Games: Integrating Domain-Adaptive Pretraining and Match Metadata 2 Apr 2025 · 1 repository · arXiv:2504.01534
-
Transformer-Based Named Entity Recognition for Automated Server Provisioning 1 Apr 2025 · 1 repository
-
Measuring Online Hate on 4chan using Pre-trained Deep Learning Models 30 Mar 2025 · 0 repositories · arXiv:2504.00045
-
Enhancing Recommender Systems Using Textual Embeddings from Pre-trained Language Models 24 Mar 2025 · 0 repositories · arXiv:2504.08746
-
LakotaBERT: A Transformer-based Model for Low Resource Lakota Language 23 Mar 2025 · 0 repositories · arXiv:2503.18212
-
Predicting Human Choice Between Textually Described Lotteries 18 Mar 2025 · 0 repositories · arXiv:2503.14004
-
An Evaluation of LLMs for Detecting Harmful Computing Terms 12 Mar 2025 · 0 repositories · arXiv:2503.09341
-
Multimodal Emotion Recognition and Sentiment Analysis in Multi-Party Conversation Contexts 9 Mar 2025 · 0 repositories · arXiv:2503.06805
-
Seeing Delta Parameters as JPEG Images: Data-Free Delta Compression with Discrete Cosine Transform 9 Mar 2025 · 0 repositories · arXiv:2503.06676
-
Constructions are Revealed in Word Distributions 8 Mar 2025 · 1 repository · arXiv:2503.06048
-
Evaluating open-source Large Language Models for automated fact-checking 7 Mar 2025 · 0 repositories · arXiv:2503.05565
-
Intermediate-Task Transfer Learning: Leveraging Sarcasm Detection for Stance Detection 5 Mar 2025 · 0 repositories · arXiv:2503.03172
Tasks archive 2025-07-28
20 shown of 477 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections