Methods › Natural Language Processing › Transformers › XLNet
XLNet
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
XLNet is an autoregressive Transformer that leverages the best of both autoregressive language modeling and autoencoding while attempting to avoid their limitations. Instead of using a fixed forward or backward factorization order as in conventional autoregressive models, XLNet maximizes the expected log likelihood of a sequence w.r.t. all possible permutations of the factorization order. Thanks to the permutation operation, the context for each position can consist of tokens from both left and right. In expectation, each position learns to utilize contextual information from all positions, i.e., capturing bidirectional context.
Additionally, inspired by the latest advancements in autogressive language modeling, XLNet integrates the segment recurrence mechanism and relative encoding scheme of Transformer-XL into pretraining, which empirically improves the performance especially for tasks involving a longer text sequence.
In Syntology repository known, no verified implementation (declared_code; repositories: 5 listed of 26 in the graph, strongest link: official). The archive credits no paper with an arXiv id for this method, so whether this answer (about arXiv:1906.08237) is about this method is not checked.
Origin paper of this answer, arXiv:1906.08237 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 15 ran (of which 2 constructed an object rather than computing a result; 10 with no instrument failure: 2 honoured, 0 violated, 8 with no contract checked; 5 where Syntology's instrument failed) · 9 unverified (of 24 harvested samples) · 4 pointer-only (licence). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 167 · with a code link: 70 · where Syntology ran a sample: 10 (10 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (10 of 167 tagged: 10 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 167, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Comparative sentiment analysis of public perception: Monkeypox vs. COVID-19 behavioral insights 12 May 2025 · 0 repositories · arXiv:2505.07430
-
A Character-based Diffusion Embedding Algorithm for Enhancing the Generation Quality of Generative Linguistic Steganographic Texts 2 May 2025 · 0 repositories · arXiv:2505.00977
-
Explainable AI for Sentiment Analysis of Human Metapneumovirus (HMPV) Using XLNet 1 Feb 2025 · 0 repositories · arXiv:2502.01663
-
Assessing Text Classification Methods for Cyberbullying Detection on Social Media Platforms 27 Dec 2024 · 0 repositories · arXiv:2412.19928
-
Feature Alignment-Based Knowledge Distillation for Efficient Compression of Large Language Models 27 Dec 2024 · 0 repositories · arXiv:2412.19449
-
Enhancing Multi-Class Disease Classification: Neoplasms, Cardiovascular, Nervous System, and Digestive Disorders Using Advanced LLMs 19 Nov 2024 · 0 repositories · arXiv:2411.12712
-
Rethinking Legal Judgement Prediction in a Realistic Scenario in the Era of Large Language Models 14 Oct 2024 · 1 repository · arXiv:2410.10542
-
What Matters in Explanations: Towards Explainable Fake Review Detection Focusing on Transformers 24 Jul 2024 · 0 repositories · arXiv:2407.21056
-
Why do you cite? An investigation on citation intents and decision-making classification processes 18 Jul 2024 · 0 repositories · arXiv:2407.13329
-
Ensemble Model With Bert,Roberta and Xlnet For Molecular property prediction 30 May 2024 · 0 repositories · arXiv:2406.06553
-
A Hybrid Deep Learning Framework for Stock Price Prediction Considering the Investor Sentiment of Online Forum Enhanced by Popularity 17 May 2024 · 0 repositories · arXiv:2405.10584
-
Eliciting Personality Traits in Large Language Models 13 Feb 2024 · 0 repositories · arXiv:2402.08341
-
Breaking Free Transformer Models: Task-specific Context Attribution Promises Improved Generalizability Without Fine-tuning Pre-trained LLMs 30 Jan 2024 · 1 repository · arXiv:2401.16638
-
An Assessment on Comprehending Mental Health through Large Language Models 9 Jan 2024 · 0 repositories · arXiv:2401.04592
-
Argumentation Element Annotation Modeling using XLNet 10 Nov 2023 · 0 repositories · arXiv:2311.06239
-
Modelling Sentiment Analysis: LLMs and data augmentation techniques 7 Nov 2023 · 1 repository · arXiv:2311.04139
-
An Ensemble Method Based on the Combination of Transformers with Convolutional Neural Networks to Detect Artificially Generated Text 26 Oct 2023 · 0 repositories · arXiv:2310.17312
-
Exploring Graph Neural Networks for Indian Legal Judgment Prediction 19 Oct 2023 · 0 repositories · arXiv:2310.12800
-
Lexical Squad@Multimodal Hate Speech Event Detection 2023: Multimodal Hate Speech Detection using Fused Ensemble Approach 23 Sep 2023 · 1 repository · arXiv:2309.13354
-
Simple is Better and Large is Not Enough: Towards Ensembling of Foundational Language Models 23 Aug 2023 · 0 repositories · arXiv:2308.12272
-
A Hybrid Machine Learning Model for Classifying Gene Mutations in Cancer using LSTM, BiLSTM, CNN, GRU, and GloVe 24 Jul 2023 · 0 repositories · arXiv:2307.14361
-
Resume Information Extraction via Post-OCR Text Processing 23 Jun 2023 · 0 repositories · arXiv:2306.13775
-
Research on Named Entity Recognition in Improved transformer with R-Drop structure 14 Jun 2023 · 0 repositories · arXiv:2306.08315
-
Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Health 20 Apr 2023 · 0 repositories · arXiv:2304.10447
-
Deep Learning for Opinion Mining and Topic Classification of Course Reviews 6 Apr 2023 · 0 repositories · arXiv:2304.03394
-
TrojText: Test-time Invisible Textual Trojan Insertion 3 Mar 2023 · 1 repository · arXiv:2303.02242Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
HULAT at SemEval-2023 Task 10: Data augmentation for pre-trained transformers applied to the detection of sexism in social media 24 Feb 2023 · 1 repository · arXiv:2302.12840
-
A benchmark for toxic comment classification on Civil Comments dataset 26 Jan 2023 · 1 repository · arXiv:2301.11125
-
Analyzing Semantic Faithfulness of Language Models via Input Intervention on Question Answering 21 Dec 2022 · 1 repository · arXiv:2212.10696
-
Learning-To-Embed: Adopting Transformer based models for E-commerce Products Representation Learning 7 Dec 2022 · 0 repositories · arXiv:2212.03725
-
Dark patterns in e-commerce: a dataset and its baseline evaluations 12 Nov 2022 · 1 repository · arXiv:2211.06543
-
Probing for targeted syntactic knowledge through grammatical error detection 28 Oct 2022 · 1 repository · arXiv:2210.16228
-
The (In)Effectiveness of Intermediate Task Training For Domain Adaptation and Cross-Lingual Transfer Learning 3 Oct 2022 · 0 repositories · arXiv:2210.01091
-
Can Transformer Models Effectively Detect Software Aspects in StackOverflow Discussion? 24 Sep 2022 · 0 repositories · arXiv:2209.12065
-
Adaptation of domain-specific transformer models with text oversampling for sentiment analysis of social media posts on Covid-19 vaccines 22 Sep 2022 · 1 repository · arXiv:2209.10966
-
Deep Learning Reveals Patterns of Diverse and Changing Sentiments Towards COVID-19 Vaccines Based on 11 Million Tweets 5 Jul 2022 · 0 repositories · arXiv:2207.10641
-
Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution 7 Jun 2022 · 1 repository · arXiv:2206.11815
-
Training and Inference on Any-Order Autoregressive Models the Right Way 26 May 2022 · 1 repository · arXiv:2205.13554
-
TangoBERT: Reducing Inference Cost by using Cascaded Architecture 13 Apr 2022 · 0 repositories · arXiv:2204.06271
-
ANNA: Enhanced Language Representation for Question Answering 28 Mar 2022 · 0 repositories · arXiv:2203.14507
-
SciNLI: A Corpus for Natural Language Inference on Scientific Text 13 Mar 2022 · 1 repository · arXiv:2203.06728
-
Regression Transformer: Concurrent sequence regression and generation for molecular language modeling 1 Feb 2022 · 1 repository · arXiv:2202.01338Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 8 harvested samples)
-
Named entity recognition architecture combining contextual and global features 15 Dec 2021 · 1 repository · arXiv:2112.08033
-
A Comparative Study of Transformers on Word Sense Disambiguation 30 Nov 2021 · 0 repositories · arXiv:2111.15417
-
ANNA: Enhanced Language Representation for Question Answering 16 Nov 2021 · 0 repositories
-
Interpreting the Robustness of Neural NLP Models to Textual Perturbations 16 Nov 2021 · 0 repositories
-
IBERT: Idiom Cloze-style reading comprehension with Attention 5 Nov 2021 · 0 repositories · arXiv:2112.02994
-
Backdoor Pre-trained Models Can Transfer to All 30 Oct 2021 · 1 repository · arXiv:2111.00197
-
Interpreting the Robustness of Neural NLP Models to Textual Perturbations 14 Oct 2021 · 0 repositories · arXiv:2110.07159
-
Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models 1 Oct 2021 · 0 repositories · arXiv:2110.00672
-
What to Prioritize? Natural Language Processing for the Development of a Modern Bug Tracking Solution in Hardware Development 28 Sep 2021 · 0 repositories · arXiv:2109.13825
-
Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representations 13 Sep 2021 · 0 repositories · arXiv:2109.05958
-
Automated Mining of Leaderboards for Empirical AI Research 31 Aug 2021 · 1 repository · arXiv:2109.13089
-
Evaluating the Robustness of Neural Language Models to Input Perturbations 27 Aug 2021 · 1 repository · arXiv:2108.12237Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Misleading the Covid-19 vaccination discourse on Twitter: An exploratory study of infodemic around the pandemic 16 Aug 2021 · 1 repository · arXiv:2108.10735
-
Variable-Length Music Score Infilling via XLNet and Musically Specialized Positional Encoding 11 Aug 2021 · 1 repository · arXiv:2108.05064
-
YoungSheldon at SemEval-2021 Task 5: Fine-tuning Pre-trained Language Models for Toxic Spans Detection using Token classification Objective 1 Aug 2021 · 1 repository
-
Clinical Relation Extraction Using Transformer-based Models 19 Jul 2021 · 1 repository · arXiv:2107.08957
-
Transformers with multi-modal features and post-fusion context for e-commerce session-based recommendation 11 Jul 2021 · 0 repositories · arXiv:2107.05124
-
What Helps Transformers Recognize Conversational Structure? Importance of Context, Punctuation, and Labels in Dialog Act Recognition 5 Jul 2021 · 1 repository · arXiv:2107.02294
-
Incorporating Word Sense Disambiguation in Neural Language Models 15 Jun 2021 · 2 repositories · arXiv:2106.07967
-
Target Model Agnostic Adversarial Attacks with Query Budgets on Language Understanding Models 13 Jun 2021 · 0 repositories · arXiv:2106.07047
-
Neural Combinatory Constituency Parsing 12 Jun 2021 · 1 repository · arXiv:2106.06689
-
Contextualized and Generalized Sentence Representations by Contrastive Self-Supervised Learning: A Case Study on Discourse Relation Analysis 1 Jun 2021 · 0 repositories
-
Multi-Grained Knowledge Distillation for Named Entity Recognition 1 Jun 2021 · 0 repositories
-
Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers 1 Jun 2021 · 0 repositories
-
How transfer learning impacts linguistic knowledge in deep NLP models? 31 May 2021 · 0 repositories · arXiv:2105.15179
-
Killing One Bird with Two Stones: Model Extraction and Attribute Inference Attacks against BERT-based APIs 23 May 2021 · 0 repositories · arXiv:2105.10909
-
Explainable Tsetlin Machine framework for fake news detection with credibility score assessment 19 May 2021 · 6 repositories · arXiv:2105.09114
-
BERT Busters: Outlier Dimensions that Disrupt Transformers 14 May 2021 · 0 repositories · arXiv:2105.06990
-
Transformers: "The End of History" for NLP? 9 Apr 2021 · 0 repositories · arXiv:2105.00813
-
Exploring Transformers in Emotion Recognition: a comparison of BERT, DistillBERT, RoBERTa, XLNet and ELECTRA 5 Apr 2021 · 0 repositories · arXiv:2104.02041
-
Through the Looking Glass: Learning to Attribute Synthetic Text Generated by Language Models 1 Apr 2021 · 0 repositories
-
K-XLNet: A General Method for Combining Explicit Knowledge with Language Model Pretraining 25 Mar 2021 · 0 repositories · arXiv:2104.10649
-
Comparing the Performance of NLP Toolkits and Evaluation measures in Legal Tech 12 Mar 2021 · 0 repositories · arXiv:2103.11792
-
Orthogonal Attention: A Cloze-Style Approach to Negation Scope Resolution 7 Mar 2021 · 0 repositories · arXiv:2103.04294
-
NLP-CUET@DravidianLangTech-EACL2021: Investigating Visual and Textual Features to Identify Trolls from Multimodal Social Media Memes 28 Feb 2021 · 0 repositories · arXiv:2103.00466
-
NLP-CUET@LT-EDI-EACL2021: Multilingual Code-Mixed Hope Speech Detection using Cross-lingual Representation Learner 28 Feb 2021 · 1 repository · arXiv:2103.00464
-
Few Shot Learning for Information Verification 22 Feb 2021 · 0 repositories · arXiv:2102.10956
-
Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet 16 Feb 2021 · 1 repository · arXiv:2102.08036
-
Fake News Detection System using XLNet model with Topic Distributions: CONSTRAINT@AAAI2021 Shared Task 12 Jan 2021 · 0 repositories · arXiv:2101.11425
-
Cisco at AAAI-CAD21 shared task: Predicting Emphasis in Presentation Slides using Contextualized Embeddings 10 Jan 2021 · 1 repository · arXiv:2101.11422
-
Transformer-based approach towards music emotion recognition from lyrics 6 Jan 2021 · 1 repository · arXiv:2101.02051
-
Syntactic Relevance XLNet Word Embedding Generation in Low-Resource Machine Translation 1 Jan 2021 · 0 repositories
-
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets 31 Dec 2020 · 1 repository · arXiv:2101.00063Syntology official (archive's flag): 1 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
DialogXL: All-in-One XLNet for Multi-Party Conversation Emotion Recognition 16 Dec 2020 · 4 repositories · arXiv:2012.08695
-
Discriminative Pre-training for Low Resource Title Compression in Conversational Grocery 13 Dec 2020 · 0 repositories · arXiv:2012.06943
-
Yelp Review Rating Prediction: Machine Learning and Deep Learning Models 12 Dec 2020 · 1 repository · arXiv:2012.06690
-
BERT at SemEval-2020 Task 8: Using BERT to Analyse Meme Emotions 1 Dec 2020 · 0 repositories
-
Comparing Probabilistic, Distributional and Transformer-Based Models on Logical Metonymy Interpretation 1 Dec 2020 · 0 repositories
-
Evaluating Unsupervised Representation Learning for Detecting Stances of Fake News 1 Dec 2020 · 0 repositories
-
Hitachi at SemEval-2020 Task 11: An Empirical Study of Pre-Trained Transformer Family for Propaganda Detection 1 Dec 2020 · 0 repositories
-
Hy-NLI: a Hybrid system for Natural Language Inference 1 Dec 2020 · 2 repositories
-
Neural language models for text classification in evidence-based medicine 1 Dec 2020 · 0 repositories · arXiv:2012.00584
-
NLP@JUST at SemEval-2020 Task 4: Ensemble Technique for BERT and Roberta to Evaluate Commonsense Validation 1 Dec 2020 · 0 repositories
-
SINAI at SemEval-2020 Task 12: Offensive Language Identification Exploring Transfer Learning Models 1 Dec 2020 · 0 repositories
-
SSN-NLP at SemEval-2020 Task 4: Text Classification and Generation on Common Sense Context Using Neural Networks 1 Dec 2020 · 0 repositories
-
Multitask Learning of Negation and Speculation using Transformers 20 Nov 2020 · 1 repository
-
BioNerFlair: biomedical named entity recognition using flair embedding and sequence tagger 3 Nov 2020 · 1 repository · arXiv:2011.01504
-
CHIME: Cross-passage Hierarchical Memory Network for Generative Review Question Answering 1 Nov 2020 · 1 repository · arXiv:2011.00519
Tasks archive 2025-07-28
20 shown of 190 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections