Methods › Natural Language Processing › Transformers › ALBERT
ALBERT
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
ALBERT is a Transformer architecture based on BERT but with much fewer parameters. It achieves this through two parameter reduction techniques. The first is a factorized embeddings parameterization. By decomposing the large vocabulary embedding matrix into two small matrices, the size of the hidden layers is separated from the size of vocabulary embedding. This makes it easier to grow the hidden size without significantly increasing the parameter size of the vocabulary embeddings. The second technique is cross-layer parameter sharing. This technique prevents the parameter from growing with the depth of the network.
Additionally, ALBERT utilises a self-supervised loss for sentence-order prediction (SOP). SOP primary focuses on inter-sentence coherence and is designed to address the ineffectiveness of the next sentence prediction (NSP) loss proposed in the original BERT.
In Syntology repository known, no verified implementation (declared_code; repositories: 5 listed of 48 in the graph, strongest link: official). The archive credits no paper with an arXiv id for this method, so whether this answer (about arXiv:1909.11942) is about this method is not checked.
Origin paper of this answer, arXiv:1909.11942 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 81 ran (of which 17 constructed an object rather than computing a result; 56 with no instrument failure: 4 honoured, 0 violated, 52 with no contract checked; 25 where Syntology's instrument failed) · 45 unverified (of 126 harvested samples) · 28 pointer-only (licence). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 172 · with a code link: 68 · where Syntology ran a sample: 14 (8 with a run with no instrument failure, 6 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (14 of 172 tagged: 8 with a run with no instrument failure, 6 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 172, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
ALBERT: Advanced Localization and Bidirectional Encoder Representations from Transformers for Automotive Damage Evaluation 12 Jun 2025 · 0 repositories · arXiv:2506.10524
-
Rapid yet accurate Tile-circuit and device modeling for Analog In-Memory Computing 5 May 2025 · 0 repositories · arXiv:2506.00004
-
Don't Fight Hallucinations, Use Them: Estimating Image Realism using NLI over Atomic Facts 20 Mar 2025 · 1 repository · arXiv:2503.15948
-
Efficient or Powerful? Trade-offs Between Machine Learning and Deep Learning for Mental Illness Detection on Social Media 3 Mar 2025 · 0 repositories · arXiv:2503.01082
-
Robust Bias Detection in MLMs and its Application to Human Trait Ratings 21 Feb 2025 · 1 repository · arXiv:2502.15600
-
Meursault as a Data Point 3 Feb 2025 · 0 repositories · arXiv:2502.01364
-
Aligning Brain Activity with Advanced Transformer Models: Exploring the Role of Punctuation in Semantic Processing 10 Jan 2025 · 1 repository · arXiv:2501.06278
-
A Comparative Analysis of Transformer and LSTM Models for Detecting Suicidal Ideation on Reddit 23 Nov 2024 · 1 repository · arXiv:2411.15404
-
BERT-Based Approach for Automating Course Articulation Matrix Construction with Explainable AI 21 Nov 2024 · 1 repository · arXiv:2411.14254
-
ProTransformer: Robustify Transformers via Plug-and-Play Paradigm 30 Oct 2024 · 1 repository · arXiv:2410.23182
-
A Bayesian Perspective on the Maximum Score Problem 22 Oct 2024 · 0 repositories · arXiv:2410.17153
-
Meta-RTL: Reinforcement-Based Meta-Transfer Learning for Low-Resource Commonsense Reasoning 27 Sep 2024 · 0 repositories · arXiv:2409.19075
-
Profiling Patient Transcript Using Large Language Model Reasoning Augmentation for Alzheimer's Disease Detection 19 Sep 2024 · 1 repository · arXiv:2409.12541
-
BioMNER: A Dataset for Biomedical Method Entity Recognition 28 Jun 2024 · 0 repositories · arXiv:2406.20038
-
Concept Formation and Alignment in Language Models: Bridging Statistical Patterns in Latent Space to Concept Taxonomy 8 Jun 2024 · 0 repositories · arXiv:2406.05315
-
Effect of antibody levels on the spread of disease in multiple infections 31 May 2024 · 0 repositories · arXiv:2405.20702
-
CEEBERT: Cross-Domain Inference in Early Exit BERT 23 May 2024 · 1 repository · arXiv:2405.15039Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
A Named Entity Recognition and Topic Modeling-based Solution for Locating and Better Assessment of Natural Disasters in Social Media 1 May 2024 · 0 repositories · arXiv:2405.00903
-
Exploring Internal Numeracy in Language Models: A Case Study on ALBERT 25 Apr 2024 · 0 repositories · arXiv:2404.16574
-
Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge 20 Apr 2024 · 1 repository · arXiv:2404.13292
-
mALBERT: Is a Compact Multilingual BERT Model Still Worth It? 27 Mar 2024 · 0 repositories · arXiv:2403.18338
-
An Exploratory Study on Automatic Identification of Assumptions in the Development of Deep Learning Frameworks 8 Jan 2024 · 1 repository · arXiv:2401.03653
-
Tensor-Aware Energy Accounting 19 Nov 2023 · 1 repository · arXiv:2311.11424
-
Generative AI for Hate Speech Detection: Evaluation and Findings 16 Nov 2023 · 0 repositories · arXiv:2311.09993
-
AMPLIFY:Attention-based Mixup for Performance Improvement and Label Smoothing in Transformer 22 Sep 2023 · 1 repository · arXiv:2309.12689
-
Identification of the Relevance of Comments in Codes Using Bag of Words and Transformer Based Models 11 Aug 2023 · 1 repository · arXiv:2308.06144
-
Performance Analysis of Transformer Based Models (BERT, ALBERT and RoBERTa) in Fake News Detection 9 Aug 2023 · 1 repository · arXiv:2308.04950
-
Gradient-Based Word Substitution for Obstinate Adversarial Examples Generation in Language Models 24 Jul 2023 · 0 repositories · arXiv:2307.12507
-
SparseOptimizer: Sparsify Language Models through Moreau-Yosida Regularization and Accelerate via Compiler Co-design 27 Jun 2023 · 0 repositories · arXiv:2306.15656
-
F-PABEE: Flexible-patience-based Early Exiting for Single-label and Multi-label text Classification Tasks 21 May 2023 · 0 repositories · arXiv:2305.11916
-
Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learning 29 Mar 2023 · 1 repository · arXiv:2303.16445
-
Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures 25 Mar 2023 · 0 repositories · arXiv:2303.16100
-
Elementwise Language Representation 27 Feb 2023 · 0 repositories · arXiv:2302.13475
-
A Stability Analysis of Fine-Tuning a Pre-Trained Model 24 Jan 2023 · 0 repositories · arXiv:2301.09820
-
Floods Relevancy and Identification of Location from Twitter Posts using NLP Techniques 1 Jan 2023 · 0 repositories · arXiv:2301.00321
-
Learning-To-Embed: Adopting Transformer based models for E-commerce Products Representation Learning 7 Dec 2022 · 0 repositories · arXiv:2212.03725
-
Entity-Assisted Language Models for Identifying Check-worthy Sentences 19 Nov 2022 · 0 repositories · arXiv:2211.10678
-
Dark patterns in e-commerce: a dataset and its baseline evaluations 12 Nov 2022 · 1 repository · arXiv:2211.06543
-
The Architectural Bottleneck Principle 11 Nov 2022 · 0 repositories · arXiv:2211.06420
-
Collateral facilitation in humans and language models 9 Nov 2022 · 1 repository · arXiv:2211.05198Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Vote'n'Rank: Revision of Benchmarking with Social Choice Theory 11 Oct 2022 · 1 repository · arXiv:2210.05769Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 11 unverified (of 17 harvested samples)
-
ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs 6 Oct 2022 · 1 repository · arXiv:2210.03052
-
DECK: Behavioral Tests to Improve Interpretability and Generalizability of BERT Models Detecting Depression from Text 12 Sep 2022 · 0 repositories · arXiv:2209.05286
-
Addressing Token Uniformity in Transformers via Singular Value Transformation 24 Aug 2022 · 1 repository · arXiv:2208.11790Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
A Comparative Study on COVID-19 Fake News Detection Using Different Transformer Based Models 2 Aug 2022 · 0 repositories · arXiv:2208.01355
-
KnowDA: All-in-One Knowledge Mixture Model for Data Augmentation in Low-Resource NLP 21 Jun 2022 · 0 repositories · arXiv:2206.10265
-
Emoji-based Fine-grained Attention Network for Sentiment Analysis in the Microblog Comments 10 Jun 2022 · 0 repositories · arXiv:2206.12262
-
Life after BERT: What do Other Muppets Understand about Language? 21 May 2022 · 1 repository · arXiv:2205.10696
-
ALBETO and DistilBETO: Lightweight Spanish Language Models 19 Apr 2022 · 2 repositories · arXiv:2204.09145
-
Accelerating Attention through Gradient-Based Learned Runtime Pruning 7 Apr 2022 · 0 repositories · arXiv:2204.03227
-
PALBERT: Teaching ALBERT to Ponder 7 Apr 2022 · 1 repository · arXiv:2204.03276Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; the one sample that ran constructed an object rather than computing a result (of 2 harvested samples)
-
Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guiding 6 Apr 2022 · 0 repositories · arXiv:2204.02922
-
ANNA: Enhanced Language Representation for Question Answering 28 Mar 2022 · 0 repositories · arXiv:2203.14507
-
Mono vs Multilingual BERT: A Case Study in Hindi and Marathi Named Entity Recognition 24 Mar 2022 · 0 repositories · arXiv:2203.12907
-
Detecting and Diagnosing Terrestrial Gravitational-Wave Mimics Through Feature Learning 9 Mar 2022 · 0 repositories · arXiv:2203.05086
-
Does Transliteration Help Multilingual Language Modeling? 29 Jan 2022 · 1 repository · arXiv:2201.12501Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
An Application of Pseudo-Log-Likelihoods to Natural Language Scoring 23 Jan 2022 · 0 repositories · arXiv:2201.09377
-
Bridge the Gap Between CV and NLP! A Gradient-based Textual Adversarial Attack Framework 16 Jan 2022 · 0 repositories
-
DECK: Behavioral Tests to Improve Interpretability and Generalizability of BERT Models Detecting Depression from Text 16 Jan 2022 · 0 repositories
-
Multi-Stage Pre-Training for Math-Understanding: μ²(AL)BERT 16 Jan 2022 · 0 repositories
-
PCEE-BERT: Accelerating BERT Inference via Patient and Confident Early Exiting 16 Jan 2022 · 0 repositories
-
A Comparative Study of Transformers on Word Sense Disambiguation 30 Nov 2021 · 0 repositories · arXiv:2111.15417
-
How Emotionally Stable is ALBERT? Testing Robustness with Stochastic Weight Averaging on a Sentiment Analysis Task 18 Nov 2021 · 1 repository · arXiv:2111.09612
-
ANNA: Enhanced Language Representation for Question Answering 16 Nov 2021 · 0 repositories
-
Life after BERT: What do Other Muppets Understand about Language? 16 Nov 2021 · 0 repositories
-
PALBERT: Teaching ALBERT to Ponder 16 Nov 2021 · 0 repositories
-
Understanding Attention in Machine Reading Comprehension 16 Nov 2021 · 0 repositories
-
Character-level HyperNetworks for Hate Speech Detection 11 Nov 2021 · 1 repository · arXiv:2111.06336
-
The Golden Rule as a Heuristic to Measure the Fairness of Texts Using Machine Learning 29 Oct 2021 · 0 repositories · arXiv:2111.00107
-
Bridge the Gap Between CV and NLP! A Gradient-based Textual Adversarial Attack Framework 28 Oct 2021 · 1 repository · arXiv:2110.15317
-
Ensemble ALBERT on SQuAD 2.0 19 Oct 2021 · 1 repository · arXiv:2110.09665
-
Local and Global Context-Based Pairwise Models for Sentence Ordering 8 Oct 2021 · 1 repository · arXiv:2110.04291
-
Transliteration: A Simple Technique For Improving Multilingual Language Modeling 29 Sep 2021 · 0 repositories
-
Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers 17 Sep 2021 · 0 repositories · arXiv:2109.08406
-
Transformer-based Language Models for Factoid Question Answering at BioASQ9b 15 Sep 2021 · 1 repository · arXiv:2109.07185
-
So Cloze yet so Far: N400 Amplitude is Better Predicted by Distributional Information than Human Predictability Judgements 2 Sep 2021 · 0 repositories · arXiv:2109.01226
-
Fight Fire with Fire: Fine-tuning Hate Detectors using Large Samples of Generated Hate Speech 1 Sep 2021 · 0 repositories · arXiv:2109.00591
-
Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models 26 Aug 2021 · 1 repository · arXiv:2108.11574
-
BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks 1 Aug 2021 · 1 repository
-
DeepBlueAI at SemEval-2021 Task 7: Detecting and Rating Humor and Offense with Stacking Diverse Language Model-Based Methods 1 Aug 2021 · 0 repositories
-
Gulu at SemEval-2021 Task 7: Detecting and Rating Humor and Offense 1 Aug 2021 · 0 repositories
-
How Many Layers and Why? An Analysis of the Model Depth in Transformers 1 Aug 2021 · 0 repositories
-
hub at SemEval-2021 Task 7: Fusion of ALBERT and Word Frequency Information Detecting and Rating Humor and Offense 1 Aug 2021 · 0 repositories
-
YNU-HPCC at SemEval-2021 Task 5: Using a Transformer-based Model with Auxiliary Information for Toxic Span Detection 1 Aug 2021 · 1 repository
-
YNU-HPCC at SemEval-2021 Task 6: Combining ALBERT and Text-CNN for Persuasion Detection in Texts and Images 1 Aug 2021 · 0 repositories
-
ZYJ at SemEval-2021 Task 7: HaHackathon: Detecting and Rating Humor and Offense with ALBERT-Based Model 1 Aug 2021 · 0 repositories
-
Go Wider Instead of Deeper 25 Jul 2021 · 1 repository · arXiv:2107.11817
-
Digital Einstein Experience: Fast Text-to-Speech for Conversational AI 21 Jul 2021 · 0 repositories · arXiv:2107.10658
-
Elbert: Fast Albert with Confidence-Window Based Early Exit 1 Jul 2021 · 0 repositories · arXiv:2107.00175
-
Benchmarking Differential Privacy and Federated Learning for BERT Models 26 Jun 2021 · 1 repository · arXiv:2106.13973
-
Distributed Deep Learning in Open Collaborations 18 Jun 2021 · 2 repositories · arXiv:2106.10207Syntology official: harvested, nothing ran · 0 ran · 5 unverified (of 5 harvested samples)
-
CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark 15 Jun 2021 · 2 repositories · arXiv:2106.08087Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 1 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 5 unverified (of 16 harvested samples) · 2 pointer-only (licence)
-
Why Can You Lay Off Heads? Investigating How BERT Heads Transfer 14 Jun 2021 · 0 repositories · arXiv:2106.07137
-
A Semi-supervised Multi-task Learning Approach to Classify Customer Contact Intents 10 Jun 2021 · 0 repositories · arXiv:2106.07381
-
Transient Chaos in BERT 6 Jun 2021 · 0 repositories · arXiv:2106.03181
-
BERT-Based Sentiment Analysis: A Software Engineering Perspective 4 Jun 2021 · 2 repositories · arXiv:2106.02581
-
The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models 3 Jun 2021 · 3 repositories · arXiv:2106.01950Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample) · 1 pointer-only (licence)
-
iCompass at NLP4IF-2021–Fighting the COVID-19 Infodemic 1 Jun 2021 · 1 repository
-
Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers 1 Jun 2021 · 0 repositories
-
Neural Models for Offensive Language Detection 30 May 2021 · 0 repositories · arXiv:2106.14609
Tasks archive 2025-07-28
20 shown of 167 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections