Methods › Natural Language Processing › Language Models › Bort
Bort
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Bort is a parametric architectural variant of the BERT architecture. It extracts an optimal subset of architectural parameters for the BERT architecture through a neural architecture search approach; in particular, a fully polynomial-time approximation scheme (FPTAS). This optimal subset - “Bort” - is demonstrably smaller, having an effective size of 5.5 % the original BERT-large architecture, and 16% of the net size. Bort is also able to be pretrained in $288$ GPU hours, which is 1.2% less than the time required to pretrain the highest-performing BERT parametric architecture variant, RoBERTa-large (RoBERTa), and about $33\%
Papers archive 2025-07-28
5 shown of 5, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Bort: Towards Explainable Neural Networks with Bounded Orthogonal Constraint 18 Dec 2022 · 1 repository · arXiv:2212.09062Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
-
BORT: Back and Denoising Reconstruction for End-to-End Task-Oriented Dialog 5 May 2022 · 1 repository · arXiv:2205.02471
-
RobBERTje: a Distilled Dutch BERT Model 28 Apr 2022 · 0 repositories · arXiv:2204.13511
-
BORT: Back and Denoising Reconstruction for End-to-End Task-Oriented Dialog 16 Nov 2021 · 0 repositories
-
Optimal Subarchitecture Extraction For BERT 20 Oct 2020 · 3 repositories · arXiv:2010.10499
Tasks archive 2025-07-28
7 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Denoising | 2 |
| CPU | 1 |
| GPU | 1 |
| Lightweight Deployment | 1 |
| Natural Language Understanding | 1 |
| Neural Architecture Search | 1 |
| model | 1 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections