Papers › ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut
Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at https://github.com/google-research/ALBERT.
In Syntology View this paper on Syntology, its page in Syntology's graph. That page adds which of its functions are also present in other papers' code Syntology holds, with dates, and its dated run record as JSON and a README badge. It also lists the repositories linked to the paper, the abstract and the calls for agents.
Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
For agents, on Syntology's MCP service (how to connect):
get_harvested_code_for_paper(arxiv_id="1909.11942")get_code(code_sha256="…")with a sample'scode_sha256from the list belowget_citation_path(paper_1="1909.11942", paper_2="…")with another paper's arXiv id or titleget_concepts_for_paper(arxiv_id="1909.11942")
Code
Syntology 126 code samples harvested from 18 repositories linked to this paper: 81 ran (of which 17 constructed an object rather than computing a result; 56 with no instrument failure: 4 honoured, 0 violated, 52 with no contract checked; 25 where Syntology's instrument failed) · 45 unverified. Where Syntology's instrument failed: 21 where our contract draft was wrong; 4 where the contract check could not run; failures of Syntology's instrument, not of the code.
By repository: official repository: 6 samples from 1 repository, 6 ran (4 with no instrument failure; 2 where Syntology's instrument failed); community (archive-listed): 120 samples from 17 repositories, 75 ran (52 with no instrument failure; 23 where Syntology's instrument failed). The run record, sample by sample. “Ran” means executed on a synthesized input, not that the code is correct or reproduces the paper.
No sample is linked to this paper from 9 of its repositories that Syntology harvested for other papers:
- KnightZhang625/BERT_TF, community (archive-listed): harvested for another paper, listed under the repository
- Soikonomou/albert_final_infer12, community (archive-listed): harvested for another paper, listed under the repository
- appcoreopc/berty, community (archive-listed): harvested for 2 other papers, listed under the repository
- huggingface/transformers, community (archive-listed): harvested for 4 other papers, listed under the repository
- lucidrains/compressive-transformer-pytorch, community (archive-listed): harvested for another paper, listed under the repository
- lucidrains/routing-transformer, community (archive-listed): harvested for another paper, listed under the repository
- tensorflow/models, community (archive-listed): harvested for 4 other papers, listed under the repository
- PaddlePaddle/PaddleNLP, community (archive-listed): harvested for another paper, listed under the repository
- pwc-1/Paper-8, community (archive-listed): harvested for another paper, listed under the repository
Levels written between 2026-09-22 and 2026-09-27 (UTC): the dates Syntology's graph wrote each sample's verification level, not when the code executed. 1 of the 81 samples that ran has no recorded date.
48 repositories listed; official and paper-mentioned ones first.
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.
Code Syntology ran Syntology
126 samples harvested: 81 ran (of which 17 constructed an object rather than computing a result; 56 with no instrument failure: 4 honoured, 0 violated, 52 with no contract checked; 25 where Syntology's instrument failed) · 45 unverified. “Where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. That date is not when the samples ran.
Licence: 28 of the 126 samples are pointer only, meaning Syntology does not serve that copy's text. For 6 of them, this paper's copy is under a permissive licence and cleared, but the stored copy of the same code that Syntology serves is not; the licence label's tooltip gives Syntology's recorded reason. This page shows no code text for any sample; each one links to its file in the repository.
Harvested from 18 repositories linked to this paper, official or community; the list below is grouped by repository, each group headed by its name and what it is to the paper. “Ran” means the sample executed on a synthesized input. It does not mean the output is correct, and nothing here reproduces the paper's results. A line beginning “fixture:” under a sample that ran gives the arguments Syntology's fixture called the sample with, each as name:type/rank/element kind (? where none was recorded), and the type of what the fixture's run returned, exactly as the harness recorded them; the arguments and the returned type were recorded by two separate runs of the same fixture. No output value is shown, and a returned type says nothing about whether the output was right. “Level recorded” gives the UTC date when Syntology's graph wrote the sample's verification level, not when the code executed; where the level was written before Syntology's graph kept that date, the row says “level recorded, date not recorded”. For 5 samples that ran, the graph records a signature this page does not show: for a class the fixture built without calling, the harness called the object with inputs it made up itself, so the recorded signature is the harness's probe, not the fixture's arguments. “Honoured” and “violated” refer to a contract Syntology drafted from the code itself. “Our draft was wrong” and “the contract check could not run” are failures of Syntology's instrument, not of the code: the sample ran on Syntology's synthesized input, and then the drafted contract did not describe it, or the check itself raised an error (RAISES), lacked a dependency (DEP_MISSING) or ran out of time (TIMEOUT); hover the label for which. A metamorphic tier, where shown, is evidence of the output's shape, not that the sample implements the method.
Each sample ends with its code_sha256. A sample's code_sha256 is Syntology's identity for that exact code. It is printed in full (16 hexadecimal characters for every sample in this build), and get_code(code_sha256="…") takes it exactly as printed. An agent calls it on Syntology's MCP service (how to connect); click an identity to copy that call.
Repository labels, per sample. official repository: The archive marks this repository official for the paper. named in the paper: The archive records that the paper mentions this repository; it is not marked official. community (archive-listed): In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper. found in paper text by Syntology: Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. community: Not in the archive's code links for this paper; a community repository Syntology harvested. Samples are grouped by repository: a repository marked official first, then by how directly the repository is tied to the paper, then in the archive's order; never by how many ran. Licence labels name the repository's licence as recorded at harvest. “Pointer only” means Syntology does not serve that copy's text, for one of six reasons: no licence file was found; the licence was not identified; the licence is recorded as permissive but that copy's record is not marked cleared; the licence is outside the permissive list Syntology serves text under (MIT, Apache-2.0, BSD and similar); this paper's copy was not located, so its licence is not recorded; or this paper's copy is cleared, but the stored copy of the same code that Syntology serves is not. Some licences outside that list permit redistribution, such as WTFPL, and GPL-3.0 under its conditions; they are simply not on the list. Hover a licence label for the reason. File links open the file on GitHub at the default branch, which may have changed since the harvest.
A function or class name links to its page when the name occurs in the harvested code of at least 5 papers; the name is a string, not an identity. A census tag appears when Syntology's census ran implementations with that name on one shared input per bucket: it gives how many were compared, from how many papers, and how many distinct outputs they produced, and names this sample's bucket when this sample was one of them.
5b0793599d029a67 · report
fixture: signature values:list/1/float · returned Feature · level recorded 2026-09-25 b03d45af664a39e3 · report
fixture: signature values:list/1/int · returned Feature · level recorded 2026-09-26 ff6bac8bf3944cbe · report
fixture: signature input_file:list/1/str, seq_length:int/0/int, is_training:bool/0/bool, drop_remainder:bool/0/bool, task_name:str/0/str, use_tpu:bool/0/bool, bsz:int/0/int, multiple:int/0/int · returned function · level recorded 2026-09-25 census: 43 implementations from 74 papers, 6 distinct outputs on one shared input modeling.py ran · our draft was wrong Apache-2.0 (permissive) ·
51d3bc0cfa6415a3 · report
fixture: signature x:SymbolicTensor/2/float · returned SymbolicTensor · level recorded 2026-09-26 217fe96a5945a42d · report
fixture: signature activation_string:str/0/str · returned function · level recorded 2026-09-26 6bf234f32ff01978 · report
fixture: signature tvars:list/1/RefVariable, init_checkpoint:str/0/str, num_of_group:int/0/int · returned tuple · level recorded 2026-09-25 350a1a7ab2f301c8 · report
fixture: returned RecAdam · level recorded 2026-09-22 b8301a2a061e30a2 · report
fixture: signature function:str/0/str, step:int/0/int, k:float/0/float, t0:int/0/int, weight:float/0/float · returned float · level recorded 2026-09-22 census: 43 implementations from 74 papers, 6 distinct outputs on one shared input this sample: bucket 1 of 1, arg 1: rank 2, kind float, dtype float32 src/model/ALBERT/modeling_bert.py ran · honoured contract fingerprinted Apache-2.0 (permissive) here; the stored copy is pointer only ·
211753ba29188d4e · report
fixture: signature x:Tensor/2/float · returned Tensor · level recorded 2026-09-22 census: 3 implementations from 7 papers, 1 distinct output on one shared input this sample: bucket 1 of 1, arg 1: rank 2, kind float, dtype float32 src/model/ALBERT/modeling_bert.py ran · honoured contract fingerprinted Apache-2.0 (permissive) here; the stored copy is pointer only ·
77601724cad03f95 · report
fixture: signature x:Tensor/2/float · returned Tensor · level recorded 2026-09-22 b2be6c932ce41084 · report
fixture: signature identifier:str/0/str, postfix:str/0/str, cdn:bool/0/bool · returned str · level recorded 2026-09-25 4381d1f588a7f42a · report
fixture: signature url_or_filename:str/0/str · returned bool · level recorded 2026-09-26 8cc492d40c50497b · report
fixture: signature layer:callable/0/?, index:Tensor/1/int, dim:int/0/int · returned Conv1D · level recorded 2026-09-25 d30d6c3098df2c42 · report
fixture: signature layer:callable/0/?, index:Tensor/1/int, dim:int/0/int · returned Linear · level recorded 2026-09-22 census: 8 implementations from 8 papers, 4 distinct outputs across 2 buckets, one shared input per bucket src/model/ALBERT/modeling_utils.py ran Apache-2.0 (permissive) ·
8e873fc150477bd1 · report
fixture: returned Tensor · level recorded 2026-09-27 e827d94caab33f59 · report
fixture: signature url:str/0/str, etag:str/0/str · returned str · level recorded 2026-09-24 4f56c9a37e92815f · report
ca4a7d28d683cd33 · report
cf9ffa02a42184af · report
fixture: signature text:str/0/str · returned list · level recorded, date not recorded 0c145728a35789fa · report
fixture: signature model:callable/0/?, config:Config/0/?, tf_checkpoint_path:str/0/str · returned BertModel · level recorded 2026-09-22 659f3761c1dc81fa · report
fixture: signature vocab_file:str/0/str · returned OrderedDict · level recorded 2026-09-24 7183a17dd79327a8 · report
fixture: signature layer:callable/0/?, index:Tensor/1/int, dim:int/0/int · returned Conv1D · level recorded 2026-09-25 07a2e3e996a6dc8e · report
fixture: signature url:str/0/str, etag:str/0/str · returned str · level recorded 2026-09-24 e31093ffc8275cff · report
0afbeb40a45a72b6 · report
fixture: returned MultiHeadBlockSelf · level recorded 2026-09-22 2bf900d0da165894 · report
fixture: signature input_tensor:EagerTensor/2/float, dropout_prob:float/0/float · returned EagerTensor · level recorded 2026-09-22 census: 43 implementations from 74 papers, 6 distinct outputs on one shared input modeling_google.py ran · honoured contract fingerprinted no licence file found · pointer only ·
cb61421413d30b87 · report
fixture: signature x:EagerTensor/2/float · returned EagerTensor · level recorded 2026-09-22 ae9ebb2661d2ddd4 · report
fixture: signature activation_string:str/0/str · returned function · level recorded 2026-09-22 91c075af55caeadc · report
94f549fd1bf6a555 · report
154407ccae03925e · report
f9aab49e4bfb299f · report
babf39a9a9bf5587 · report
b0e3d2a74b1bb3e4 · report
847351b08a6a93c6 · report
058d2e8f50dd0987 · report
8efebf3cb9b8ca4d · report
4e61fb4457fa4a37 · report
e447e4b29a5d024b · report
ff36be852a0ce11b · report
e1935088b1c311cc · report
fixture: signature lm_logits:Tensor/2/float, blanks_logits:Tensor/2/float, masked_for_pred:Tensor/1/int, blank_labels:Tensor/2/float, tokenizer:SimpleTokenizer/0/?, print_:bool/0/bool · returned tuple · level recorded 2026-09-24 ee14d5932438dd30 · report
fixture: signature output:Tensor/2/float, labels:Tensor/2/int, ignore_idx:int/0/int · returned tuple · level recorded 2026-09-24 a1a367cc1b634bfd · report
fixture: signature x:list/1/int, e1_id:int/0/int, e2_id:int/0/int · returned tuple · level recorded 2026-09-24 540b1fdc61fb8b88 · report
fixture: signature sent_:MockSentence/0/? · returned list · level recorded 2026-09-24 bb4b4b5ebab73228 · report
fixture: signature filename:str/0/str · returned dict · level recorded 2026-09-22 67bc2e6f6f0fb28d · report
fixture: signature filename:str/0/str · returned dict · level recorded 2026-09-24 203cc08d9f87f46e · report
fixture: returned tuple · level recorded 2026-09-24 87f6efb51cc0c000 · report
fixture: returned tuple · level recorded 2026-09-24 7b81e295d0e6fff5 · report
fixture: signature text:list/1/str, mode:str/0/str · returned tuple · level recorded 2026-09-24 abe15205f0893ad7 · report
a8f8b3ec011da390 · report
e60b8c4379569b89 · report
6fbd1282310c47ff · report
0556e020c58f45e5 · report
4d87999c53e2b001 · report
b2116e6e872a2a35 · report
fixture: signature model:callable/0/? · returned ModuleList · level recorded 2026-09-22 b6a5b207adc026a6 · report
00437373da100756 · report
fixture: returned Embeddings · level recorded 2026-09-22 census: 5 implementations from 5 papers, 5 distinct outputs across 3 buckets, one shared input per bucket models.py ran · metamorphic tier: deterministic fingerprinted Apache-2.0 (permissive) ·
31b6e6cb2bf23d35 · report
fixture: returned LayerNorm · level recorded 2026-09-22 8734ddba5f76437a · report
fixture: returned MultiHeadedSelfAttention · level recorded 2026-09-22 e952b1d84bed0af6 · report
fixture: returned PositionWiseFeedForward · level recorded 2026-09-22 b11322b4f500cef1 · report
fixture: signature x:Tensor/2/int, seg:Tensor/2/int, mask:Tensor/2/float · returned Tensor · level recorded 2026-09-24 d5e69fc7b0ee7c8c · report
fixture: signature x:Tensor/4/float, n_dims:int/0/int · returned Tensor · level recorded 2026-09-22 a4a5354a3138c8ef · report
fixture: signature x:Tensor/3/float, shape:tuple/1/int · returned Tensor · level recorded 2026-09-22 f92f017fe0d849f2 · report
fixture: signature output:Tensor/2/float, labels:Tensor/2/int, ignore_idx:int/0/int · returned tuple · level recorded 2026-09-24 ee0c581972df0258 · report
fixture: signature x:list/1/int, e1_id:int/0/int, e2_id:int/0/int · returned tuple · level recorded 2026-09-24 6bf6a61197367217 · report
fixture: signature text:list/1/str, mode:str/0/str · returned tuple · level recorded 2026-09-24 27b5913d3a803ad4 · report
1923fc05163d207d · report
fixture: signature text:str/0/str · returned str · level recorded 2026-09-22 6e2a0964af8411ad · report
fixture: signature sp_model:MockSentencePiece/0/?, text:str/0/str, return_unicode:bool/0/bool, sample:bool/0/bool · returned list · level recorded 2026-09-24 30b05f2a415dd132 · report
fixture: signature vocab_file:str/0/str · returned OrderedDict · level recorded 2026-09-25 8a6e1a7c602e8104 · report
fixture: signature name:str/0/str, prefix:str/0/str · returned str · level recorded 2026-09-24 d05c370def845f01 · report
fixture: signature name:str/0/str, prefix:str/0/str · returned str · level recorded 2026-09-24 20e27f52f8beba5e · report
fixture: signature inputs:str/0/str, remove_space:bool/0/bool, lower:bool/0/bool · returned str · level recorded 2026-09-24 0e5615f8994003cf · report
fixture: signature text:str/0/str · returned str · level recorded 2026-09-26 73a4c1c1621ab1fc · report
9c4914e087369deb · report
fixture: signature sp_model:MockSentencePieceModel/0/?, text:str/0/str, sample:bool/0/bool · returned list · level recorded 2026-09-24 306ef06ed54230e4 · report
fixture: signature sp_model:MockSentencePieceModel/0/?, text:str/0/str, return_unicode:bool/0/bool, sample:bool/0/bool · returned list · level recorded 2026-09-24 53ab86e6b6c686ae · report
fixture: signature inputs:str/0/str, remove_space:bool/0/bool, do_lower_case:bool/0/bool · returned str · level recorded 2026-09-24 5eff22fa0a651276 · report
fixture: signature url:str/0/str, etag:str/0/str · returned str · level recorded 2026-09-22 c7c4917ba776e16d · report
c405dd24b1c4e3bc · report
d143b86f7eb37e1d · report
fixture: returned AttentionSortNet · level recorded 2026-09-22 04cc0c0e9fb1cf4c · report
fixture: returned SimpleSortNet · level recorded 2026-09-22 8429e6520e353d4d · report
fixture: signature values:Tensor/3/float, indices:Tensor/2/int · returned Tensor · level recorded 2026-09-22 048663070f64ac68 · report
fixture: signature buckets:int/0/int, t:Tensor/2/float, dim:int/0/int · returned Tensor · level recorded 2026-09-22 9477e59f3a606128 · report
fixture: signature x:Tensor/3/float, k:int/0/int, temperature:float/0/float · returned Tensor · level recorded 2026-09-22 e5d5e9b3569db056 · report
fixture: signature b:int/0/int, t:Tensor/3/float · returned Tensor · level recorded 2026-09-22 75494c9e9243dc8c · report
fixture: signature r:Tensor/3/float, n_iters:int/0/int, temperature:float/0/float · returned Tensor · level recorded 2026-09-22 9e2c5a79e9d85bc3 · report
fixture: signature ind_from:int/0/int, ind_to:int/0/int, tensor:Tensor/4/float · returned Tensor · level recorded 2026-09-22 1ef7a31f789d0d5a · report
fixture: signature t:Tensor/4/float, r:Tensor/3/float · returned Tensor · level recorded 2026-09-22 0cbc5bf5174a2a25 · report
fixture: signature r:Tensor/3/float, n_iters:int/0/int · returned Tensor · level recorded 2026-09-22 caf39b2dd49a315b · report
fixture: signature t:Tensor/4/float, dim:int/0/int · returned Tensor · level recorded 2026-09-22 15d94e7dd53dd4b7 · report
30f0276ce4caf77f · report
census: 3 implementations from 5 papers, 1 distinct output on one shared input sinkhorn_transformer/sinkhorn_transformer.py unverified MIT (permissive) ·
0d18b4b512155169 · report
9850e836a8bb97ce · report
fixture: returned FactorizedEmbedding · level recorded 2026-09-22 a0dff78471b54854 · report
fixture: returned NystromAttention · level recorded 2026-09-22 census: 6 implementations from 6 papers, 4 distinct outputs across 3 buckets, one shared input per bucket albert/model.py ran fingerprinted MIT (permissive) ·
d6e09a6ed82fe99e · report
fixture: returned PositionalEmbedding · level recorded 2026-09-22 979c42dd40b4b6bb · report
fixture: returned SoftmaxAttention · level recorded 2026-09-22 3fbbc185d6497778 · report
8434ec6cb6412213 · report
census: 5 implementations from 5 papers, 5 distinct outputs across 3 buckets, one shared input per bucket models/albert_modules/albert.py ran · metamorphic tier: invariant fingerprinted no licence file found · pointer only ·
3f504ebe7af3ba40 · report
fixture: signature x:Tensor/2/float · returned Tensor · level recorded 2026-09-22 80bf93fd89246988 · report
fixture: returned TokenEmbedding · level recorded 2026-09-22 fa81a9ae49ac9b0d · report
2bd58eb578fc4911 · report
b0a2ed9c8469e733 · report
d36a9fea67ab62a3 · report
35df638989183cf3 · report
fixture: returned AlbertConfig · level recorded 2026-09-22 215c14713864147c · report
fixture: returned Dense3D · level recorded 2026-09-22 6cbf6bc0afb44768 · report
fixture: returned EmbeddingLookup · level recorded 2026-09-22 bc41b5cdd27882da · report
fixture: returned EmbeddingPostprocessor · level recorded 2026-09-22 591598c6e9e9294a · report
fixture: signature initializer_range:float/0/float · returned TruncatedNormal · level recorded 2026-09-22 census: 13 implementations from 12 papers, 10 distinct outputs across 4 buckets, one shared input per bucket albert.py unverified Apache-2.0 (permissive) ·
a8a24272ed7b74f4 · report
64da3191abb9f8c9 · report
37b7e327abf6ebe3 · report
b9e24c3e3e0e88a7 · report
fixture: signature args:Namespace/0/?, cfg:dict/0/? · returned dict · level recorded 2026-09-22 37834d67b70e1be4 · report
fixture: signature parser:ArgumentParser/0/?, cfg:dict/0/?, helper:dict/0/?, choices:dict/0/?, cfg_path:str/0/str · returned Namespace · level recorded 2026-09-24 b0f7ef341475d3d7 · report
fixture: signature yaml_path:str/0/str · returned tuple · level recorded 2026-09-24 Method as code Syntology
What Syntology's graph records as this paper's claimed method, lifted into code: each assembly names its bricks (functions and classes), each with its own role and its own verification level. An assembly is not a check that the code implements the method correctly or reproduces the paper's results. Levels are each brick's own verification level in Syntology's graph (level 0: no recorded run), listed brick by brick and never combined into one level; “ran” means the brick executed on a synthesized input, not that its output is correct, and the instrument figure counts failures of Syntology's instrument, not of the code. Where a model read the code against the paper's claim, each assembly's label verdict is one model's judgement, kept in this page's JSON twin; it is not calibrated, so no card prints it. Siblings are assemblies lifted from this paper in the same batch for the same method: alternative readings kept side by side; none replaces or supersedes another.
9 assemblies on this paper: 9 harvested, 0 generated, listed in 1 groups (siblings lifted in one batch form one group).
9 sibling assemblies for ALBERT (harvested), lifted in the same batch on 2026-09-09: alternative readings kept side by side; none replaces or supersedes another.
harvested Assembly for ALBERT (graph method: ALBERT). 1 brick; individually on a fixture: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result. The assembly has not been run end-to-end.
one of 9 siblings
- MultiHeadBlockSelf entry · level 1 · ran ·
0afbeb40a45a72b6
harvested Assembly for ALBERT (graph method: ALBERT). 6 bricks; individually on a fixture: 2 ran (of which 1 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified. The assembly has not been run end-to-end.
one of 9 siblings
- ALBERT entry · level 0 · unverified ·
fa81a9ae49ac9b0d - ALBERTEmbedding model · level 0 · unverified ·
2bd58eb578fc4911 - TransformerLayer model · level 0 · unverified ·
d36a9fea67ab62a3 - LayerNorm helper · level 1 · ran ·
3f504ebe7af3ba40 - TokenEmbedding helper · level 1 · ran ·
80bf93fd89246988 - PositionwiseFeedForward helper · level 0 · unverified ·
b0a2ed9c8469e733
harvested Assembly for ALBERT (graph method: ALBERT). 8 bricks; individually on a fixture: 8 ran (of which 4 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified. The assembly has not been run end-to-end.
one of 9 siblings
- Transformer entry · level 1 · ran ·
b11322b4f500cef1 - Embeddings model · level 1 · ran ·
00437373da100756 - LayerNorm model · level 1 · ran ·
31b6e6cb2bf23d35 - MultiHeadedSelfAttention model · level 1 · ran ·
8734ddba5f76437a - PositionWiseFeedForward model · level 1 · ran ·
e952b1d84bed0af6 - merge_last helper · level 1 · ran; the contract check could not run (RAISES) ·
d5e69fc7b0ee7c8c - split_last helper · level 1 · ran; the contract check could not run (RAISES) ·
a4a5354a3138c8ef - gelu helper · level 1 · ran · honoured contract ·
62e7409a816626ac
harvested Assembly for ALBERT (graph method: ALBERT). 11 bricks; individually on a fixture: 5 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified. The assembly has not been run end-to-end.
one of 9 siblings
- AlbertModel entry · level 0 · unverified ·
252e953d83e45726 - AlbertConfig helper · level 1 · ran ·
35df638989183cf3 - EmbeddingLookup model · level 1 · ran ·
6cbf6bc0afb44768 - EmbeddingPostprocessor model · level 1 · ran ·
bc41b5cdd27882da - Transformer model · level 0 · unverified ·
fe12e4019156a236 - create_attention_mask_from_input_mask helper · level 0 · unverified ·
37b7e327abf6ebe3 - get_initializer helper · level 1 · ran · our draft was wrong ·
591598c6e9e9294a - Dense2DProjection model · level 0 · unverified ·
aa9a005ff917d7df - TransformerBlock model · level 0 · unverified ·
64da3191abb9f8c9 - Attention model · level 0 · unverified ·
a8a24272ed7b74f4 - Dense3D model · level 1 · ran ·
215c14713864147c
harvested Assembly for ALBERT (graph method: ALBERT). 9 bricks; individually on a fixture: 5 ran (of which 4 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 1 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified. The assembly has not been run end-to-end.
one of 9 siblings
- ALBERT entry · level 0 · unverified ·
fea0515d9942074e - FactorizedEmbedding model · level 1 · ran ·
9850e836a8bb97ce - HelperModule helper · level 0 · unverified ·
394400bd8d814f2c - PositionalEmbedding model · level 1 · ran ·
d6e09a6ed82fe99e - TransformerLayer model · level 0 · unverified ·
3fbbc185d6497778 - NystromAttention helper · level 1 · ran ·
a0dff78471b54854 - SoftmaxAttention model · level 1 · ran ·
979c42dd40b4b6bb - exists helper · level 1 · ran · violated contract ·
aa5486a3650902d8 - moore_penrose_iter_pinv helper · level 0 · unverified ·
8434ec6cb6412213
harvested Assembly for ALBERT (graph method: ALBERT). 18 bricks; individually on a fixture: 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 14 unverified. The assembly has not been run end-to-end.
one of 9 siblings
- AlbertModel entry · level 0 · unverified ·
91c075af55caeadc - create_initializer helper · level 0 · unverified ·
e3981aa2fadfb2eb - embedding_lookup helper · level 0 · unverified ·
058d2e8f50dd0987 - embedding_postprocessor helper · level 0 · unverified ·
8efebf3cb9b8ca4d - get_activation helper · level 1 · ran · our draft was wrong ·
ae9ebb2661d2ddd4 - get_shape_list helper · level 1 · ran · our draft was wrong ·
4c33136de32333c9 - transformer_model model · level 0 · unverified ·
e447e4b29a5d024b - layer_norm_and_dropout helper · level 0 · unverified ·
4e61fb4457fa4a37 - gelu helper · level 1 · ran · honoured contract ·
cb61421413d30b87 - assert_rank helper · level 0 · unverified ·
01e4a365b28cda00 - attention_ffn_block helper · level 0 · unverified ·
94f549fd1bf6a555 - dense_layer_2d helper · level 0 · unverified ·
f9aab49e4bfb299f - dropout helper · level 1 · ran; the contract check could not run (RAISES) ·
2bf900d0da165894 - layer_norm helper · level 0 · unverified ·
fedeb6766bad1fad - attention_layer model · level 0 · unverified ·
154407ccae03925e - dense_layer_3d_proj helper · level 0 · unverified ·
b0e3d2a74b1bb3e4 - dense_layer_3d helper · level 0 · unverified ·
babf39a9a9bf5587 - dot_product_attention helper · level 0 · unverified ·
847351b08a6a93c6
harvested Assembly for ALBERT (graph method: ALBERT). 2 bricks; individually on a fixture: 2 ran (of which 1 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified. The assembly has not been run end-to-end.
one of 9 siblings
- RecAdam entry · level 1 · ran ·
350a1a7ab2f301c8 - anneal_function helper · level 1 · ran · honoured contract ·
b8301a2a061e30a2
harvested Assembly for ALBERT (graph method: ALBERT). 1 brick; individually on a fixture: 0 ran · 1 unverified. The assembly has not been run end-to-end.
one of 9 siblings
- CalbertForMaskedLM entry · level 0 · unverified ·
ff36be852a0ce11b
harvested Assembly for ALBERT (graph method: ALBERT). 3 bricks; individually on a fixture: 2 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified. The assembly has not been run end-to-end.
one of 9 siblings
- ALBERTCollaborativeAdapter entry · level 0 · unverified ·
b6a5b207adc026a6 - CollaborativeLayerAdapter helper · level 1 · ran ·
e6c28dfaa7ebc812 - albert_layers helper · level 1 · ran · our draft was wrong ·
b2116e6e872a2a35
Each brick ends with its code_sha256, Syntology's identity for that exact code; click one to copy the get_code call that returns it. For agents, on Syntology's MCP service: assembly(arxiv_id="1909.11942") returns these assemblies with each brick's role and wiring (how to connect).
Tasks
1 archive task tag without a task page not shown.
Results from the paper archive 2025-07-28
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections