Papers › Unsupervised Cross-lingual Representation Learning at Scale

Unsupervised Cross-lingual Representation Learning at Scale

5 Nov 2019ACL 2020 6arXiv:1911.02116archive 2025-07-28

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov

This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6% average accuracy on XNLI, +13% average F1 score on MLQA, and +2.4% F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 15.7% in XNLI accuracy for Swahili and 11.4% for Urdu over previous XLM models. We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-R is very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make our code, data and models publicly available.

PaperPDFConference PDFCodeCode Syntology ranMethod as code

In Syntology View this paper on Syntology, its page in Syntology's graph. That page adds which of its functions are also present in other papers' code Syntology holds, with dates, and its dated run record as JSON and a README badge. It also lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

Syntology 59 code samples: 55 from 9 repositories linked to this paper, 4 identical to code harvested from another paper's repository. Across all 59: 40 ran (of which 13 constructed an object rather than computing a result; 35 with no instrument failure: 4 honoured, 1 violated, 30 with no contract checked; 5 where Syntology's instrument failed) · 19 unverified. Where Syntology's instrument failed: 2 where our contract draft was wrong; 3 where the contract check could not run; failures of Syntology's instrument, not of the code.

By repository: official repository: 15 samples from 1 repository, 12 ran (12 with no instrument failure; 0 where Syntology's instrument failed); community (archive-listed): 40 samples from 8 repositories, 24 ran (21 with no instrument failure; 3 where Syntology's instrument failed); 4 identical to code first harvested elsewhere. The run record, sample by sample. “Ran” means executed on a synthesized input, not that the code is correct or reproduces the paper.

No sample is linked to this paper from 10 of its repositories that Syntology harvested for other papers:

Levels written between 2026-09-22 and 2026-09-25 (UTC): the dates Syntology's graph wrote each sample's verification level, not when the code executed.

35 repositories listed; official and paper-mentioned ones first.

facebookresearch/cc_net officialmentioned in papermentioned on GitHubMITSyntology: 15 samples, 12 ran (0 instrument-failed) · 3 unverified report
facebookresearch/XLM officialmentioned on GitHubpytorchSyntology: not harvested report
1-punchMan/CLTS mentioned on GitHubpytorchSyntology: 7 samples, 4 ran (0 instrument-failed) · 3 unverified report
Stanley-Zheng/ignitionhacks mentioned on GitHubtfSyntology: not harvested report
Tikquuss/meta_XLM mentioned on GitHubpytorchSyntology: 22 samples, 15 ran (3 instrument-failed) · 7 unverified report
Unbabel/COMET mentioned on GitHubpytorchSyntology: 3 samples, 0 ran · 3 unverified report
YuweiYin/HLT-MT mentioned on GitHubpytorchSyntology: not harvested report
aleksac99/SQuAD-SR mentioned on GitHubSyntology: not harvested report
apple/ml-mkqa mentioned on GitHubApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering (arXiv:2007.15207; that paper's own run record: 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
ashwanitanwar/nmt-transfer-learning-xlm-r mentioned on GitHubpytorchNOASSERTIONSyntology: not harvested report
bcaitech1/p2-klue-Heeseok-Jeong mentioned on GitHubpytorchSyntology: not harvested report
bitextor/bicleaner-ai mentioned on GitHubtfGPL-3.0Syntology: not harvested report
deepmind/xquad mentioned on GitHubSyntology: not harvested report
deepset-ai/FARM mentioned on GitHubpytorchSyntology: 1 sample, 0 ran · 1 unverified report
deterministic-algorithms-lab/Large-XLM mentioned on GitHubpytorchSyntology: 2 samples, 1 ran (0 instrument-failed) · 1 unverified report
feyzaakyurek/XLM-LwLL mentioned on GitHubpytorchNOASSERTIONSyntology: not harvested report
google-deepmind/xquad mentioned on GitHubSyntology: not harvested report
heraclex12/vietpunc mentioned on GitHubjaxSyntology: 1 sample, 1 ran (0 instrument-failed) · 0 unverified report
huggingface/transformers mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Attention Is All You Need (arXiv:1706.03762; that paper's own run record: 610 ran (of which 293 constructed an object rather than computing a result; 529 with no instrument failure: 45 honoured, 15 violated, 469 with no contract checked; 81 where Syntology's instrument failed) · 336 unverified), HuggingFace's Transformers: State-of-the-art Natural Language Processing (arXiv:1910.03771; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified), DeBERTa: Decoding-enhanced BERT with Disentangled Attention (arXiv:2006.03654; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified) and 1 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
huhailinguist/ChineseNLIProbing mentioned on GitHubpytorchSyntology: not harvested report
ilhamfp/indonesian-text-classification-multilingual mentioned on GitHubpytorchSyntology: 1 sample, 0 ran · 1 unverified report
inzva/turkish-glove mentioned on GitHubMITSyntology: not harvested report
kheeong/XLM_OWN mentioned on GitHubpytorchNOASSERTIONSyntology: not harvested report
lsqdecodebox/NLP-Comptition-review mentioned on GitHubApache-2.0Syntology: not harvested report
martincjespersen/DaAnonymization mentioned on GitHubApache-2.0Syntology: 3 samples, 3 ran (0 instrument-failed) · 0 unverified report
nlp-uoregon/trankit mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Unlearning What Matters: Token-Level Attribution for Precise Language Model Unlearning (arXiv:2605.00364; that paper's own run record: 5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 4 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified); nothing checks that they implement this paper's method. report
sanderwood/clamp3 mentioned on GitHubpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages (arXiv:2502.10362; that paper's own run record: 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
somiltg/xquad mentioned on GitHubSyntology: not harvested report
stanleyjzheng/ignitionhacks mentioned on GitHubtfSyntology: not harvested report
MindCode-4/code-13 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learning (arXiv:2212.08663; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report
MindCode-4/code-5 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to ViTamin: Designing Scalable Vision Models in the Vision-Language Era (arXiv:2404.02132; that paper's own run record: 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 1 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 6 unverified); nothing checks that they implement this paper's method. report
MindSpore-scientific-2/code-3 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to OctFormer: Octree-based Transformers for 3D Point Clouds (arXiv:2305.03045; that paper's own run record: 6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified); nothing checks that they implement this paper's method. report
MindSpore-scientific/code-12 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (arXiv:2503.09573; that paper's own run record: 14 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 7 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report
MindSpore-scientific/code-14 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data,... (arXiv:2306.01116; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
pwc-1/Paper-9 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to RandAugment: Practical automated data augmentation with a reduced search space (arXiv:1909.13719; that paper's own run record: 58 ran (of which 1 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 1 violated, 6 with no contract checked; 51 where Syntology's instrument failed) · 7 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

59 samples harvested: 40 ran (of which 13 constructed an object rather than computing a result; 35 with no instrument failure: 4 honoured, 1 violated, 30 with no contract checked; 5 where Syntology's instrument failed) · 19 unverified. “Where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Syntology's record for this page has not changed since , the first build that kept a record date for it; when this build read Syntology's graph is in the build record. That date is not when the samples ran.

4ran · honoured contract
1ran · violated contract
2ran · our draft was wrong
3ran; the contract check could not run
30ran
19unverified

Licence: 52 of the 59 samples are pointer only, meaning Syntology does not serve that copy's text. This page shows no code text for any sample; each one links to its file in the repository.

Harvested from 9 repositories linked to this paper, official or community; the list below is grouped by repository, each group headed by its name and what it is to the paper. 4 samples are listed last, in a group without a repository. “Ran” means the sample executed on a synthesized input. It does not mean the output is correct, and nothing here reproduces the paper's results. A line beginning “fixture:” under a sample that ran gives the arguments Syntology's fixture called the sample with, each as name:type/rank/element kind (? where none was recorded), and the type of what the fixture's run returned, exactly as the harness recorded them; the arguments and the returned type were recorded by two separate runs of the same fixture. No output value is shown, and a returned type says nothing about whether the output was right. “Level recorded” gives the UTC date when Syntology's graph wrote the sample's verification level, not when the code executed; where the level was written before Syntology's graph kept that date, the row says “level recorded, date not recorded”. For 8 samples that ran, the graph records a signature this page does not show: for a class the fixture built without calling, the harness called the object with inputs it made up itself, so the recorded signature is the harness's probe, not the fixture's arguments. “Honoured” and “violated” refer to a contract Syntology drafted from the code itself. “Our draft was wrong” and “the contract check could not run” are failures of Syntology's instrument, not of the code: the sample ran on Syntology's synthesized input, and then the drafted contract did not describe it, or the check itself raised an error (RAISES), lacked a dependency (DEP_MISSING) or ran out of time (TIMEOUT); hover the label for which. A metamorphic tier, where shown, is evidence of the output's shape, not that the sample implements the method.

Each sample ends with its code_sha256. A sample's code_sha256 is Syntology's identity for that exact code. It is printed in full (16 hexadecimal characters for every sample in this build), and get_code(code_sha256="…") takes it exactly as printed. An agent calls it on Syntology's MCP service (how to connect); click an identity to copy that call.

Repository labels, per sample. official repository: The archive marks this repository official for the paper. named in the paper: The archive records that the paper mentions this repository; it is not marked official. community (archive-listed): In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper. found in paper text by Syntology: Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. community: Not in the archive's code links for this paper; a community repository Syntology harvested. Samples are grouped by repository: a repository marked official first, then by how directly the repository is tied to the paper, then in the archive's order; never by how many ran. Licence labels name the repository's licence as recorded at harvest. “Pointer only” means Syntology does not serve that copy's text, for one of six reasons: no licence file was found; the licence was not identified; the licence is recorded as permissive but that copy's record is not marked cleared; the licence is outside the permissive list Syntology serves text under (MIT, Apache-2.0, BSD and similar); this paper's copy was not located, so its licence is not recorded; or this paper's copy is cleared, but the stored copy of the same code that Syntology serves is not. Some licences outside that list permit redistribution, such as WTFPL, and GPL-3.0 under its conditions; they are simply not on the list. Hover a licence label for the reason. File links open the file on GitHub at the default branch, which may have changed since the harvest.

A function or class name links to its page when the name occurs in the harvested code of at least 5 papers; the name is a string, not an identity. A census tag appears when Syntology's census ran implementations with that name on one shared input per bucket: it gives how many were compared, from how many papers, and how many distinct outputs they produced, and names this sample's bucket when this sample was one of them.

facebookresearch/cc_net official repository 15 samples, 12 ran (0 instrument-failed) · 3 unverified
as_pipe cc_net/jsonql.py ran MIT recorded; this copy not marked cleared · pointer only · b9f2456deb48efe3 · report fixture: signature transformer:callable/0/?, kwargs:dict/0/? · returned TransformerClass · level recorded 2026-09-25
cc_wet_paths_url cc_net/process_wet_file.py ran fingerprinted MIT recorded; this copy not marked cleared · pointer only · c8b6d34869aa3762 · report fixture: signature dump_id:str/0/str · returned str · level recorded 2026-09-25
encode_as_hashes cc_net/minify.py ran MIT recorded; this copy not marked cleared · pointer only · 6cb4b3cddb2b34b6 · report fixture: signature lines:list/1/str · returned str · level recorded 2026-09-25
encode_hashes cc_net/minify.py ran MIT recorded; this copy not marked cleared · pointer only · 78ae9912884277ad · report fixture: signature hashes:list/1/bytes · returned str · level recorded 2026-09-25
fast_reshard cc_net/regroup.py ran MIT recorded; this copy not marked cleared · pointer only · 3143d897968b04d0 · report fixture: signature inputs:list/1/PosixPath, output:PosixPath/0/?, tmp:PosixPath/0/?, free_original:bool/0/bool, rm_original:bool/0/bool · returned PosixPath · level recorded 2026-09-25
get_hashes cc_net/minify.py ran MIT recorded; this copy not marked cleared · pointer only · 1e7f7cbd3125f871 · report fixture: signature lines:list/1/str · returned list · level recorded 2026-09-25
get_index in 27 papers cc_net/regroup.py ran MIT recorded; this copy not marked cleared · pointer only · 1f765197d3df9584 · report fixture: signature file:PosixPath/0/? · returned PosixPath · level recorded 2026-09-25
parse_doc cc_net/process_wet_file.py ran fingerprinted MIT recorded; this copy not marked cleared · pointer only · aeaad31adb65d876 · report fixture: signature headers:list/1/str, doc:list/1/str · returned dict · level recorded 2026-09-25
pp cc_net/perplexity.py ran fingerprinted MIT recorded; this copy not marked cleared · pointer only · 50c73a5e3a282bc8 · report fixture: signature log_score:float/0/float, length:float/0/float · returned float · level recorded 2026-09-25
predict in 126 papers cc_net/split_by_lang.py ran MIT recorded; this copy not marked cleared · pointer only · 0d75948d09185811 · report fixture: signature model:callable/0/?, text:str/0/str, k:int/0/int · returned tuple · level recorded 2026-09-25
tmp cc_net/get_wiki_cirrus.py ran MIT recorded; this copy not marked cleared · pointer only · cda9917a1ba9c984 · report fixture: signature file:PosixPath/0/? · returned PosixPath · level recorded 2026-09-25
tmp cc_net/mine.py ran MIT recorded; this copy not marked cleared · pointer only · 8e1af02efb15e082 · report fixture: signature output:PosixPath/0/? · returned PosixPath · level recorded 2026-09-25
avg_predict cc_net/split_by_lang.py unverified MIT recorded; this copy not marked cleared · pointer only · 9a3ea499eb0114dc · report
compose in 9 papers cc_net/jsonql.py unverified MIT recorded; this copy not marked cleared · pointer only · 4bb30f62c8aa7df1 · report
run_pipe cc_net/jsonql.py unverified MIT recorded; this copy not marked cleared · pointer only · 8e7895f398977866 · report
1-punchMan/CLTS community (archive-listed) 7 samples, 4 ran (0 instrument-failed) · 3 unverified
LabelSmoothingLoss in 6 papers src/model/transformer.py ran licence not identified · pointer only · 4a9d2d7dd384f8bf · report fixture: signature output:Tensor/2/float, target:Tensor/1/int · returned Tensor · level recorded 2026-09-24
MultiHeadAttention in 124 papers
census: 9 implementations from 7 papers, 9 distinct outputs across 3 buckets, one shared input per bucket
src/model/transformer.py ran licence not identified · pointer only · 87278bd6d018466e · report fixture: returned MultiHeadAttention · level recorded 2026-09-22
PredLayer src/model/transformer.py ran licence not identified · pointer only · ba5fdac8a37a5455 · report fixture: returned PredLayer · level recorded 2026-09-22
QueryIdentity src/model/transformer.py ran licence not identified · pointer only · 2c9fdf8b95e37258 · report fixture: returned QueryIdentity · level recorded 2026-09-22
CrossLingualTransformerModel src/model/transformer.py unverified licence not identified · pointer only · 33c46f658371888d · report
HashingMemoryFlat src/model/transformer.py unverified licence not identified · pointer only · c4f0cfb3b4341157 · report
HashingMemoryProductFast src/model/transformer.py unverified licence not identified · pointer only · 200a1dfd844a9fe4 · report
Tikquuss/meta_XLM community (archive-listed) 22 samples, 15 ran (3 instrument-failed) · 7 unverified
BeamHypotheses XLM/src/model/transformer.py ran licence not identified · pointer only · ea265a8f7f9d376b · report fixture: returned BeamHypotheses · level recorded 2026-09-22
BottleneckResidualConv2d XLM/src/model/transformer.py ran fingerprinted licence not identified · pointer only · 2d39718257c802dc · report fixture: returned BottleneckResidualConv2d · level recorded 2026-09-22
GroupedLinear XLM/src/model/transformer.py ran licence not identified · pointer only · d470784790e9dc66 · report fixture: signature input:Tensor/2/float · returned Tensor · level recorded 2026-09-24
MultiHeadAttention in 124 papers
census: 9 implementations from 7 papers, 9 distinct outputs across 3 buckets, one shared input per bucket
XLM/src/model/transformer.py ran licence not identified · pointer only · 59c6324aa51d05e7 · report fixture: returned MultiHeadAttention · level recorded 2026-09-22
PredLayer XLM/src/model/transformer.py ran licence not identified · pointer only · d204f4ff6c43fc6d · report fixture: returned PredLayer · level recorded 2026-09-22
QueryConv XLM/src/model/transformer.py ran licence not identified · pointer only · ddc3ebfb70be4b09 · report fixture: returned QueryConv · level recorded 2026-09-22
QueryIdentity XLM/src/model/transformer.py ran licence not identified · pointer only · 8ba7ef0726990c84 · report fixture: returned QueryIdentity · level recorded 2026-09-22
QueryMLP XLM/src/model/transformer.py ran fingerprinted licence not identified · pointer only · 72cc8733f4306eba · report fixture: returned QueryMLP · level recorded 2026-09-22
TransformerFFN in 5 papers XLM/src/model/transformer.py ran licence not identified · pointer only · 38525832e182b5f3 · report fixture: returned TransformerFFN · level recorded 2026-09-22
cartesian_product XLM/src/model/transformer.py ran · our draft was wrong fingerprinted licence not identified · pointer only · 424066fae6f1c350 · report fixture: signature a:Tensor/2/float, b:Tensor/2/float · returned Tensor · level recorded 2026-09-22
convs XLM/src/model/transformer.py ran; the contract check could not run (RAISES) licence not identified · pointer only · e7bd7636a1df749c · report fixture: signature channel_sizes:list/1/int, kernel_sizes:list/1/int, bias:bool/0/bool, batchnorm:bool/0/bool, residual:bool/0/bool, groups:int/0/int · returned Sequential · level recorded 2026-09-22
get_gaussian_keys XLM/src/model/transformer.py ran · honoured contract licence not identified · pointer only · 1d77d0309d035999 · report fixture: signature n_keys:int/0/int, dim:int/0/int, normalized:bool/0/bool, seed:int/0/int · returned ndarray · level recorded 2026-09-22
get_slices XLM/src/model/transformer.py ran · honoured contract fingerprinted licence not identified · pointer only · 91283fd5a725ec99 · report fixture: signature dim:int/0/int, head_id:int/0/int · returned list · level recorded 2026-09-22
get_uniform_keys XLM/src/model/transformer.py ran · honoured contract licence not identified · pointer only · 3fa144c23db44a7e · report fixture: signature n_keys:int/0/int, dim:int/0/int, normalized:bool/0/bool, seed:int/0/int · returned ndarray · level recorded 2026-09-22
mlp in 108 papers XLM/src/model/transformer.py ran; the contract check could not run (RAISES) licence not identified · pointer only · a05791210b07b83d · report fixture: signature sizes:list/1/int, bias:bool/0/bool, batchnorm:bool/0/bool, groups:int/0/int · returned Sequential · level recorded 2026-09-22
HashingMemoryFlat XLM/src/model/transformer.py unverified licence not identified · pointer only · 3d693a4a286601f2 · report
HashingMemoryProductFast XLM/src/model/transformer.py unverified licence not identified · pointer only · f8da1eb21868463a · report
TransformerModel in 26 papers XLM/src/model/transformer.py unverified licence not identified · pointer only · e836ec1da3a517b3 · report
create_sinusoidal_embeddings in 5 papers XLM/src/model/transformer.py unverified licence not identified · pointer only · 99c39b47ca18e1b5 · report
get_knn_faiss XLM/src/model/transformer.py unverified licence not identified · pointer only · 80bf3f1cdd634770 · report
swig_ptr_from_FloatTensor in 5 papers XLM/src/model/transformer.py unverified licence not identified · pointer only · ded0b9ac1f3c803c · report
swig_ptr_from_LongTensor in 5 papers XLM/src/model/transformer.py unverified licence not identified · pointer only · 41f2f12e6d5e6d33 · report
Unbabel/COMET community (archive-listed) 3 samples, 0 ran · 3 unverified
BERTEncoder comet/encoders/xlmr.py unverified Apache-2.0 (permissive) · bee8cacb03b2b055 · report
Encoder in 278 papers comet/encoders/xlmr.py unverified Apache-2.0 (permissive) · ce7a4a01c9b960a2 · report
XLMREncoder comet/encoders/xlmr.py unverified Apache-2.0 (permissive) · 292808e3a5d973ed · report
deepset-ai/FARM community (archive-listed) 1 sample, 0 ran · 1 unverified
XLMRoberta farm/modeling/language_model.py unverified Apache-2.0 (permissive) · 12b4ab9bb368a1e8 · report
deterministic-algorithms-lab/Large-XLM community (archive-listed) 2 samples, 1 ran (0 instrument-failed) · 1 unverified
MultiHeadAttention in 124 papers
census: 9 implementations from 7 papers, 9 distinct outputs across 3 buckets, one shared input per bucket
src/model/transformer.py ran licence not identified · pointer only · 8ed44ce7206b1f24 · report fixture: returned MultiHeadAttention · level recorded 2026-09-22
TransformerModel in 26 papers src/model/transformer.py unverified licence not identified · pointer only · 68f606f965d049f0 · report
heraclex12/vietpunc community (archive-listed) 1 sample, 1 ran (0 instrument-failed) · 0 unverified
PuncXLMRModel models/xlm_roberta.py ran no licence file found · pointer only · b8082687538a83e1 · report fixture: signature input_ids:Tensor/2/int, token_type_ids:Tensor/2/int, attention_mask:Tensor/2/int, labels:Tensor/2/int, valid_ids:Tensor/2/int, attention_mask_label:Tensor/2/int · returned Tensor · level recorded 2026-09-24
ilhamfp/indonesian-text-classification-multilingual community (archive-listed) 1 sample, 0 ran · 1 unverified
FeatureExtractor in 24 papers src/extract-feature.py unverified no licence file found · pointer only · d30d1ecf1454760e · report
martincjespersen/DaAnonymization community (archive-listed) 3 samples, 3 ran (0 instrument-failed) · 0 unverified
get_float textprivacy/utils.py ran fingerprinted Apache-2.0 (permissive) · 1e9b7714930dc642 · report fixture: signature number:str/0/str · returned tuple · level recorded 2026-09-24
get_integer textprivacy/utils.py ran fingerprinted Apache-2.0 (permissive) · b2b68b9410ef8d8b · report fixture: signature number:str/0/str · returned int · level recorded 2026-09-24
is_valid_number in 5 papers textprivacy/utils.py ran fingerprinted Apache-2.0 (permissive) · 81d3a619bdd21b88 · report fixture: signature number:str/0/str · returned str · level recorded 2026-09-24
Repository not located 4 samples, 4 ran (2 instrument-failed) · 0 unverified
4 samples are identical to code Syntology harvested from another paper's repository; this paper's own copy was not located, so no repository is named and no licence is recorded.
Linear in 127 papers repository not located ran · our draft was wrong licence of this copy not recorded · b5254f10aa821ab8 · report fixture: signature in_features:int/0/int, out_features:int/0/int, bias:bool/0/bool · returned Linear · level recorded 2026-09-22
bool_flag in 47 papers repository not located ran · violated contract licence of this copy not recorded · cc66d68d2046d6a3 · report fixture: signature s:str/0/str · returned bool · level recorded 2026-09-22
gelu in 308 papers
census: 43 implementations from 74 papers, 6 distinct outputs on one shared input this sample: bucket 1 of 1, arg 1: rank 2, kind float, dtype float32
repository not located ran · honoured contract fingerprinted licence of this copy not recorded · 66c21a3911091d29 · report fixture: signature x:Tensor/2/float · returned Tensor · level recorded 2026-09-22
get_masks in 23 papers repository not located ran; the contract check could not run (RAISES) · metamorphic tier: deterministic licence of this copy not recorded · b2470f4f0c09db41 · report fixture: signature slen:int/0/int, lengths:Tensor/1/int, causal:bool/0/bool · returned tuple · level recorded 2026-09-22

Method as code Syntology

What Syntology's graph records as this paper's claimed method, lifted into code: each assembly names its bricks (functions and classes), each with its own role and its own verification level. An assembly is not a check that the code implements the method correctly or reproduces the paper's results. Levels are each brick's own verification level in Syntology's graph (level 0: no recorded run), listed brick by brick and never combined into one level; “ran” means the brick executed on a synthesized input, not that its output is correct, and the instrument figure counts failures of Syntology's instrument, not of the code. Where a model read the code against the paper's claim, each assembly's label verdict is one model's judgement, kept in this page's JSON twin; it is not calibrated, so no card prints it. Siblings are assemblies lifted from this paper in the same batch for the same method: alternative readings kept side by side; none replaces or supersedes another.

7 assemblies on this paper: 7 harvested, 0 generated, listed in 1 groups (siblings lifted in one batch form one group).

7 sibling assemblies for XLM-R (harvested), lifted in the same batch on 2026-09-09: alternative readings kept side by side; none replaces or supersedes another.

harvested Assembly for XLM-R (graph method: XLM-R). 1 brick; individually on a fixture: 0 ran · 1 unverified. The assembly has not been run end-to-end.

one of 7 siblings

  1. FeatureExtractor entry · level 0 · unverified · d30d1ecf1454760e

harvested Assembly for XLM-R (graph method: XLM-R). 3 bricks; individually on a fixture: 0 ran · 3 unverified. The assembly has not been run end-to-end.

one of 7 siblings

  1. XLMRoberta entry · level 0 · unverified · 12b4ab9bb368a1e8
  2. s3e_pooling helper · level 0 · unverified · 55ce9c59feb542e9
  3. Bert helper · level 0 · unverified · 36da368f07acddb7

harvested Assembly for XLM-R (graph method: XLM-R). 27 bricks; individually on a fixture: 20 ran (of which 10 constructed an object rather than computing a result; 14 with no instrument failure: 4 honoured, 1 violated, 9 with no contract checked; 6 where Syntology's instrument failed) · 7 unverified. The assembly has not been run end-to-end.

one of 7 siblings

  1. TransformerModel entry · level 0 · unverified · e836ec1da3a517b3
  2. BeamHypotheses helper · level 1 · ran · ea265a8f7f9d376b
  3. Embedding model · level 1 · ran; the contract check could not run · a3e27bd1e8b628ed
  4. MultiHeadAttention model · level 1 · ran · 59c6324aa51d05e7
  5. PredLayer model · level 1 · ran · d204f4ff6c43fc6d
  6. TransformerFFN model · level 1 · ran · 38525832e182b5f3
  7. create_sinusoidal_embeddings helper · level 0 · unverified · 99c39b47ca18e1b5
  8. get_masks helper · level 1 · ran; the contract check could not run (RAISES) · b2470f4f0c09db41
  9. Linear helper · level 1 · ran · our draft was wrong · b5254f10aa821ab8
  10. gelu helper · level 1 · ran · honoured contract · 66c21a3911091d29
  11. BottleneckResidualConv2d helper · level 1 · ran · 2d39718257c802dc
  12. GroupedLinear helper · level 1 · ran · d470784790e9dc66
  13. HashingMemoryFlat helper · level 0 · unverified · 3d693a4a286601f2
  14. HashingMemoryProductFast helper · level 0 · unverified · f8da1eb21868463a
  15. QueryConv helper · level 1 · ran · ddc3ebfb70be4b09
  16. QueryIdentity helper · level 1 · ran · 8ba7ef0726990c84
  17. QueryMLP helper · level 1 · ran · 72cc8733f4306eba
  18. bool_flag helper · level 1 · ran · violated contract · cc66d68d2046d6a3
  19. cartesian_product helper · level 1 · ran · our draft was wrong · 424066fae6f1c350
  20. convs helper · level 1 · ran; the contract check could not run (RAISES) · e7bd7636a1df749c
  21. get_gaussian_keys helper · level 1 · ran · honoured contract · 1d77d0309d035999
  22. get_knn_faiss helper · level 0 · unverified · 80bf3f1cdd634770
  23. get_slices helper · level 1 · ran · honoured contract · 91283fd5a725ec99
  24. get_uniform_keys helper · level 1 · ran · honoured contract · 3fa144c23db44a7e
  25. mlp helper · level 1 · ran; the contract check could not run (RAISES) · a05791210b07b83d
  26. swig_ptr_from_FloatTensor helper · level 0 · unverified · ded0b9ac1f3c803c
  27. swig_ptr_from_LongTensor helper · level 0 · unverified · 41f2f12e6d5e6d33

harvested Assembly for XLM-R (graph method: XLM-R). 26 bricks; individually on a fixture: 20 ran (of which 9 constructed an object rather than computing a result; 15 with no instrument failure: 4 honoured, 1 violated, 10 with no contract checked; 5 where Syntology's instrument failed) · 6 unverified. The assembly has not been run end-to-end.

one of 7 siblings

  1. CrossLingualTransformerModel entry · level 0 · unverified · 33c46f658371888d
  2. BeamHypotheses helper · level 1 · ran · ea265a8f7f9d376b
  3. Linear model · level 1 · ran · our draft was wrong · b5254f10aa821ab8
  4. MultiHeadAttention model · level 1 · ran · 87278bd6d018466e
  5. PredLayer model · level 1 · ran · ba5fdac8a37a5455
  6. TransformerFFN model · level 1 · ran · 38525832e182b5f3
  7. get_masks helper · level 1 · ran; the contract check could not run (RAISES) · b2470f4f0c09db41
  8. LabelSmoothingLoss loss · level 1 · ran · 4a9d2d7dd384f8bf
  9. gelu helper · level 1 · ran · honoured contract · 66c21a3911091d29
  10. BottleneckResidualConv2d helper · level 1 · ran · 2d39718257c802dc
  11. GroupedLinear helper · level 1 · ran · d470784790e9dc66
  12. HashingMemoryFlat helper · level 0 · unverified · c4f0cfb3b4341157
  13. HashingMemoryProductFast helper · level 0 · unverified · 200a1dfd844a9fe4
  14. QueryConv helper · level 1 · ran · ddc3ebfb70be4b09
  15. QueryIdentity helper · level 1 · ran · 2c9fdf8b95e37258
  16. QueryMLP helper · level 1 · ran · 72cc8733f4306eba
  17. bool_flag helper · level 1 · ran · violated contract · cc66d68d2046d6a3
  18. cartesian_product helper · level 1 · ran · our draft was wrong · 424066fae6f1c350
  19. convs helper · level 1 · ran; the contract check could not run (RAISES) · e7bd7636a1df749c
  20. get_gaussian_keys helper · level 1 · ran · honoured contract · 1d77d0309d035999
  21. get_knn_faiss helper · level 0 · unverified · 80bf3f1cdd634770
  22. get_slices helper · level 1 · ran · honoured contract · 91283fd5a725ec99
  23. get_uniform_keys helper · level 1 · ran · honoured contract · 3fa144c23db44a7e
  24. mlp helper · level 1 · ran; the contract check could not run (RAISES) · a05791210b07b83d
  25. swig_ptr_from_FloatTensor helper · level 0 · unverified · ded0b9ac1f3c803c
  26. swig_ptr_from_LongTensor helper · level 0 · unverified · 41f2f12e6d5e6d33

harvested Assembly for XLM-R (graph method: XLM-R). 27 bricks; individually on a fixture: 20 ran (of which 10 constructed an object rather than computing a result; 14 with no instrument failure: 4 honoured, 1 violated, 9 with no contract checked; 6 where Syntology's instrument failed) · 7 unverified. The assembly has not been run end-to-end.

one of 7 siblings

  1. TransformerModel entry · level 0 · unverified · 68f606f965d049f0
  2. BeamHypotheses helper · level 1 · ran · ea265a8f7f9d376b
  3. Embedding model · level 1 · ran; the contract check could not run · a3e27bd1e8b628ed
  4. MultiHeadAttention model · level 1 · ran · 8ed44ce7206b1f24
  5. PredLayer model · level 1 · ran · d204f4ff6c43fc6d
  6. TransformerFFN model · level 1 · ran · 38525832e182b5f3
  7. create_sinusoidal_embeddings model · level 0 · unverified · 99c39b47ca18e1b5
  8. get_masks helper · level 1 · ran; the contract check could not run (RAISES) · b2470f4f0c09db41
  9. Linear model · level 1 · ran · our draft was wrong · b5254f10aa821ab8
  10. gelu model · level 1 · ran · honoured contract · 66c21a3911091d29
  11. BottleneckResidualConv2d helper · level 1 · ran · 2d39718257c802dc
  12. GroupedLinear helper · level 1 · ran · d470784790e9dc66
  13. HashingMemoryFlat helper · level 0 · unverified · 3d693a4a286601f2
  14. HashingMemoryProductFast helper · level 0 · unverified · f8da1eb21868463a
  15. QueryConv helper · level 1 · ran · ddc3ebfb70be4b09
  16. QueryIdentity helper · level 1 · ran · 8ba7ef0726990c84
  17. QueryMLP helper · level 1 · ran · 72cc8733f4306eba
  18. bool_flag helper · level 1 · ran · violated contract · cc66d68d2046d6a3
  19. cartesian_product helper · level 1 · ran · our draft was wrong · 424066fae6f1c350
  20. convs helper · level 1 · ran; the contract check could not run (RAISES) · e7bd7636a1df749c
  21. get_gaussian_keys helper · level 1 · ran · honoured contract · 1d77d0309d035999
  22. get_knn_faiss helper · level 0 · unverified · 80bf3f1cdd634770
  23. get_slices helper · level 1 · ran · honoured contract · 91283fd5a725ec99
  24. get_uniform_keys helper · level 1 · ran · honoured contract · 3fa144c23db44a7e
  25. mlp helper · level 1 · ran; the contract check could not run (RAISES) · a05791210b07b83d
  26. swig_ptr_from_FloatTensor helper · level 0 · unverified · ded0b9ac1f3c803c
  27. swig_ptr_from_LongTensor helper · level 0 · unverified · 41f2f12e6d5e6d33

harvested Assembly for XLM-R (graph method: XLM-R). 1 brick; individually on a fixture: 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified. The assembly has not been run end-to-end.

one of 7 siblings

  1. PuncXLMRModel entry · level 1 · ran · b8082687538a83e1

harvested Assembly for XLM-R (graph method: XLM-R). 4 bricks; individually on a fixture: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified; the one sample that ran constructed an object rather than computing a result. The assembly has not been run end-to-end.

one of 7 siblings

  1. XLMREncoder entry · level 0 · unverified · 292808e3a5d973ed
  2. BERTEncoder helper · level 0 · unverified · bee8cacb03b2b055
  3. Encoder helper · level 0 · unverified · ce7a4a01c9b960a2
  4. LabelSet helper · level 1 · ran · fc8285a0c6ac6166

Each brick ends with its code_sha256, Syntology's identity for that exact code; click one to copy the get_code call that returns it. For agents, on Syntology's MCP service: assembly(arxiv_id="1911.02116") returns these assemblies with each brick's role and wiring (how to connect).

Tasks

Cross-Lingual TransferLanguage ModelingLanguage ModellingMultilingual NLPRepresentation LearningXLM-R

Datasets

Introduced by this paper, per the archive.

CC100

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

Introduced by this paper: XLM-R

AdamAttentionAttention DropoutBERTBPEDense ConnectionsDropoutLayer NormalizationLinear LayerLinear Warmup With Linear DecayMulti-Head AttentionResidual ConnectionSoftmaxWeight DecayWordPieceXLMXLM-R

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections