Semantic Textual Similarity

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

13 leaderboard tables shown for this task, 13 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted. 10 shown of 13 until expanded.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

19 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

2 subtasks in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 693 papers with code (2,381 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  11 Oct 2018 534 repositories listed Syntology ran 204 of 659 samples · 455 unverified · 149 pointer-only (licence)
  We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers.
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
  26 Jul 2019 67 repositories listed Syntology ran 22 of 48 samples · 26 unverified · 23 pointer-only (licence)
  Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging.
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
  27 Aug 2019 64 repositories listed Syntology ran 20 of 58 samples · 38 unverified · 11 pointer-only (licence)
  However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10, 000 sentences requires about 50 million inference…
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  23 Oct 2019 57 repositories listed Syntology ran 2 of 31 samples · 29 unverified
  Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP).
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  26 Sep 2019 48 repositories listed Syntology ran 46 of 126 samples · 80 unverified · 22 pointer-only (licence)
  Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks.
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
  2 Oct 2019 37 repositories listed Syntology ran 19 of 27 samples · 8 unverified
  As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or…
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
  19 Jun 2019 27 repositories listed Syntology ran 10 of 24 samples · 14 unverified · 3 pointer-only (licence)
  With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling.
• Universal Sentence Encoder
  29 Mar 2018 24 repositories listed Syntology ran 1 of 22 samples · 21 unverified · 1 pointer-only (licence)
  For both variants, we investigate and report the relationship between model complexity, resource consumption, the availability of transfer task training data, and task performance.
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
  18 Apr 2021 23 repositories listed Syntology ran 17 of 30 samples · 13 unverified · 19 pointer-only (licence)
  This paper presents SimCSE, a simple contrastive learning framework that greatly advances state-of-the-art sentence embeddings.
• Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
  5 May 2017 23 repositories listed Syntology ran 6 of 7 samples · 1 unverified · 7 pointer-only (licence)
  Many modern NLP systems rely on word embeddings, previously trained in an unsupervised manner on large corpora, as base features.
• ERNIE: Enhanced Representation through Knowledge Integration
  19 Apr 2019 19 repositories listed Syntology ran 0 of 7 samples · 7 unverified
  We present a novel language representation model enhanced by knowledge called ERNIE (Enhanced Representation through kNowledge IntEgration).
• Big Bird: Transformers for Longer Sequences
  28 Jul 2020 14 repositories listed Syntology ran 10 of 15 samples · 5 unverified · 11 pointer-only (licence)
  To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this quadratic dependency to linear.
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
  5 Jun 2020 14 repositories listed Syntology ran 4 of 13 samples · 9 unverified · 3 pointer-only (licence)
  Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks.
• Improving Language Understanding by Generative Pre-Training
  11 Jun 2018 13 repositories listed
  We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task.
• FNet: Mixing Tokens with Fourier Transforms
  9 May 2021 12 repositories listed Syntology ran 2 of 2 samples · 0 unverified · 1 pointer-only (licence)
  At longer input lengths, our FNet model is significantly faster: when compared to the "efficient" Transformers on the Long Range Arena benchmark, FNet matches the accuracy of the most accurate models, while outpacing…
• Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention
  7 Feb 2021 10 repositories listed Syntology ran 1 of 2 samples · 1 unverified · 1 pointer-only (licence)
  The scalability of Nystr\"{o}mformer enables application to longer sequences with thousands of tokens.
• TinyBERT: Distilling BERT for Natural Language Understanding
  23 Sep 2019 10 repositories listed Syntology ran 0 of 4 samples · 4 unverified · 4 pointer-only (licence)
  To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based…
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
  6 Apr 2020 7 repositories listed
  Then, we conduct knowledge transfer from this teacher to MobileBERT.
• TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning
  14 Apr 2021 6 repositories listed Syntology ran 0 of 4 samples · 4 unverified
  Learning sentence embeddings often requires a large amount of labeled data.
• Language-agnostic BERT Sentence Embedding
  3 Jul 2020 6 repositories listed
  While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning (Reimers and Gurevych, 2019), BERT based cross-lingual sentence embeddings…
• SqueezeBERT: What can computer vision teach NLP about efficient neural networks?
  19 Jun 2020 6 repositories listed Syntology ran 0 of 1 samples · 1 unverified
  Humans read and write hundreds of billions of messages every day.
• SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
  8 Nov 2019 6 repositories listed Syntology ran 6 of 8 samples · 2 unverified · 1 pointer-only (licence)
  However, due to limited data resources from downstream tasks and the extremely large capacity of pre-trained models, aggressive fine-tuning often causes the adapted model to overfit the data of downstream tasks and…
• SpanBERT: Improving Pre-training by Representing and Predicting Spans
  24 Jul 2019 6 repositories listed Syntology ran 3 of 15 samples · 12 unverified · 4 pointer-only (licence)
  We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text.
• MTEB: Massive Text Embedding Benchmark
  13 Oct 2022 5 repositories listed Syntology ran 3 of 13 samples · 10 unverified
  MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages.
• RealFormer: Transformer Likes Residual Attention
  21 Dec 2020 5 repositories listed
  Transformer is the backbone of modern NLP models.
• Q8BERT: Quantized 8Bit BERT
  14 Oct 2019 5 repositories listed Syntology ran 3 of 11 samples · 8 unverified · 3 pointer-only (licence)
  Recently, pre-trained Transformer based language models such as BERT and GPT, have shown great improvement in many Natural Language Processing (NLP) tasks.
• MedSTS: A Resource for Clinical Semantic Textual Similarity
  28 Aug 2018 5 repositories listed
  A subset of MedSTS (MedSTS_ann) containing 1, 068 sentence pairs was annotated by two medical experts with semantic similarity scores of 0-5 (low to high similarity).
• One Embedder, Any Task: Instruction-Finetuned Text Embeddings
  19 Dec 2022 4 repositories listed Syntology ran 10 of 20 samples · 10 unverified
  Our analysis suggests that INSTRUCTOR is robust to changes in instructions, and that instruction finetuning mitigates the challenge of training a single model on diverse datasets.
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
  15 Aug 2022 4 repositories listed Syntology ran 2 of 5 samples · 3 unverified
  We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance.
• KLUE: Korean Language Understanding Evaluation
  20 May 2021 4 repositories listed
  We introduce Korean Language Understanding Evaluation (KLUE) benchmark.

Syntology lines on 24 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.