Papers › WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context
WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context
Anna Breit, Artem Revenko, Kiamehr Rezaee, Mohammad Taher Pilehvar, Jose Camacho-Collados
We present WiC-TSV, a new multi-domain evaluation benchmark for Word Sense Disambiguation. More specifically, we introduce a framework for Target Sense Verification of Words in Context which grounds its uniqueness in the formulation as a binary classification task thus being independent of external sense inventories, and the coverage of various domains. This makes the dataset highly flexible for the evaluation of a diverse set of models and systems in and across domains. WiC-TSV provides three different evaluation settings, depending on the input signals provided to the model. We set baseline performance on the dataset using state-of-the-art language models. Experimental results show that even though these models can perform decently on the task, there remains a gap between machine and human performance, especially in out-of-domain settings. WiC-TSV data is available at https://competitions.codalab.org/competitions/23683
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Datasets
Introduced by this paper, per the archive.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Entity Linking | WiC-TSV | Bert-base | Task 1 Accuracy: all | 75.3 | #4 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Bert-base | Task 1 Accuracy: domain specific | 77.9 | #4 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Bert-base | Task 1 Accuracy: general purpose | 73.3 | #4 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Bert-base | Task 2 Accuracy: all | 71.7 | #4 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Bert-base | Task 2 Accuracy: domain specific | 74.7 | #4 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Bert-base | Task 2 Accuracy: general purpose | 68.6 | #4 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Bert-base | Task 3 Accuracy: all | 76.6 | #4 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Bert-base | Task 3 Accuracy: domain specific | 80.4 | #4 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Bert-base | Task 3 Accuracy: general purpose | 73.5 | #4 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Unsupervised Bert | Task 1 Accuracy: all | 54.4 | #5 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Unsupervised Bert | Task 1 Accuracy: domain specific | 60.6 | #5 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Unsupervised Bert | Task 1 Accuracy: general purpose | 49.2 | #5 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Unsupervised Bert | Task 2 Accuracy: all | 62.8 | #5 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Unsupervised Bert | Task 2 Accuracy: domain specific | 69.1 | #5 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Unsupervised Bert | Task 2 Accuracy: general purpose | 57.6 | #5 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Unsupervised Bert | Task 3 Accuracy: all | 60.5 | #5 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Unsupervised Bert | Task 3 Accuracy: domain specific | 67.9 | #5 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Unsupervised Bert | Task 3 Accuracy: general purpose | 54.4 | #5 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | FastText | Task 1 Accuracy: all | 53.7 | #6 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | FastText | Task 1 Accuracy: domain specific | 50.6 | #6 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | FastText | Task 1 Accuracy: general purpose | 56.2 | #6 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | FastText | Task 2 Accuracy: all | 52.7 | #6 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | FastText | Task 2 Accuracy: domain specific | 47.7 | #6 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | FastText | Task 2 Accuracy: general purpose | 56.8 | #6 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | FastText | Task 3 Accuracy: all | 53.4 | #6 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | FastText | Task 3 Accuracy: domain specific | 49.0 | #6 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | FastText | Task 3 Accuracy: general purpose | 57.1 | #6 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | All true | Task 1 Accuracy: all | 50.8 | #7 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | All true | Task 1 Accuracy: domain specific | 47.0 | #7 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | All true | Task 1 Accuracy: general purpose | 53.8 | #7 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | All true | Task 2 Accuracy: all | 50.8 | #7 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | All true | Task 2 Accuracy: domain specific | 47.0 | #7 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | All true | Task 2 Accuracy: general purpose | 53.8 | #7 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | All true | Task 3 Accuracy: all | 50.8 | #7 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | All true | Task 3 Accuracy: domain specific | 47.0 | #7 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | All true | Task 3 Accuracy: general purpose | 53.8 | #7 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Human | Task 3 Accuracy: all | 85.3 | #8 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Human | Task 3 Accuracy: domain specific | 89.2 | #8 of 8 | Archive leaderboard | report |
| Entity Linking | WiC-TSV | Human | Task 3 Accuracy: general purpose | 82.1 | #8 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Bert-base | Task 1 Accuracy: all | 75.3 | #4 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Bert-base | Task 1 Accuracy: domain specific | 77.9 | #4 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Bert-base | Task 1 Accuracy: general purpose | 73.3 | #4 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Bert-base | Task 2 Accuracy: all | 71.7 | #4 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Bert-base | Task 2 Accuracy: domain specific | 74.7 | #4 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Bert-base | Task 2 Accuracy: general purpose | 68.6 | #4 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Bert-base | Task 3 Accuracy: all | 76.6 | #4 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Bert-base | Task 3 Accuracy: domain specific | 80.4 | #4 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Bert-base | Task 3 Accuracy: general purpose | 73.5 | #4 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Unsupervised Bert | Task 1 Accuracy: all | 54.4 | #5 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Unsupervised Bert | Task 1 Accuracy: domain specific | 60.6 | #5 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Unsupervised Bert | Task 1 Accuracy: general purpose | 49.2 | #5 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Unsupervised Bert | Task 2 Accuracy: all | 62.8 | #5 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Unsupervised Bert | Task 2 Accuracy: domain specific | 69.1 | #5 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Unsupervised Bert | Task 2 Accuracy: general purpose | 57.6 | #5 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Unsupervised Bert | Task 3 Accuracy: all | 60.5 | #5 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Unsupervised Bert | Task 3 Accuracy: domain specific | 67.9 | #5 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Unsupervised Bert | Task 3 Accuracy: general purpose | 54.4 | #5 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | FastText | Task 1 Accuracy: all | 53.7 | #6 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | FastText | Task 1 Accuracy: domain specific | 50.6 | #6 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | FastText | Task 1 Accuracy: general purpose | 56.2 | #6 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | FastText | Task 2 Accuracy: all | 52.7 | #6 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | FastText | Task 2 Accuracy: domain specific | 47.7 | #6 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | FastText | Task 2 Accuracy: general purpose | 56.8 | #6 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | FastText | Task 3 Accuracy: all | 53.4 | #6 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | FastText | Task 3 Accuracy: domain specific | 49.0 | #6 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | FastText | Task 3 Accuracy: general purpose | 57.1 | #6 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | All true | Task 1 Accuracy: all | 50.8 | #7 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | All true | Task 1 Accuracy: domain specific | 47.0 | #7 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | All true | Task 1 Accuracy: general purpose | 53.8 | #7 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | All true | Task 2 Accuracy: all | 50.8 | #7 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | All true | Task 2 Accuracy: domain specific | 47.0 | #7 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | All true | Task 2 Accuracy: general purpose | 53.8 | #7 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | All true | Task 3 Accuracy: all | 50.8 | #7 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | All true | Task 3 Accuracy: domain specific | 47.0 | #7 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | All true | Task 3 Accuracy: general purpose | 53.8 | #7 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Human | Task 3 Accuracy: all | 85.3 | #8 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Human | Task 3 Accuracy: domain specific | 89.2 | #8 of 8 | Archive leaderboard | report |
| Word Sense Disambiguation | WiC-TSV | Human | Task 3 Accuracy: general purpose | 82.1 | #8 of 8 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
AdamAttentionAttention DropoutBERTDense ConnectionsDropoutLayer NormalizationLinear LayerLinear Warmup With Linear DecayMulti-Head AttentionResidual ConnectionSoftmaxWeight DecayWordPiece
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections