Papers › Overview of the SV-Ident 2022 Shared Task on Survey Variable Identification in Social...
Overview of the SV-Ident 2022 Shared Task on Survey Variable Identification in Social Science Publications
Tornike Tsereteli, Yavuz Selim Kartal, Simone Paolo Ponzetto, Andrea Zielinski, Kai Eckert, Philipp Mayr
In this paper, we provide an overview of the SV-Ident shared task as part of the 3rd Workshop on Scholarly Document Processing (SDP) at COLING 2022. In the shared task, participants were provided with a sentence and a vocabulary of variables, and asked to identify which variables, if any, are mentioned in individual sentences from scholarly documents in full text. Two teams made a total of 9 submissions to the shared task leaderboard. While none of the teams improve on the baseline systems, we still draw insights from their submissions. Furthermore, we provide a detailed evaluation. Data and baselines for our shared task are freely available at https://github.com/vadis-project/sv-ident
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Datasets
Introduced by this paper, per the archive.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Variable Detection | SV-Ident | SsciBERT | F1 | 66.1 | #1 of 2 | Archive leaderboard | report |
| Variable Detection | SV-Ident | Sentence-T5 | F1 | 60.17 | #2 of 2 | Archive leaderboard | report |
| Variable Disambiguation | SV-Ident | sentence-transformers/distiluse-base-multilingual-cased-v1 | mAP@10 | 18.93 | #1 of 4 | Archive leaderboard | report |
| Variable Disambiguation | SV-Ident | Sentence-T5 | mAP@10 | 13.59 | #2 of 4 | Archive leaderboard | report |
| Variable Disambiguation | SV-Ident | SPARTA | mAP@10 | 11.27 | #3 of 4 | Archive leaderboard | report |
| Variable Disambiguation | SV-Ident | BM25 | mAP@10 | 9.43 | #4 of 4 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections