</> Code with Papers by Syntology

Papers › Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in...

Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema

16 Apr 2021EMNLP 2021 11arXiv:2104.08161archive 2025-07-28

Yanai Elazar, Hongming Zhang, Yoav Goldberg, Dan Roth

The Winograd Schema (WS) has been proposed as a test for measuring commonsense capabilities of models. Recently, pre-trained language model-based approaches have boosted performance on some WS benchmarks but the source of improvement is still not clear. This paper suggests that the apparent progress on WS may not necessarily reflect progress in commonsense reasoning. To support this claim, we first show that the current evaluation method of WS is sub-optimal and propose a modification that uses twin sentences for evaluation. We also propose two new baselines that indicate the existence of artifacts in WS benchmarks. We then develop a method for evaluating WS-like sentences in a zero-shot setting to account for the commonsense reasoning abilities acquired during the pretraining and observe that popular language models perform randomly in this setting when using our more strict evaluation. We conclude that the observed progress is mostly due to the use of supervision in training WS models, which is not likely to successfully support all the required commonsense reasoning skills and knowledge.

PaperPDFConference PDF

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Artifact DetectionBias DetectionCommon Sense ReasoningCoreference ResolutionDisentanglementLanguage ModelingLanguage Modelling

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Common Sense Reasoning WinoGrande ALBERT-xxlarge 235M Accuracy 58.7 #54 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande RoBERTa-base 125M Accuracy 56.3 #59 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande BERT-large 345M Accuracy 55.6 #62 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande RoBERTa-large 355M Accuracy 54.9 #66 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande BERT-base 110M Accuracy 53.1 #69 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande ALBERT-base 11M Accuracy 52.8 #70 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande Random baseline Accuracy 50 #76 of 77 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge ALBERT-xxlarge 235M Accuracy 78.8 #24 of 82 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge RoBERTa-large 354M Accuracy 73.9 #28 of 82 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge RoBERTa-base 125M Accuracy 63 #48 of 82 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge BERT-large 340M Accuracy 61.4 #56 of 82 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge BERT-base 110M Accuracy 56.5 #68 of 82 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge ALBERT-base 11M Accuracy 55.4 #70 of 82 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge Random chance baseline Accuracy 50 #77 of 82 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections