{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unreasonable-effectiveness-of-rule-based","title":"Unreasonable Effectiveness of Rule-Based Heuristics in Solving Russian SuperGLUE Tasks","arxiv_id":"2105.01192","date":"2021-05-03","proceeding":null,"authors":["Tatyana Iazykova","Denis Kapelyushnik","Olga Bystrova","Andrey Kutuzov"],"abstract":"Leader-boards like SuperGLUE are seen as important incentives for active development of NLP, since they provide standard benchmarks for fair comparison of modern language models. They have driven the world's best engineering teams as well as their resources to collaborate and solve a set of tasks for general language understanding. Their performance scores are often claimed to be close to or even higher than the human performance. These results encouraged more thorough analysis of whether the benchmark datasets featured any statistical cues that machine learning based language models can exploit. For English datasets, it was shown that they often contain annotation artifacts. This allows solving certain tasks with very simple rules and achieving competitive rankings. In this paper, a similar analysis was done for the Russian SuperGLUE (RSG), a recently published benchmark set and leader-board for Russian natural language understanding. We show that its test datasets are vulnerable to shallow heuristics. Often approaches based on simple rules outperform or come close to the results of the notorious pre-trained language models like GPT-3 or BERT. It is likely (as the simplest explanation) that a significant part of the SOTA models performance in the RSG leader-board is due to exploiting these shallow heuristics and that has nothing in common with real language understanding. We provide a set of recommendations on how to improve these datasets, making the RSG leader-board even more representative of the real progress in Russian NLU.","url_abs":"https://arxiv.org/abs/2105.01192v1","url_pdf":"https://arxiv.org/pdf/2105.01192v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"word-sense-disambiguation","task_name":"Word Sense Disambiguation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/common-sense-reasoning-on-parus","task":"Common Sense Reasoning","dataset":"PARus","model":"majority_class","rank_in_archive_order":17,"of":22,"metrics":{"Accuracy":"0.498"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-parus","task":"Common Sense Reasoning","dataset":"PARus","model":"Random weighted","rank_in_archive_order":20,"of":22,"metrics":{"Accuracy":"0.48"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-parus","task":"Common Sense Reasoning","dataset":"PARus","model":"heuristic majority","rank_in_archive_order":21,"of":22,"metrics":{"Accuracy":"0.478"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-rwsd","task":"Common Sense Reasoning","dataset":"RWSD","model":"Random weighted","rank_in_archive_order":3,"of":22,"metrics":{"Accuracy":"0.597"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-rwsd","task":"Common Sense Reasoning","dataset":"RWSD","model":"heuristic majority","rank_in_archive_order":17,"of":22,"metrics":{"Accuracy":"0.669"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-rwsd","task":"Common Sense Reasoning","dataset":"RWSD","model":"majority_class","rank_in_archive_order":20,"of":22,"metrics":{"Accuracy":"0.669"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-rucos","task":"Common Sense Reasoning","dataset":"RuCoS","model":"heuristic majority","rank_in_archive_order":15,"of":22,"metrics":{"Average F1":"0.26","EM ":"0.257"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-rucos","task":"Common Sense Reasoning","dataset":"RuCoS","model":"Random weighted","rank_in_archive_order":17,"of":22,"metrics":{"Average F1":"0.25","EM ":"0.247"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-rucos","task":"Common Sense Reasoning","dataset":"RuCoS","model":"majority_class","rank_in_archive_order":18,"of":22,"metrics":{"Average F1":"0.25","EM ":"0.247"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-lidirus","task":"Natural Language Inference","dataset":"LiDiRus","model":"heuristic majority","rank_in_archive_order":13,"of":22,"metrics":{"MCC":"0.147"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-lidirus","task":"Natural Language Inference","dataset":"LiDiRus","model":"Random weighted","rank_in_archive_order":20,"of":22,"metrics":{"MCC":"0"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-lidirus","task":"Natural Language Inference","dataset":"LiDiRus","model":"majority_class","rank_in_archive_order":21,"of":22,"metrics":{"MCC":"0"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rcb","task":"Natural Language Inference","dataset":"RCB","model":"heuristic majority","rank_in_archive_order":6,"of":22,"metrics":{"Accuracy":"0.438","Average F1":"0.4"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rcb","task":"Natural Language Inference","dataset":"RCB","model":"Random weighted","rank_in_archive_order":17,"of":22,"metrics":{"Accuracy":"0.374","Average F1":"0.319"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rcb","task":"Natural Language Inference","dataset":"RCB","model":"majority_class","rank_in_archive_order":22,"of":22,"metrics":{"Accuracy":"0.484","Average F1":"0.217"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-terra","task":"Natural Language Inference","dataset":"TERRa","model":"heuristic majority","rank_in_archive_order":17,"of":22,"metrics":{"Accuracy":"0.549"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-terra","task":"Natural Language Inference","dataset":"TERRa","model":"majority_class","rank_in_archive_order":18,"of":22,"metrics":{"Accuracy":"0.513"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-terra","task":"Natural Language Inference","dataset":"TERRa","model":"Random weighted","rank_in_archive_order":21,"of":22,"metrics":{"Accuracy":"0.483"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-danetqa","task":"Question Answering","dataset":"DaNetQA","model":"heuristic majority","rank_in_archive_order":11,"of":22,"metrics":{"Accuracy":"0.642"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-danetqa","task":"Question Answering","dataset":"DaNetQA","model":"Random weighted","rank_in_archive_order":21,"of":22,"metrics":{"Accuracy":"0.52"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-danetqa","task":"Question Answering","dataset":"DaNetQA","model":"majority_class","rank_in_archive_order":22,"of":22,"metrics":{"Accuracy":"0.503"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-muserc","task":"Reading Comprehension","dataset":"MuSeRC","model":"heuristic majority","rank_in_archive_order":15,"of":22,"metrics":{"Average F1":"0.671","EM ":"0.237"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-muserc","task":"Reading Comprehension","dataset":"MuSeRC","model":"Random weighted","rank_in_archive_order":21,"of":22,"metrics":{"Average F1":"0.45","EM ":"0.071"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-muserc","task":"Reading Comprehension","dataset":"MuSeRC","model":"majority_class","rank_in_archive_order":22,"of":22,"metrics":{"Average F1":"0.0","EM ":" 0.0"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-russe","task":"Word Sense Disambiguation","dataset":"RUSSE","model":"heuristic majority","rank_in_archive_order":15,"of":22,"metrics":{"Accuracy":"0.595"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-russe","task":"Word Sense Disambiguation","dataset":"RUSSE","model":"majority_class","rank_in_archive_order":18,"of":22,"metrics":{"Accuracy":"0.587"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-russe","task":"Word Sense Disambiguation","dataset":"RUSSE","model":"Random weighted","rank_in_archive_order":22,"of":22,"metrics":{"Accuracy":"0.528"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}