{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sling-sino-linguistic-evaluation-of-large","title":"SLING: Sino Linguistic Evaluation of Large Language Models","arxiv_id":"2210.11689","date":"2022-10-21","proceeding":null,"authors":["Yixiao Song","Kalpesh Krishna","Rajesh Bhatt","Mohit Iyyer"],"abstract":"To understand what kinds of linguistic knowledge are encoded by pretrained Chinese language models (LMs), we introduce the benchmark of Sino LINGuistics (SLING), which consists of 38K minimal sentence pairs in Mandarin Chinese grouped into 9 high-level linguistic phenomena. Each pair demonstrates the acceptability contrast of a specific syntactic or semantic phenomenon (e.g., The keys are lost vs. The keys is lost), and an LM should assign lower perplexity to the acceptable sentence. In contrast to the CLiMP dataset (Xiang et al., 2021), which also contains Chinese minimal pairs and was created by translating the vocabulary of the English BLiMP dataset, the minimal pairs in SLING are derived primarily by applying syntactic and lexical transformations to naturally-occurring, linguist-annotated sentences from the Chinese Treebank 9.0, thus addressing severe issues in CLiMP's data generation process. We test 18 publicly available pretrained monolingual (e.g., BERT-base-zh, CPM) and multi-lingual (e.g., mT5, XLM) language models on SLING. Our experiments show that the average accuracy for LMs is far below human performance (69.7% vs. 97.1%), while BERT-base-zh achieves the highest accuracy (84.8%) of all tested LMs, even much larger ones. Additionally, we find that most LMs have a strong gender and number (singular/plural) bias, and they perform better on local phenomena than hierarchical ones.","url_abs":"https://arxiv.org/abs/2210.11689v1","url_pdf":"https://arxiv.org/pdf/2210.11689v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sling-sino-linguistic-evaluation-of-large","repo_url":"https://github.com/yixiao-song/sling_data_code","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"sentence","task_name":"Sentence"}],"methods":[{"method_slug":"adafactor","method_name":"Adafactor"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"glu","method_name":"Gated Linear Unit"},{"method_slug":"inverse-square-root-schedule","method_name":"Inverse Square Root Schedule"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sentencepiece","method_name":"SentencePiece"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"t5","method_name":"T5"},{"method_slug":"test","method_name":"Test"},{"method_slug":"mt5","method_name":"mT5"}],"datasets_introduced":[{"slug":"sling","name":"SLING","full_name":"Sino LINGuistics"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.11689","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.11689"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/fxsjy/jieba","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Yixiao-Song/SLING_Data_Code","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"48860298c011eb23","entry":"evaluate_response","repo":"Yixiao-Song/SLING_Data_Code","repo_kind":"official","path":"SLING_Code/gpt3_sling.py","file_url":"https://github.com/Yixiao-Song/SLING_Data_Code/blob/HEAD/SLING_Code/gpt3_sling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48860298c011eb23"}},{"code_sha256_prefix":"5cc704944bd4d2c9","entry":"get_ppl","repo":"Yixiao-Song/SLING_Data_Code","repo_kind":"official","path":"SLING_Code/utils.py","file_url":"https://github.com/Yixiao-Song/SLING_Data_Code/blob/HEAD/SLING_Code/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cc704944bd4d2c9"}},{"code_sha256_prefix":"2eff6ef3426dba90","entry":"get_response","repo":"Yixiao-Song/SLING_Data_Code","repo_kind":"official","path":"SLING_Code/gpt3_sling.py","file_url":"https://github.com/Yixiao-Song/SLING_Data_Code/blob/HEAD/SLING_Code/gpt3_sling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2eff6ef3426dba90"}},{"code_sha256_prefix":"7e07d3bc59036cf4","entry":"get_token_log_prob","repo":"Yixiao-Song/SLING_Data_Code","repo_kind":"official","path":"SLING_Code/utils.py","file_url":"https://github.com/Yixiao-Song/SLING_Data_Code/blob/HEAD/SLING_Code/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e07d3bc59036cf4"}},{"code_sha256_prefix":"897e1a480e3ffd4d","entry":"jieba_cnt","repo":"Yixiao-Song/SLING_Data_Code","repo_kind":"official","path":"count_sling_vocab.py","file_url":"https://github.com/Yixiao-Song/SLING_Data_Code/blob/HEAD/count_sling_vocab.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"897e1a480e3ffd4d"}},{"code_sha256_prefix":"54522b02308a3a16","entry":"load_climp_dataset","repo":"Yixiao-Song/SLING_Data_Code","repo_kind":"official","path":"SLING_Code/utils.py","file_url":"https://github.com/Yixiao-Song/SLING_Data_Code/blob/HEAD/SLING_Code/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54522b02308a3a16"}},{"code_sha256_prefix":"288f9b462debe734","entry":"load_climp_dataset","repo":"Yixiao-Song/SLING_Data_Code","repo_kind":"official","path":"SLING_Code/Run_CLiMP/PanGu_climp.py","file_url":"https://github.com/Yixiao-Song/SLING_Data_Code/blob/HEAD/SLING_Code/Run_CLiMP/PanGu_climp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"288f9b462debe734"}},{"code_sha256_prefix":"daf29ee91adcf61e","entry":"perplexity","repo":"Yixiao-Song/SLING_Data_Code","repo_kind":"official","path":"SLING_Code/gpt3_sling.py","file_url":"https://github.com/Yixiao-Song/SLING_Data_Code/blob/HEAD/SLING_Code/gpt3_sling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"daf29ee91adcf61e"}},{"code_sha256_prefix":"c5b0960e041cfacf","entry":"run_lstm","repo":"Yixiao-Song/SLING_Data_Code","repo_kind":"official","path":"SLING_Code/lstm_sling.py","file_url":"https://github.com/Yixiao-Song/SLING_Data_Code/blob/HEAD/SLING_Code/lstm_sling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5b0960e041cfacf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}