{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-curse-of-performance-instability-in","title":"The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions","arxiv_id":"2004.13606","date":"2020-04-28","proceeding":"EMNLP 2020 11","authors":["Xiang Zhou","Yixin Nie","Hao Tan","Mohit Bansal"],"abstract":"We find that the performance of state-of-the-art models on Natural Language Inference (NLI) and Reading Comprehension (RC) analysis/stress sets can be highly unstable. This raises three questions: (1) How will the instability affect the reliability of the conclusions drawn based on these analysis sets? (2) Where does this instability come from? (3) How should we handle this instability and what are some potential solutions? For the first question, we conduct a thorough empirical study over analysis sets and find that in addition to the unstable final performance, the instability exists all along the training curve. We also observe lower-than-expected correlations between the analysis validation set and standard validation set, questioning the effectiveness of the current model-selection routine. Next, to answer the second question, we give both theoretical explanations and empirical evidence regarding the source of the instability, demonstrating that the instability mainly comes from high inter-example correlations within analysis sets. Finally, for the third question, we discuss an initial attempt to mitigate the instability and suggest guidelines for future work such as reporting the decomposed variance for more interpretable results and fair comparison across models. Our code is publicly available at: https://github.com/owenzx/InstabilityAnalysis","url_abs":"https://arxiv.org/abs/2004.13606v2","url_pdf":"https://arxiv.org/pdf/2004.13606v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-curse-of-performance-instability-in","repo_url":"https://github.com/owenzx/InstabilityAnalysis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"model-selection","task_name":"Model Selection"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2004.13606","atlas_url":"https://app.syntology.ai/?focus=2004.13606","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.13606"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/owenzx/InstabilityAnalysis","reach":null}],"summary":{"ran_honours":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1ffeeca045375edf","entry":"get_cov_matrix","repo":"owenzx/InstabilityAnalysis","repo_kind":"official","path":"variance_report.py","file_url":"https://github.com/owenzx/InstabilityAnalysis/blob/HEAD/variance_report.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ffeeca045375edf"}},{"code_sha256_prefix":"99f1eb683cfdb51a","entry":"get_estimate_cov","repo":"owenzx/InstabilityAnalysis","repo_kind":"official","path":"variance_report.py","file_url":"https://github.com/owenzx/InstabilityAnalysis/blob/HEAD/variance_report.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99f1eb683cfdb51a"}},{"code_sha256_prefix":"8eca80b84eac3bf6","entry":"get_result_from_file","repo":"owenzx/InstabilityAnalysis","repo_kind":"official","path":"variance_report.py","file_url":"https://github.com/owenzx/InstabilityAnalysis/blob/HEAD/variance_report.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8eca80b84eac3bf6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}