{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/better-than-average-paired-evaluation-of-nlp-1","title":"Better than Average: Paired Evaluation of NLP Systems","arxiv_id":"2110.10746","date":"2021-10-20","proceeding":"ACL 2021 5","authors":["Maxime Peyrard","Wei Zhao","Steffen Eger","Robert West"],"abstract":"Evaluation in NLP is usually done by comparing the scores of competing systems independently averaged over a common set of test instances. In this work, we question the use of averages for aggregating evaluation scores into a final number used to decide which system is best, since the average, as well as alternatives such as the median, ignores the pairing arising from the fact that systems are evaluated on the same test instances. We illustrate the importance of taking the instance-level pairing of evaluation scores into account and demonstrate, both theoretically and empirically, the advantages of aggregation methods based on pairwise comparisons, such as the Bradley-Terry (BT) model, a mechanism based on the estimated probability that a given system scores better than another on the test set. By re-evaluating 296 real NLP evaluation setups across four tasks and 18 evaluation metrics, we show that the choice of aggregation mechanism matters and yields different conclusions as to which systems are state of the art in about 30% of the setups. To facilitate the adoption of pairwise evaluation, we release a practical tool for performing the full analysis of evaluation scores with the mean, median, BT, and two variants of BT (Elo and TrueSkill), alongside functionality for appropriate statistical testing.","url_abs":"https://arxiv.org/abs/2110.10746v1","url_pdf":"https://arxiv.org/pdf/2110.10746v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"better-than-average-paired-evaluation-of-nlp-1","repo_url":"https://github.com/epfl-dlab/bt-eval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2110.10746","atlas_url":"https://app.syntology.ai/?focus=2110.10746","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.10746"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/epfl-dlab/bt-eval","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/epfl-dlab/BT-eval","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"47287d5c58ead781","entry":"BT","repo":"epfl-dlab/BT-eval","repo_kind":"official","path":"bt.py","file_url":"https://github.com/epfl-dlab/BT-eval/blob/HEAD/bt.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"47287d5c58ead781"}},{"code_sha256_prefix":"a99fa076585ad2fd","entry":"ELO","repo":"epfl-dlab/BT-eval","repo_kind":"official","path":"Elo.py","file_url":"https://github.com/epfl-dlab/BT-eval/blob/HEAD/Elo.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a99fa076585ad2fd"}},{"code_sha256_prefix":"97dfdfcb4b432e36","entry":"add_outliers","repo":"epfl-dlab/BT-eval","repo_kind":"official","path":"simulations.py","file_url":"https://github.com/epfl-dlab/BT-eval/blob/HEAD/simulations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"97dfdfcb4b432e36"}},{"code_sha256_prefix":"314f397513505006","entry":"sample_dataset","repo":"epfl-dlab/BT-eval","repo_kind":"official","path":"simulations.py","file_url":"https://github.com/epfl-dlab/BT-eval/blob/HEAD/simulations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"314f397513505006"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}