{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/inherent-trade-offs-between-diversity-and","title":"Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks","arxiv_id":"2405.01719","date":"2024-05-02","proceeding":null,"authors":["Guanhua Zhang","Moritz Hardt"],"abstract":"We examine multi-task benchmarks in machine learning through the lens of social choice theory. We draw an analogy between benchmarks and electoral systems, where models are candidates and tasks are voters. This suggests a distinction between cardinal and ordinal benchmark systems. The former aggregate numerical scores into one model ranking; the latter aggregate rankings for each task. We apply Arrow's impossibility theorem to ordinal benchmarks to highlight the inherent limitations of ordinal systems, particularly their sensitivity to the inclusion of irrelevant models. Inspired by Arrow's theorem, we empirically demonstrate a strong trade-off between diversity and sensitivity to irrelevant changes in existing multi-task benchmarks. Our result is based on new quantitative measures of diversity and sensitivity that we introduce. Sensitivity quantifies the impact that irrelevant changes to tasks have on a benchmark. Diversity captures the degree of disagreement in model rankings across tasks. We develop efficient approximation algorithms for both measures, as exact computation is computationally challenging. Through extensive experiments on seven cardinal benchmarks and eleven ordinal benchmarks, we demonstrate a clear trade-off between diversity and stability: The more diverse a multi-task benchmark, the more sensitive to trivial changes it is. Additionally, we show that the aggregated rankings of existing benchmarks are highly unstable under irrelevant changes. The codes and data are available at https://socialfoundations.github.io/benchbench/.","url_abs":"https://arxiv.org/abs/2405.01719v2","url_pdf":"https://arxiv.org/pdf/2405.01719v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"inherent-trade-offs-between-diversity-and","repo_url":"https://github.com/socialfoundations/benchbench","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"sensitivity","task_name":"Sensitivity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.01719","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.01719"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/socialfoundations/benchbench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":9,"unverified":3},"by_repo_kind":{"official":{"samples":12,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1100ae4be1bf2756","entry":"appr_rank_diff","repo":"socialfoundations/benchbench","repo_kind":"official","path":"benchbench/measures/cardinal.py","file_url":"https://github.com/socialfoundations/benchbench/blob/HEAD/benchbench/measures/cardinal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1100ae4be1bf2756"}},{"code_sha256_prefix":"85be7ca4d40885ca","entry":"appr_rank_diff","repo":"socialfoundations/benchbench","repo_kind":"official","path":"benchbench/measures/ordinal.py","file_url":"https://github.com/socialfoundations/benchbench/blob/HEAD/benchbench/measures/ordinal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85be7ca4d40885ca"}},{"code_sha256_prefix":"d8821cf389d6c663","entry":"get_combinations","repo":"socialfoundations/benchbench","repo_kind":"official","path":"benchbench/utils/base.py","file_url":"https://github.com/socialfoundations/benchbench/blob/HEAD/benchbench/utils/base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8821cf389d6c663"}},{"code_sha256_prefix":"baefae3daf59272f","entry":"get_kendall_tau","repo":"socialfoundations/benchbench","repo_kind":"official","path":"benchbench/utils/metric.py","file_url":"https://github.com/socialfoundations/benchbench/blob/HEAD/benchbench/utils/metric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"baefae3daf59272f"}},{"code_sha256_prefix":"9f7f918c7b987faa","entry":"get_kendall_w","repo":"socialfoundations/benchbench","repo_kind":"official","path":"benchbench/utils/metric.py","file_url":"https://github.com/socialfoundations/benchbench/blob/HEAD/benchbench/utils/metric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9f7f918c7b987faa"}},{"code_sha256_prefix":"13665191512c83ca","entry":"get_rank_diff","repo":"socialfoundations/benchbench","repo_kind":"official","path":"benchbench/utils/metric.py","file_url":"https://github.com/socialfoundations/benchbench/blob/HEAD/benchbench/utils/metric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"13665191512c83ca"}},{"code_sha256_prefix":"4837e4c35f8872e9","entry":"get_selected_win_rate","repo":"socialfoundations/benchbench","repo_kind":"official","path":"benchbench/measures/ordinal.py","file_url":"https://github.com/socialfoundations/benchbench/blob/HEAD/benchbench/measures/ordinal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4837e4c35f8872e9"}},{"code_sha256_prefix":"706af7256ca3571e","entry":"is_int","repo":"socialfoundations/benchbench","repo_kind":"official","path":"benchbench/utils/base.py","file_url":"https://github.com/socialfoundations/benchbench/blob/HEAD/benchbench/utils/base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"706af7256ca3571e"}},{"code_sha256_prefix":"27f819180313d7a4","entry":"is_number","repo":"socialfoundations/benchbench","repo_kind":"official","path":"benchbench/utils/base.py","file_url":"https://github.com/socialfoundations/benchbench/blob/HEAD/benchbench/utils/base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27f819180313d7a4"}},{"code_sha256_prefix":"3a8ca465f7f286ad","entry":"get_diversity","repo":"socialfoundations/benchbench","repo_kind":"official","path":"benchbench/measures/cardinal.py","file_url":"https://github.com/socialfoundations/benchbench/blob/HEAD/benchbench/measures/cardinal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a8ca465f7f286ad"}},{"code_sha256_prefix":"2a977e4f402ff472","entry":"get_sensitivity","repo":"socialfoundations/benchbench","repo_kind":"official","path":"benchbench/measures/cardinal.py","file_url":"https://github.com/socialfoundations/benchbench/blob/HEAD/benchbench/measures/cardinal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a977e4f402ff472"}},{"code_sha256_prefix":"7a936e8599af8beb","entry":"get_sensitivity","repo":"socialfoundations/benchbench","repo_kind":"official","path":"benchbench/measures/ordinal.py","file_url":"https://github.com/socialfoundations/benchbench/blob/HEAD/benchbench/measures/ordinal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a936e8599af8beb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}