{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/suremap-simultaneous-mean-estimation-for","title":"SureMap: Simultaneous Mean Estimation for Single-Task and Multi-Task Disaggregated Evaluation","arxiv_id":"2411.09730","date":"2024-11-14","proceeding":null,"authors":["Mikhail Khodak","Lester Mackey","Alexandra Chouldechova","Miroslav Dudík"],"abstract":"Disaggregated evaluation -- estimation of performance of a machine learning model on different subpopulations -- is a core task when assessing performance and group-fairness of AI systems. A key challenge is that evaluation data is scarce, and subpopulations arising from intersections of attributes (e.g., race, sex, age) are often tiny. Today, it is common for multiple clients to procure the same AI model from a model developer, and the task of disaggregated evaluation is faced by each customer individually. This gives rise to what we call the multi-task disaggregated evaluation problem, wherein multiple clients seek to conduct a disaggregated evaluation of a given model in their own data setting (task). In this work we develop a disaggregated evaluation method called SureMap that has high estimation accuracy for both multi-task and single-task disaggregated evaluations of blackbox models. SureMap's efficiency gains come from (1) transforming the problem into structured simultaneous Gaussian mean estimation and (2) incorporating external data, e.g., from the AI system creator or from their other clients. Our method combines maximum a posteriori (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE). We evaluate SureMap on disaggregated evaluation tasks in multiple domains, observing significant accuracy improvements over several strong competitors.","url_abs":"https://arxiv.org/abs/2411.09730v1","url_pdf":"https://arxiv.org/pdf/2411.09730v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"suremap-simultaneous-mean-estimation-for","repo_url":"https://github.com/mkhodak/SureMap","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"fairness","task_name":"Fairness"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.09730","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.09730"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/mkhodak/SureMap","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"unverified":9},"by_repo_kind":{"official":{"samples":10,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"718ea725b847f654","entry":"map_estimator","repo":"mkhodak/suremap","repo_kind":"official","path":"suremap.py","file_url":"https://github.com/mkhodak/suremap/blob/HEAD/suremap.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"718ea725b847f654"}},{"code_sha256_prefix":"6291a9974eb0a669","entry":"auc_metric","repo":"mkhodak/SureMap","repo_kind":"official","path":"data.py","file_url":"https://github.com/mkhodak/SureMap/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6291a9974eb0a669"}},{"code_sha256_prefix":"72d851b4a4bc7f16","entry":"error_metric","repo":"mkhodak/SureMap","repo_kind":"official","path":"data.py","file_url":"https://github.com/mkhodak/SureMap/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72d851b4a4bc7f16"}},{"code_sha256_prefix":"e0498520a9ee2924","entry":"mpi4py_reduce","repo":"mkhodak/SureMap","repo_kind":"official","path":"data.py","file_url":"https://github.com/mkhodak/SureMap/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0498520a9ee2924"}},{"code_sha256_prefix":"98d6a5ada51a16d4","entry":"multitask_global","repo":"mkhodak/SureMap","repo_kind":"official","path":"estimators.py","file_url":"https://github.com/mkhodak/SureMap/blob/HEAD/estimators.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98d6a5ada51a16d4"}},{"code_sha256_prefix":"53d5f80a681f9a59","entry":"naive_estimator","repo":"mkhodak/SureMap","repo_kind":"official","path":"estimators.py","file_url":"https://github.com/mkhodak/SureMap/blob/HEAD/estimators.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53d5f80a681f9a59"}},{"code_sha256_prefix":"3a374c96c2e77cb6","entry":"pooled_estimator","repo":"mkhodak/SureMap","repo_kind":"official","path":"estimators.py","file_url":"https://github.com/mkhodak/SureMap/blob/HEAD/estimators.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a374c96c2e77cb6"}},{"code_sha256_prefix":"8fbbd38eadc537a3","entry":"rnd_round","repo":"mkhodak/SureMap","repo_kind":"official","path":"sr.py","file_url":"https://github.com/mkhodak/SureMap/blob/HEAD/sr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8fbbd38eadc537a3"}},{"code_sha256_prefix":"bed832ca0b2521d1","entry":"split_df","repo":"mkhodak/SureMap","repo_kind":"official","path":"sr.py","file_url":"https://github.com/mkhodak/SureMap/blob/HEAD/sr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bed832ca0b2521d1"}},{"code_sha256_prefix":"9dd7481f5863799b","entry":"stratified_sample","repo":"mkhodak/SureMap","repo_kind":"official","path":"sr.py","file_url":"https://github.com/mkhodak/SureMap/blob/HEAD/sr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9dd7481f5863799b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}