{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/optimizing-black-box-metrics-with-iterative","title":"Optimizing Black-box Metrics with Iterative Example Weighting","arxiv_id":"2102.09492","date":"2021-02-18","proceeding":null,"authors":["Gaurush Hiranandani","Jatin Mathur","Harikrishna Narasimhan","Mahdi Milani Fard","Oluwasanmi Koyejo"],"abstract":"We consider learning to optimize a classification metric defined by a black-box function of the confusion matrix. Such black-box learning settings are ubiquitous, for example, when the learner only has query access to the metric of interest, or in noisy-label and domain adaptation applications where the learner must evaluate the metric via performance evaluation using a small validation sample. Our approach is to adaptively learn example weights on the training dataset such that the resulting weighted objective best approximates the metric on the validation sample. We show how to model and estimate the example weights and use them to iteratively post-shift a pre-trained class probability estimator to construct a classifier. We also analyze the resulting procedure's statistical properties. Experiments on various label noise, domain shift, and fair classification setups confirm that our proposal compares favorably to the state-of-the-art baselines for each application.","url_abs":"https://arxiv.org/abs/2102.09492v2","url_pdf":"https://arxiv.org/pdf/2102.09492v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"optimizing-black-box-metrics-with-iterative","repo_url":"https://github.com/koyejolab/fweg","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"classification","task_name":"General Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2102.09492","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2102.09492"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/koyejolab/fweg","reach":null}],"summary":{"ran":2,"ran_fixture":1,"ran_violates":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"2e38763b0c02abaf","entry":"BlackBoxMetric","repo":"koyejolab/fweg","repo_kind":"official","path":"utils/fweg.py","file_url":"https://github.com/koyejolab/fweg/blob/HEAD/utils/fweg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2e38763b0c02abaf"}},{"code_sha256_prefix":"239e6a8fa23121dc","entry":"FWEG","repo":"koyejolab/fweg","repo_kind":"official","path":"utils/fweg.py","file_url":"https://github.com/koyejolab/fweg/blob/HEAD/utils/fweg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"239e6a8fa23121dc"}},{"code_sha256_prefix":"be7432c5b8bf5e94","entry":"eval_metric","repo":"koyejolab/fweg","repo_kind":"official","path":"utils/fweg.py","file_url":"https://github.com/koyejolab/fweg/blob/HEAD/utils/fweg.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"be7432c5b8bf5e94"}},{"code_sha256_prefix":"475463548cc2ec8a","entry":"eval_true_metric","repo":"koyejolab/fweg","repo_kind":"official","path":"utils/fweg.py","file_url":"https://github.com/koyejolab/fweg/blob/HEAD/utils/fweg.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"475463548cc2ec8a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}