{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/debiasing-evaluations-that-are-biased-by","title":"Debiasing Evaluations That are Biased by Evaluations","arxiv_id":"2012.00714","date":"2020-12-01","proceeding":null,"authors":["Jingyan Wang","Ivan Stelmakh","Yuting Wei","Nihar B. Shah"],"abstract":"It is common to evaluate a set of items by soliciting people to rate them. For example, universities ask students to rate the teaching quality of their instructors, and conference organizers ask authors of submissions to evaluate the quality of the reviews. However, in these applications, students often give a higher rating to a course if they receive higher grades in a course, and authors often give a higher rating to the reviews if their papers are accepted to the conference. In this work, we call these external factors the \"outcome\" experienced by people, and consider the problem of mitigating these outcome-induced biases in the given ratings when some information about the outcome is available. We formulate the information about the outcome as a known partial ordering on the bias. We propose a debiasing method by solving a regularized optimization problem under this ordering constraint, and also provide a carefully designed cross-validation method that adaptively chooses the appropriate amount of regularization. We provide theoretical guarantees on the performance of our algorithm, as well as experimental evaluations.","url_abs":"https://arxiv.org/abs/2012.00714v1","url_pdf":"https://arxiv.org/pdf/2012.00714v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"debiasing-evaluations-that-are-biased-by","repo_url":"https://github.com/jingyanw/outcome-induced-debiasing","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2012.00714","atlas_url":"https://app.syntology.ai/?focus=2012.00714","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2012.00714"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jingyanw/outcome-induced-debiasing","reach":null}],"summary":{"ran_fixture":3,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"961817fa2aa9a120","entry":"interpolate_values","repo":"jingyanw/outcome-induced-debiasing","repo_kind":"official","path":"estimator.py","file_url":"https://github.com/jingyanw/outcome-induced-debiasing/blob/HEAD/estimator.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"961817fa2aa9a120"}},{"code_sha256_prefix":"6ba36435d57119b6","entry":"l2","repo":"jingyanw/outcome-induced-debiasing","repo_kind":"official","path":"simulation.py","file_url":"https://github.com/jingyanw/outcome-induced-debiasing/blob/HEAD/simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ba36435d57119b6"}},{"code_sha256_prefix":"e70c01a0263974e7","entry":"map_mode_bias_noise","repo":"jingyanw/outcome-induced-debiasing","repo_kind":"official","path":"simulation.py","file_url":"https://github.com/jingyanw/outcome-induced-debiasing/blob/HEAD/simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e70c01a0263974e7"}},{"code_sha256_prefix":"c31cd713ff47d2b8","entry":"split_trainval_random_pair","repo":"jingyanw/outcome-induced-debiasing","repo_kind":"official","path":"estimator.py","file_url":"https://github.com/jingyanw/outcome-induced-debiasing/blob/HEAD/estimator.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c31cd713ff47d2b8"}},{"code_sha256_prefix":"f5e266b1e30106bb","entry":"generate_bias_marginal_gaussian","repo":"jingyanw/outcome-induced-debiasing","repo_kind":"official","path":"simulation.py","file_url":"https://github.com/jingyanw/outcome-induced-debiasing/blob/HEAD/simulation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5e266b1e30106bb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}