{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fairness-in-ranking-under-uncertainty","title":"Fairness in Ranking under Uncertainty","arxiv_id":"2107.06720","date":"2021-07-14","proceeding":"NeurIPS 2021 12","authors":["Ashudeep Singh","David Kempe","Thorsten Joachims"],"abstract":"Fairness has emerged as an important consideration in algorithmic decision-making. Unfairness occurs when an agent with higher merit obtains a worse outcome than an agent with lower merit. Our central point is that a primary cause of unfairness is uncertainty. A principal or algorithm making decisions never has access to the agents' true merit, and instead uses proxy features that only imperfectly predict merit (e.g., GPA, star ratings, recommendation letters). None of these ever fully capture an agent's merit; yet existing approaches have mostly been defining fairness notions directly based on observed features and outcomes. Our primary point is that it is more principled to acknowledge and model the uncertainty explicitly. The role of observed features is to give rise to a posterior distribution of the agents' merits. We use this viewpoint to define a notion of approximate fairness in ranking. We call an algorithm $\\phi$-fair (for $\\phi \\in [0,1]$) if it has the following property for all agents $x$ and all $k$: if agent $x$ is among the top $k$ agents with respect to merit with probability at least $\\rho$ (according to the posterior merit distribution), then the algorithm places the agent among the top $k$ agents in its ranking with probability at least $\\phi \\rho$. We show how to compute rankings that optimally trade off approximate fairness against utility to the principal. In addition to the theoretical characterization, we present an empirical analysis of the potential impact of the approach in simulation studies. For real-world validation, we applied the approach in the context of a paper recommendation system that we built and fielded at the KDD 2020 conference.","url_abs":"https://arxiv.org/abs/2107.06720v2","url_pdf":"https://arxiv.org/pdf/2107.06720v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fairness-in-ranking-under-uncertainty","repo_url":"https://github.com/ashudeep/ranking-fairness-uncertainty","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"fairness","task_name":"Fairness"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2107.06720","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2107.06720"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ashudeep/ranking-fairness-uncertainty","reach":null}],"summary":{"ran_fixture":6,"ran_draft_wrong":2,"ran_honours":1,"ran_violates":1,"unverified":4},"by_repo_kind":{"official":{"samples":14,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ccf9a1ee09d47c35","entry":"_sample_ranking_bernoulli","repo":"ashudeep/ranking-fairness-uncertainty","repo_kind":"official","path":"fairnessday_vs_linprog.py","file_url":"https://github.com/ashudeep/ranking-fairness-uncertainty/blob/HEAD/fairnessday_vs_linprog.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ccf9a1ee09d47c35"}},{"code_sha256_prefix":"cc181d67ff91f42b","entry":"compute_constraint_probability","repo":"ashudeep/ranking-fairness-uncertainty","repo_kind":"official","path":"fairnessday_vs_linprog.py","file_url":"https://github.com/ashudeep/ranking-fairness-uncertainty/blob/HEAD/fairnessday_vs_linprog.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc181d67ff91f42b"}},{"code_sha256_prefix":"277eaf193b100851","entry":"compute_marginal_rank_probabilities","repo":"ashudeep/ranking-fairness-uncertainty","repo_kind":"official","path":"fairnessday_vs_linprog.py","file_url":"https://github.com/ashudeep/ranking-fairness-uncertainty/blob/HEAD/fairnessday_vs_linprog.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"277eaf193b100851"}},{"code_sha256_prefix":"0503ae5484ea6888","entry":"get_dcg","repo":"ashudeep/ranking-fairness-uncertainty","repo_kind":"official","path":"fairnessday_vs_linprog.py","file_url":"https://github.com/ashudeep/ranking-fairness-uncertainty/blob/HEAD/fairnessday_vs_linprog.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0503ae5484ea6888"}},{"code_sha256_prefix":"fd39aedfc2029e77","entry":"get_fairness_day_dcg","repo":"ashudeep/ranking-fairness-uncertainty","repo_kind":"official","path":"fairnessday_vs_linprog.py","file_url":"https://github.com/ashudeep/ranking-fairness-uncertainty/blob/HEAD/fairnessday_vs_linprog.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd39aedfc2029e77"}},{"code_sha256_prefix":"dd1133e2363da4d0","entry":"get_mean_dcg","repo":"ashudeep/ranking-fairness-uncertainty","repo_kind":"official","path":"fairnessday_vs_linprog.py","file_url":"https://github.com/ashudeep/ranking-fairness-uncertainty/blob/HEAD/fairnessday_vs_linprog.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd1133e2363da4d0"}},{"code_sha256_prefix":"b4908cbf65a88643","entry":"get_v","repo":"ashudeep/ranking-fairness-uncertainty","repo_kind":"official","path":"fairnessday_vs_linprog.py","file_url":"https://github.com/ashudeep/ranking-fairness-uncertainty/blob/HEAD/fairnessday_vs_linprog.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b4908cbf65a88643"}},{"code_sha256_prefix":"39d54a02d52fdcf6","entry":"optimal_ranking","repo":"ashudeep/ranking-fairness-uncertainty","repo_kind":"official","path":"fairnessday_vs_linprog.py","file_url":"https://github.com/ashudeep/ranking-fairness-uncertainty/blob/HEAD/fairnessday_vs_linprog.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39d54a02d52fdcf6"}},{"code_sha256_prefix":"fdfc93d23e7f4bbc","entry":"sample_ranking_bernoulli","repo":"ashudeep/ranking-fairness-uncertainty","repo_kind":"official","path":"fairnessday_vs_linprog.py","file_url":"https://github.com/ashudeep/ranking-fairness-uncertainty/blob/HEAD/fairnessday_vs_linprog.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdfc93d23e7f4bbc"}},{"code_sha256_prefix":"15f4255768a4cfa8","entry":"sample_rankings","repo":"ashudeep/ranking-fairness-uncertainty","repo_kind":"official","path":"fairnessday_vs_linprog.py","file_url":"https://github.com/ashudeep/ranking-fairness-uncertainty/blob/HEAD/fairnessday_vs_linprog.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15f4255768a4cfa8"}},{"code_sha256_prefix":"f5cc0d9de7be2ff0","entry":"compare_fairness_day_with_linprog","repo":"ashudeep/ranking-fairness-uncertainty","repo_kind":"official","path":"fairnessday_vs_linprog.py","file_url":"https://github.com/ashudeep/ranking-fairness-uncertainty/blob/HEAD/fairnessday_vs_linprog.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5cc0d9de7be2ff0"}},{"code_sha256_prefix":"d51b5a28d5e2f300","entry":"get_linprog_coeffs_all_permutations","repo":"ashudeep/ranking-fairness-uncertainty","repo_kind":"official","path":"fairnessday_vs_linprog.py","file_url":"https://github.com/ashudeep/ranking-fairness-uncertainty/blob/HEAD/fairnessday_vs_linprog.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d51b5a28d5e2f300"}},{"code_sha256_prefix":"cd3c58440660f744","entry":"plot_marginal_rank_distribution","repo":"ashudeep/ranking-fairness-uncertainty","repo_kind":"official","path":"fairnessday_vs_linprog.py","file_url":"https://github.com/ashudeep/ranking-fairness-uncertainty/blob/HEAD/fairnessday_vs_linprog.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd3c58440660f744"}},{"code_sha256_prefix":"b2c79a45a970607c","entry":"plot_matrix","repo":"ashudeep/ranking-fairness-uncertainty","repo_kind":"official","path":"fairnessday_vs_linprog.py","file_url":"https://github.com/ashudeep/ranking-fairness-uncertainty/blob/HEAD/fairnessday_vs_linprog.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2c79a45a970607c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}