{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hidden-or-inferred-fair-learning-to-rank-with","title":"Hidden or Inferred: Fair Learning-To-Rank with Unknown Demographics","arxiv_id":"2407.17459","date":"2024-07-24","proceeding":null,"authors":["Oluseun Olulana","Kathleen Cachel","Fabricio Murai","Elke Rundensteiner"],"abstract":"As learning-to-rank models are increasingly deployed for decision-making in areas with profound life implications, the FairML community has been developing fair learning-to-rank (LTR) models. These models rely on the availability of sensitive demographic features such as race or sex. However, in practice, regulatory obstacles and privacy concerns protect this data from collection and use. As a result, practitioners may either need to promote fairness despite the absence of these features or turn to demographic inference tools to attempt to infer them. Given that these tools are fallible, this paper aims to further understand how errors in demographic inference impact the fairness performance of popular fair LTR strategies. In which cases would it be better to keep such demographic attributes hidden from models versus infer them? We examine a spectrum of fair LTR strategies ranging from fair LTR with and without demographic features hidden versus inferred to fairness-unaware LTR followed by fair re-ranking. We conduct a controlled empirical investigation modeling different levels of inference errors by systematically perturbing the inferred sensitive attribute. We also perform three case studies with real-world datasets and popular open-source inference methods. Our findings reveal that as inference noise grows, LTR-based methods that incorporate fairness considerations into the learning process may increase bias. In contrast, fair re-ranking strategies are more robust to inference errors. All source code, data, and experimental artifacts of our experimental study are available here: https://github.com/sewen007/hoiltr.git","url_abs":"https://arxiv.org/abs/2407.17459v1","url_pdf":"https://arxiv.org/pdf/2407.17459v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hidden-or-inferred-fair-learning-to-rank-with","repo_url":"https://github.com/sewen007/hoiltr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"fairness","task_name":"Fairness"},{"task_slug":"learning-to-rank","task_name":"Learning-To-Rank"},{"task_slug":"re-ranking","task_name":"Re-Ranking"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.17459","atlas_url":"https://app.syntology.ai/?focus=2407.17459","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.17459"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sewen007/hoiltr","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":12,"unverified":5},"by_repo_kind":{"official":{"samples":17,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"41a8fca30baa973d","entry":"MARC","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/ARC.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/ARC.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"41a8fca30baa973d"}},{"code_sha256_prefix":"6aa8a03e240c13b8","entry":"Z","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/ValidInputs.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/ValidInputs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6aa8a03e240c13b8"}},{"code_sha256_prefix":"da9eb00bc46a9af2","entry":"Z_Vector","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/ValidInputs.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/ValidInputs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"da9eb00bc46a9af2"}},{"code_sha256_prefix":"79146393cf59a664","entry":"avg_pos_diff_ndcg","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/avg_pos_dif.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/avg_pos_dif.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"79146393cf59a664"}},{"code_sha256_prefix":"ccdafee51fd54c94","entry":"avg_pos_diff_skew","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/avg_pos_dif.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/avg_pos_dif.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ccdafee51fd54c94"}},{"code_sha256_prefix":"7b97ab5c1933e24d","entry":"exposure_ratio","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/avgExp.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/avgExp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b97ab5c1933e24d"}},{"code_sha256_prefix":"bb3886f125bb7628","entry":"extract_string","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/calculate_metrics.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/calculate_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bb3886f125bb7628"}},{"code_sha256_prefix":"434b485a4d4b996f","entry":"get_files","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/combine.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/combine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"434b485a4d4b996f"}},{"code_sha256_prefix":"b4b069e72cf8355d","entry":"kT","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/kendallTau.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/kendallTau.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b4b069e72cf8355d"}},{"code_sha256_prefix":"2402d86e1a293756","entry":"prob","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/ValidInputs.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/ValidInputs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2402d86e1a293756"}},{"code_sha256_prefix":"d35a2badd140bf8e","entry":"substring_match","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/calculate_metrics.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/calculate_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d35a2badd140bf8e"}},{"code_sha256_prefix":"7e62f90b778e354e","entry":"updated_value","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/combine.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/combine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e62f90b778e354e"}},{"code_sha256_prefix":"214998a5c54ed783","entry":"NDCG","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/NDCG.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/NDCG.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"214998a5c54ed783"}},{"code_sha256_prefix":"72afe6bc46003fb9","entry":"NDKL","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/NDKL.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/NDKL.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"72afe6bc46003fb9"}},{"code_sha256_prefix":"3a7ce8e8a2119f94","entry":"avg_exp","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/avgExp.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/avgExp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3a7ce8e8a2119f94"}},{"code_sha256_prefix":"c20e980dc033dd19","entry":"dp_exp","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/avgExp.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/avgExp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c20e980dc033dd19"}},{"code_sha256_prefix":"47d147d4a32387a5","entry":"skew","repo":"sewen007/hoiltr","repo_kind":"official","path":"HOIRank/data_analysis/skew.py","file_url":"https://github.com/sewen007/hoiltr/blob/HEAD/HOIRank/data_analysis/skew.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"47d147d4a32387a5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}