{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-comparative-study-of-fairness-enhancing","title":"A comparative study of fairness-enhancing interventions in machine learning","arxiv_id":"1802.04422","date":"2018-02-13","proceeding":null,"authors":["Sorelle A. Friedler","Carlos Scheidegger","Suresh Venkatasubramanian","Sonam Choudhary","Evan P. Hamilton","Derek Roth"],"abstract":"Computers are increasingly used to make decisions that have significant\nimpact in people's lives. Often, these predictions can affect different\npopulation subgroups disproportionately. As a result, the issue of fairness has\nreceived much recent interest, and a number of fairness-enhanced classifiers\nand predictors have appeared in the literature. This paper seeks to study the\nfollowing questions: how do these different techniques fundamentally compare to\none another, and what accounts for the differences? Specifically, we seek to\nbring attention to many under-appreciated aspects of such fairness-enhancing\ninterventions. Concretely, we present the results of an open benchmark we have\ndeveloped that lets us compare a number of different algorithms under a variety\nof fairness measures, and a large number of existing datasets. We find that\nalthough different algorithms tend to prefer specific formulations of fairness\npreservations, many of these measures strongly correlate with one another. In\naddition, we find that fairness-preserving algorithms tend to be sensitive to\nfluctuations in dataset composition (simulated in our benchmark by varying\ntraining-test splits), indicating that fairness interventions might be more\nbrittle than previously thought.","url_abs":"http://arxiv.org/abs/1802.04422v1","url_pdf":"http://arxiv.org/pdf/1802.04422v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-comparative-study-of-fairness-enhancing","repo_url":"https://github.com/algofairness/fairness-comparison","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"a-comparative-study-of-fairness-enhancing","repo_url":"https://github.com/JSGoedhart/fairness-comparison","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"a-comparative-study-of-fairness-enhancing","repo_url":"https://github.com/charan223/FairDeepLearning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"a-comparative-study-of-fairness-enhancing","repo_url":"https://github.com/huanglx12/Stable-Fair-Classification","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"fairness","task_name":"Fairness"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1802.04422","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1802.04422"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/charan223/FairDeepLearning","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/algofairness/fairness-comparison","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JSGoedhart/fairness-comparison","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huanglx12/Stable-Fair-Classification","reach":null}],"summary":{"ran_draft_wrong":9,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1},"listed":{"samples":7,"ran":6,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"a8410405c788323a","entry":"all_possible_graphs","repo":"JSGoedhart/fairness-comparison","repo_kind":"listed","path":"fairness/analysis.py","file_url":"https://github.com/JSGoedhart/fairness-comparison/blob/HEAD/fairness/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a8410405c788323a"}},{"code_sha256_prefix":"47a0278c9fef3265","entry":"compute_diff_nums","repo":"huanglx12/Stable-Fair-Classification","repo_kind":"listed","path":"fairness/benchmark.py","file_url":"https://github.com/huanglx12/Stable-Fair-Classification/blob/HEAD/fairness/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"47a0278c9fef3265"}},{"code_sha256_prefix":"7da718e0807f10c7","entry":"filter_table","repo":"charan223/FairDeepLearning","repo_kind":"listed","path":"scripts/plots_tables/csvs/parse_csv.py","file_url":"https://github.com/charan223/FairDeepLearning/blob/HEAD/scripts/plots_tables/csvs/parse_csv.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7da718e0807f10c7"}},{"code_sha256_prefix":"6642c1dab7b8e92b","entry":"get_dict_sensitive_vals","repo":"algofairness/fairness-comparison","repo_kind":"official","path":"fairness/benchmark.py","file_url":"https://github.com/algofairness/fairness-comparison/blob/HEAD/fairness/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6642c1dab7b8e92b"}},{"code_sha256_prefix":"c861a2f90457c10b","entry":"get_iters","repo":"charan223/FairDeepLearning","repo_kind":"listed","path":"train/execute.py","file_url":"https://github.com/charan223/FairDeepLearning/blob/HEAD/train/execute.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c861a2f90457c10b"}},{"code_sha256_prefix":"0f5b147fa50f6183","entry":"get_sensitive_from_filename","repo":"algofairness/fairness-comparison","repo_kind":"official","path":"analysis/correlation-vis/combine_results_files.py","file_url":"https://github.com/algofairness/fairness-comparison/blob/HEAD/analysis/correlation-vis/combine_results_files.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0f5b147fa50f6183"}},{"code_sha256_prefix":"a6f24b46ffb7aa43","entry":"group_and_agg","repo":"charan223/FairDeepLearning","repo_kind":"listed","path":"scripts/plots_tables/csvs/parse_csv.py","file_url":"https://github.com/charan223/FairDeepLearning/blob/HEAD/scripts/plots_tables/csvs/parse_csv.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6f24b46ffb7aa43"}},{"code_sha256_prefix":"91ce47d82c73659b","entry":"make_metrics_list","repo":"algofairness/fairness-comparison","repo_kind":"official","path":"analysis/correlation-vis/combine_results_files.py","file_url":"https://github.com/algofairness/fairness-comparison/blob/HEAD/analysis/correlation-vis/combine_results_files.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"91ce47d82c73659b"}},{"code_sha256_prefix":"5bd1ce1fcaa81378","entry":"run_alg","repo":"algofairness/fairness-comparison","repo_kind":"official","path":"fairness/benchmark.py","file_url":"https://github.com/algofairness/fairness-comparison/blob/HEAD/fairness/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5bd1ce1fcaa81378"}},{"code_sha256_prefix":"13d270541da56dd7","entry":"run_alg","repo":"JSGoedhart/fairness-comparison","repo_kind":"listed","path":"fairness/benchmark.py","file_url":"https://github.com/JSGoedhart/fairness-comparison/blob/HEAD/fairness/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"13d270541da56dd7"}},{"code_sha256_prefix":"37de72ff09213994","entry":"group_and_max","repo":"charan223/FairDeepLearning","repo_kind":"listed","path":"scripts/plots_tables/csvs/parse_csv.py","file_url":"https://github.com/charan223/FairDeepLearning/blob/HEAD/scripts/plots_tables/csvs/parse_csv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"37de72ff09213994"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}