{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/systematic-evaluation-of-predictive-fairness","title":"Systematic Evaluation of Predictive Fairness","arxiv_id":"2210.08758","date":"2022-10-17","proceeding":null,"authors":["Xudong Han","Aili Shen","Trevor Cohn","Timothy Baldwin","Lea Frermann"],"abstract":"Mitigating bias in training on biased datasets is an important open problem. Several techniques have been proposed, however the typical evaluation regime is very limited, considering very narrow data conditions. For instance, the effect of target class imbalance and stereotyping is under-studied. To address this gap, we examine the performance of various debiasing methods across multiple tasks, spanning binary classification (Twitter sentiment), multi-class classification (profession prediction), and regression (valence prediction). Through extensive experimentation, we find that data conditions have a strong influence on relative model performance, and that general conclusions cannot be drawn about method efficacy when evaluating only on standard datasets, as is current practice in fairness research.","url_abs":"https://arxiv.org/abs/2210.08758v1","url_pdf":"https://arxiv.org/pdf/2210.08758v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"systematic-evaluation-of-predictive-fairness","repo_url":"https://github.com/hanxudong/systematic_evaluation_of_predictive_fairness","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"binary-classification","task_name":"Binary Classification"},{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"fairness","task_name":"Fairness"},{"task_slug":"multi-class-classification","task_name":"Multi-class Classification"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"regression-1","task_name":"regression"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}