{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/kelly-is-a-warm-person-joseph-is-a-role-model","title":"\"Kelly is a Warm Person, Joseph is a Role Model\": Gender Biases in LLM-Generated Reference Letters","arxiv_id":"2310.09219","date":"2023-10-13","proceeding":null,"authors":["Yixin Wan","George Pu","Jiao Sun","Aparna Garimella","Kai-Wei Chang","Nanyun Peng"],"abstract":"Large Language Models (LLMs) have recently emerged as an effective tool to assist individuals in writing various types of content, including professional documents such as recommendation letters. Though bringing convenience, this application also introduces unprecedented fairness concerns. Model-generated reference letters might be directly used by users in professional scenarios. If underlying biases exist in these model-constructed letters, using them without scrutinization could lead to direct societal harms, such as sabotaging application success rates for female applicants. In light of this pressing issue, it is imminent and necessary to comprehensively study fairness issues and associated harms in this real-world use case. In this paper, we critically examine gender biases in LLM-generated reference letters. Drawing inspiration from social science findings, we design evaluation methods to manifest biases through 2 dimensions: (1) biases in language style and (2) biases in lexical content. We further investigate the extent of bias propagation by analyzing the hallucination bias of models, a term that we define to be bias exacerbation in model-hallucinated contents. Through benchmarking evaluation on 2 popular LLMs- ChatGPT and Alpaca, we reveal significant gender biases in LLM-generated recommendation letters. Our findings not only warn against using LLMs for this application without scrutinization, but also illuminate the importance of thoroughly studying hidden biases and harms in LLM-generated professional documents.","url_abs":"https://arxiv.org/abs/2310.09219v5","url_pdf":"https://arxiv.org/pdf/2310.09219v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"kelly-is-a-warm-person-joseph-is-a-role-model","repo_url":"https://github.com/uclanlp/biases-llm-reference-letters","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"fairness","task_name":"Fairness"},{"task_slug":"hallucination","task_name":"Hallucination"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.09219","atlas_url":"https://app.syntology.ai/?focus=2310.09219","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.09219"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uclanlp/biases-llm-reference-letters","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6347eb74fe6ccde8","entry":"calculate_dict","repo":"uclanlp/biases-llm-reference-letters","repo_kind":"official","path":"biases_lexical_content.py","file_url":"https://github.com/uclanlp/biases-llm-reference-letters/blob/HEAD/biases_lexical_content.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6347eb74fe6ccde8"}},{"code_sha256_prefix":"3aca3023253a4a1d","entry":"data_collator","repo":"uclanlp/biases-llm-reference-letters","repo_kind":"official","path":"agency_classifier/finetune_bert_bias.py","file_url":"https://github.com/uclanlp/biases-llm-reference-letters/blob/HEAD/agency_classifier/finetune_bert_bias.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3aca3023253a4a1d"}},{"code_sha256_prefix":"91fb03f05a38e2cb","entry":"generate_response_rec_alpaca","repo":"uclanlp/biases-llm-reference-letters","repo_kind":"official","path":"generation_util.py","file_url":"https://github.com/uclanlp/biases-llm-reference-letters/blob/HEAD/generation_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91fb03f05a38e2cb"}},{"code_sha256_prefix":"66fc6b55d97f7cb9","entry":"odds_ratio","repo":"uclanlp/biases-llm-reference-letters","repo_kind":"official","path":"biases_lexical_content.py","file_url":"https://github.com/uclanlp/biases-llm-reference-letters/blob/HEAD/biases_lexical_content.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66fc6b55d97f7cb9"}},{"code_sha256_prefix":"abf1dc7ccaae0104","entry":"predict","repo":"uclanlp/biases-llm-reference-letters","repo_kind":"official","path":"classifier.py","file_url":"https://github.com/uclanlp/biases-llm-reference-letters/blob/HEAD/classifier.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"abf1dc7ccaae0104"}},{"code_sha256_prefix":"1cd818c7ce732b1c","entry":"compute_metrics","repo":"uclanlp/biases-llm-reference-letters","repo_kind":"official","path":"agency_classifier/finetune_bert_bias.py","file_url":"https://github.com/uclanlp/biases-llm-reference-letters/blob/HEAD/agency_classifier/finetune_bert_bias.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1cd818c7ce732b1c"}},{"code_sha256_prefix":"530258d882726289","entry":"run_inference","repo":"uclanlp/biases-llm-reference-letters","repo_kind":"official","path":"bert_inference.py","file_url":"https://github.com/uclanlp/biases-llm-reference-letters/blob/HEAD/bert_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"530258d882726289"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}