{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ceb-compositional-evaluation-benchmark-for","title":"CEB: Compositional Evaluation Benchmark for Fairness in Large Language Models","arxiv_id":"2407.02408","date":"2024-07-02","proceeding":null,"authors":["Song Wang","Peng Wang","Tong Zhou","Yushun Dong","Zhen Tan","Jundong Li"],"abstract":"As Large Language Models (LLMs) are increasingly deployed to handle various natural language processing (NLP) tasks, concerns regarding the potential negative societal impacts of LLM-generated content have also arisen. To evaluate the biases exhibited by LLMs, researchers have recently proposed a variety of datasets. However, existing bias evaluation efforts often focus on only a particular type of bias and employ inconsistent evaluation metrics, leading to difficulties in comparison across different datasets and LLMs. To address these limitations, we collect a variety of datasets designed for the bias evaluation of LLMs, and further propose CEB, a Compositional Evaluation Benchmark that covers different types of bias across different social groups and tasks. The curation of CEB is based on our newly proposed compositional taxonomy, which characterizes each dataset from three dimensions: bias types, social groups, and tasks. By combining the three dimensions, we develop a comprehensive evaluation strategy for the bias in LLMs. Our experiments demonstrate that the levels of bias vary across these dimensions, thereby providing guidance for the development of specific bias mitigation methods.","url_abs":"https://arxiv.org/abs/2407.02408v1","url_pdf":"https://arxiv.org/pdf/2407.02408v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"fairness","task_name":"Fairness"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.02408","atlas_url":"https://app.syntology.ai/?focus=2407.02408","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.02408"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/SongW-SW/CEB","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"21af368cd41ae8b1","entry":"RtA","repo":"SongW-SW/CEB","repo_kind":"found_in_text","path":"src/utils/metrics.py","file_url":"https://github.com/SongW-SW/CEB/blob/HEAD/src/utils/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"21af368cd41ae8b1"}},{"code_sha256_prefix":"7b4d46fb37dd8e23","entry":"auto_assign_paths","repo":"SongW-SW/CEB","repo_kind":"found_in_text","path":"src/task/pipeline.py","file_url":"https://github.com/SongW-SW/CEB/blob/HEAD/src/task/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b4d46fb37dd8e23"}},{"code_sha256_prefix":"1047e1dc044991ea","entry":"calculate_toxicity","repo":"SongW-SW/CEB","repo_kind":"found_in_text","path":"src/utils/metrics.py","file_url":"https://github.com/SongW-SW/CEB/blob/HEAD/src/utils/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1047e1dc044991ea"}},{"code_sha256_prefix":"f296603a45a5725f","entry":"find_best_match","repo":"SongW-SW/CEB","repo_kind":"found_in_text","path":"src/task/pipeline.py","file_url":"https://github.com/SongW-SW/CEB/blob/HEAD/src/task/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f296603a45a5725f"}},{"code_sha256_prefix":"0a0aee58770717a3","entry":"is_chinese_ratio","repo":"SongW-SW/CEB","repo_kind":"found_in_text","path":"src/utils/metrics.py","file_url":"https://github.com/SongW-SW/CEB/blob/HEAD/src/utils/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0a0aee58770717a3"}},{"code_sha256_prefix":"51658330a10c0e19","entry":"load_json","repo":"SongW-SW/CEB","repo_kind":"found_in_text","path":"src/utils/file_process.py","file_url":"https://github.com/SongW-SW/CEB/blob/HEAD/src/utils/file_process.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"51658330a10c0e19"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}