{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-17506","title":"Evaluating Second-Order Bias of LLMs Through Epistemic Entitlement","arxiv_id":"2606.17506","date":"2026-06-16","proceeding":null,"authors":["Ramaravind Kommiya Mothilal","Terry Jingchen Zhang","Raiyan Ahmed","Zhijing Jin","Shion Guha","Syed Ishtiaque Ahmed"],"abstract":"Evaluations of social bias in LLMs largely focus on whether models generate or imply biased content. However, as LLMs are increasingly used as judges of bias, they may exhibit social biases in subtler ways in how they evaluate biased content, which current methods do not systematically capture. We call this second-order bias: social bias in an LLM's judgment about social bias, which we evaluate through a novel, philosophically grounded reasoning task. Drawing on entitlement epistemology, we conceptualize bias as misplaced foundational knowledge that shapes an agent's rational inquiry, and derive a logical reasoning task for LLMs to judge to whom a biased text is acceptable or non-acceptable. We develop two simple metrics to measure how biased LLM judges are in inferring demographics for acceptability without sufficient support, and how these inferences vary across groups targeted by biased texts. Evaluating open and closed models, we find that our task evades safety guardrails by surfacing bias in model judgment. It varies systematically across target groups, reflects implicit social maps, and shows how models are still triggered by demographic labels. Our work points to the need for LLM bias evaluation in judgment tasks and broadly, for more theoretically grounded approaches to bias evaluation in NLP. We release our code and model responses at https://github.com/uofthcdslab/second-order-bias.","url_abs":"https://arxiv.org/abs/2606.17506","url_pdf":"https://arxiv.org/pdf/2606.17506","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.17506"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/uofthcdslab/second-order-bias","reach":null}],"summary":{"ran_honours":4,"ran_violates":1},"by_repo_kind":{"found_in_text":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7dfc7046e2a536c9","entry":"cleaned_dict_len","repo":"uofthcdslab/second-order-bias","repo_kind":"found_in_text","path":"utils/scoring.py","file_url":"https://github.com/uofthcdslab/second-order-bias/blob/HEAD/utils/scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7dfc7046e2a536c9"}},{"code_sha256_prefix":"9ca793d1e5fc57d0","entry":"compute_scaled_score","repo":"uofthcdslab/second-order-bias","repo_kind":"found_in_text","path":"utils/scoring.py","file_url":"https://github.com/uofthcdslab/second-order-bias/blob/HEAD/utils/scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9ca793d1e5fc57d0"}},{"code_sha256_prefix":"242f04443bfd4bba","entry":"is_unknown_value","repo":"uofthcdslab/second-order-bias","repo_kind":"found_in_text","path":"utils/scoring.py","file_url":"https://github.com/uofthcdslab/second-order-bias/blob/HEAD/utils/scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"242f04443bfd4bba"}},{"code_sha256_prefix":"a58ed85b3033a2c9","entry":"mean_dict_len","repo":"uofthcdslab/second-order-bias","repo_kind":"found_in_text","path":"utils/scoring.py","file_url":"https://github.com/uofthcdslab/second-order-bias/blob/HEAD/utils/scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a58ed85b3033a2c9"}},{"code_sha256_prefix":"09dd661bb8e1619e","entry":"safe_div","repo":"uofthcdslab/second-order-bias","repo_kind":"found_in_text","path":"utils/scoring.py","file_url":"https://github.com/uofthcdslab/second-order-bias/blob/HEAD/utils/scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"09dd661bb8e1619e"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}