{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-00307","title":"BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Models","arxiv_id":"2510.00307","date":"2025-09-30","proceeding":"ICLR","authors":["Thierry Blankenstein","Jialin Yu","Zixuan Li","Vassilis Plachouras","Sunando Sengupta","Philip Torr","Yarin Gal","Alasdair Paren","Adel Bibi"],"abstract":"Agents backed by large language models (LLMs) increasingly rely on external tools drawn from marketplaces where multiple providers offer functionally equivalent options. This raises a critical fairness concern: systematic bias in tool selection can degrade user experience and distort competition by privileging certain providers over others. We introduce a benchmark of diverse tool categories, each containing multiple functionally equivalent tools, to systematically evaluate tool-selection bias. Using this benchmark, we evaluate seven LLMs and show that substantial bias persists, with models either fixating on a single provider or disproportionately favoring tools that appear earlier in the context. To uncover the sources of this behavior, we conduct controlled experiments that isolate the effects of tool features, exposed metadata (name, description, and parameters), and pre-training exposure. We find that (1) semantic alignment between user queries and tool metadata is the strongest driver of selection; (2) small perturbations to tool descriptions can significantly shift choices; and (3) repeated pre-training exposure to a single endpoint amplifies provider-level bias. Finally, we propose a lightweight mitigation strategy that first filters tools to a relevant subset and then samples uniformly, substantially reducing selection bias while maintaining strong task coverage. Our results highlight tool-selection bias as a key obstacle to the fair deployment of tool-augmented LLM agents. Our code and benchmark are publicly available at https://github.com/thierry123454/tool-selection-bias.","url_abs":"https://arxiv.org/abs/2510.00307","url_pdf":"https://arxiv.org/pdf/2510.00307","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.00307","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.00307"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/thierry123454/tool-selection-bias","reach":{"status":"ok"}}],"summary":{"unverified":12},"by_repo_kind":{"found_in_text":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"9100f6fca4585d96","entry":"approx_tokens","repo":"thierry123454/tool-selection-bias","repo_kind":"found_in_text","path":"5_bias_investigation/continued_pretraining/generate_data_gemini.py","file_url":"https://github.com/thierry123454/tool-selection-bias/blob/HEAD/5_bias_investigation/continued_pretraining/generate_data_gemini.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9100f6fca4585d96"}},{"code_sha256_prefix":"93a14e5a9ccc1a74","entry":"compute_stats","repo":"thierry123454/tool-selection-bias","repo_kind":"found_in_text","path":"5_bias_investigation/analyze_features/calc_stats_feats.py","file_url":"https://github.com/thierry123454/tool-selection-bias/blob/HEAD/5_bias_investigation/analyze_features/calc_stats_feats.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"93a14e5a9ccc1a74"}},{"code_sha256_prefix":"2bad52a890367fbe","entry":"cosine_similarity","repo":"thierry123454/tool-selection-bias","repo_kind":"found_in_text","path":"5_bias_investigation/analyze_features/sense_of_scale.py","file_url":"https://github.com/thierry123454/tool-selection-bias/blob/HEAD/5_bias_investigation/analyze_features/sense_of_scale.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2bad52a890367fbe"}},{"code_sha256_prefix":"ff4ecc06bb7d6319","entry":"count_existing_lines","repo":"thierry123454/tool-selection-bias","repo_kind":"found_in_text","path":"5_bias_investigation/continued_pretraining/generate_data_gemini.py","file_url":"https://github.com/thierry123454/tool-selection-bias/blob/HEAD/5_bias_investigation/continued_pretraining/generate_data_gemini.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff4ecc06bb7d6319"}},{"code_sha256_prefix":"2498e76b5f8a7bd9","entry":"count_words","repo":"thierry123454/tool-selection-bias","repo_kind":"found_in_text","path":"5_bias_investigation/continued_pretraining/generate_data_gemini.py","file_url":"https://github.com/thierry123454/tool-selection-bias/blob/HEAD/5_bias_investigation/continued_pretraining/generate_data_gemini.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2498e76b5f8a7bd9"}},{"code_sha256_prefix":"7d76e26c69b1b999","entry":"escape_tex","repo":"thierry123454/tool-selection-bias","repo_kind":"found_in_text","path":"5_bias_investigation/analyze_features/lr.py","file_url":"https://github.com/thierry123454/tool-selection-bias/blob/HEAD/5_bias_investigation/analyze_features/lr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d76e26c69b1b999"}},{"code_sha256_prefix":"58cafcae23a256be","entry":"example_call","repo":"thierry123454/tool-selection-bias","repo_kind":"found_in_text","path":"5_bias_investigation/continued_pretraining/generate_data.py","file_url":"https://github.com/thierry123454/tool-selection-bias/blob/HEAD/5_bias_investigation/continued_pretraining/generate_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"58cafcae23a256be"}},{"code_sha256_prefix":"eab40e1435b85709","entry":"get_conv_template","repo":"thierry123454/tool-selection-bias","repo_kind":"found_in_text","path":"toolbench/tool_conversation.py","file_url":"https://github.com/thierry123454/tool-selection-bias/blob/HEAD/toolbench/tool_conversation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eab40e1435b85709"}},{"code_sha256_prefix":"a0c7393409fd0be9","entry":"load_json","repo":"thierry123454/tool-selection-bias","repo_kind":"found_in_text","path":"5_bias_investigation/extract_features/add_selection_rate.py","file_url":"https://github.com/thierry123454/tool-selection-bias/blob/HEAD/5_bias_investigation/extract_features/add_selection_rate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a0c7393409fd0be9"}},{"code_sha256_prefix":"98bbe08b08d7a02b","entry":"parse_action","repo":"thierry123454/tool-selection-bias","repo_kind":"found_in_text","path":"5_bias_investigation/continued_pretraining/plot_compare_base_vs_adapted.py","file_url":"https://github.com/thierry123454/tool-selection-bias/blob/HEAD/5_bias_investigation/continued_pretraining/plot_compare_base_vs_adapted.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98bbe08b08d7a02b"}},{"code_sha256_prefix":"64903b61e4b03492","entry":"render_param","repo":"thierry123454/tool-selection-bias","repo_kind":"found_in_text","path":"5_bias_investigation/continued_pretraining/generate_data.py","file_url":"https://github.com/thierry123454/tool-selection-bias/blob/HEAD/5_bias_investigation/continued_pretraining/generate_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"64903b61e4b03492"}},{"code_sha256_prefix":"853c161b9ef4dc1e","entry":"standardize","repo":"thierry123454/tool-selection-bias","repo_kind":"found_in_text","path":"5_bias_investigation/continued_pretraining/plot_compare_base_vs_adapted.py","file_url":"https://github.com/thierry123454/tool-selection-bias/blob/HEAD/5_bias_investigation/continued_pretraining/plot_compare_base_vs_adapted.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"853c161b9ef4dc1e"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.AI","source":"arxiv_api"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}