{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-26119","title":"DeflectBench: A Benchmark for Evaluating Rhetorical Fallacy Generation in LLMs","arxiv_id":"2608.26119","date":"2026-06-18","proceeding":null,"authors":["Art Kanke"],"abstract":"Whether large language models can be prompted to generate rhetorical fallacies on demand, and whether current safety post-training constrains this behavior, has received less attention than the related question of detecting fallacies in existing text. We close this gap with DeflectBench, evaluating 23,990 generations from four frontier models across three deflection strategies (whataboutism, ad hominem, red herring), seven prompt framings, and 80 claims spanning four controversy levels. Refusal is governed primarily by request structure rather than claim content. Per claim refusal varies by only 11 percentage points across the 80 claims, while a single prompt frame change can swing within model refusal by nearly 100 percentage points and switching the requested fallacy type can swing it by over 80 percentage points within explicit framings. An educational debate coach prompt framing collapses refusal to near zero across all four model families, but the bypassed behavior is not clean compliance. Models typically produce labeled compliance, naming the requested manipulation in the same response that contains it. The four models distribute differently across refusal, labeled compliance, soft refusal, and clean compliance. The code and dataset are released at https://github.com/ArtKanke/DeflectBench.","url_abs":"https://arxiv.org/abs/2608.26119","url_pdf":"https://arxiv.org/pdf/2608.26119","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.26119"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ArtKanke/DeflectBench","reach":null}],"summary":{"ran":13,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":15,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"daf7568ada96c4ea","entry":"block_bootstrap_ci","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"analysis/stats_utils.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/analysis/stats_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"daf7568ada96c4ea"}},{"code_sha256_prefix":"ab3732812ade633e","entry":"build_judge_prompt","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"evaluation/core.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/evaluation/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab3732812ade633e"}},{"code_sha256_prefix":"8737904f0167a07d","entry":"build_prompt","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"config.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8737904f0167a07d"}},{"code_sha256_prefix":"7ddce77b9e7eaf8d","entry":"call","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"api/deepseek_client.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/api/deepseek_client.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ddce77b9e7eaf8d"}},{"code_sha256_prefix":"ea617968c1d64b1d","entry":"call","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"api/openai_client.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/api/openai_client.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea617968c1d64b1d"}},{"code_sha256_prefix":"01e3cd66934c566e","entry":"call","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"api/xai_client.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/api/xai_client.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01e3cd66934c566e"}},{"code_sha256_prefix":"7983974d78cf4633","entry":"fallacy_match","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"analysis/analysis.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/analysis/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7983974d78cf4633"}},{"code_sha256_prefix":"a9c03e1e3e6f2443","entry":"gwet_ac1_binary","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"analysis/stats_utils.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/analysis/stats_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9c03e1e3e6f2443"}},{"code_sha256_prefix":"a37c144ed754e7b0","entry":"judge_slug","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"evaluation/core.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/evaluation/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a37c144ed754e7b0"}},{"code_sha256_prefix":"e7de84f0219659c9","entry":"load_consensus","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"analysis/stats_utils.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/analysis/stats_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7de84f0219659c9"}},{"code_sha256_prefix":"f08c31f49e36fb1e","entry":"model_slug","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"generation/core.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/generation/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f08c31f49e36fb1e"}},{"code_sha256_prefix":"e5a8f59d47654ab4","entry":"output_path","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"generation/core.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/generation/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5a8f59d47654ab4"}},{"code_sha256_prefix":"c2a77303ec2c2d90","entry":"parse_scores","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"evaluation/core.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/evaluation/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2a77303ec2c2d90"}},{"code_sha256_prefix":"d4bf8ab065ce119f","entry":"call","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"api/anthropic_client.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/api/anthropic_client.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d4bf8ab065ce119f"}},{"code_sha256_prefix":"d097e4bc334086cd","entry":"pct","repo":"ArtKanke/DeflectBench","repo_kind":"found_in_text","path":"analysis/analysis.py","file_url":"https://github.com/ArtKanke/DeflectBench/blob/HEAD/analysis/analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d097e4bc334086cd"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,795 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6795},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}