{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-29889","title":"Internal Representation, Not Clinical Knowledge: Where Apparent LLM Triage Failures Originate","arxiv_id":"2605.29889","date":"2026-05-28","proceeding":null,"authors":["David Fraile Navarro","Berardino Como","Jialei Sheng","Soundariya Ananthan","Shlomo Berkovsky"],"abstract":"Patient-voiced clinical-triage benchmarks report high under-triage rates for consumer LLMs for constrained multiple-choice output, yet the same cases score differently with free-text. We ask whether output format changes the model's \\emph{clinical representation} or only the mapping from a preserved representation to an answer. Using sparse-autoencoder (SAE) features in Gemma 3 4B/12B IT and Qwen3-8B, we find the same medical features fire on the shared clinical narrative under both formats but go {silent} at the multiple-choice decision token in all the cases at every model. Three independent methods (natural-language autoencoder verbalization, decision-token logit attribution, and top-feature characterization) agree that scaffold and format features, but not medical features, drive the decision logits. Behaviorally, the multiple-choice penalty inverts under both structured and natural-language input, option-order shuffle rules out positional bias, and the gap is dominated by off-by-one decision (the model picks an adjacent acuity letter to the gold answer) rather than knowledge failure. Thus, the failure originates in the output format and not in the clinical representation.","url_abs":"https://arxiv.org/abs/2605.29889","url_pdf":"https://arxiv.org/pdf/2605.29889","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.29889","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.29889"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/dafraile/SAE_mad","reach":null}],"summary":{"ran":10,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":14,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"02a36b163f03cc76","entry":"format_mcq","repo":"dafraile/SAE_mad","repo_kind":"found_in_text","path":"exploratory/v3_layer_sweep.py","file_url":"https://github.com/dafraile/SAE_mad/blob/HEAD/exploratory/v3_layer_sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"02a36b163f03cc76"}},{"code_sha256_prefix":"e5591ca5aec42e12","entry":"format_mcq","repo":"dafraile/SAE_mad","repo_kind":"found_in_text","path":"exploratory/v2_medical_rescue.py","file_url":"https://github.com/dafraile/SAE_mad/blob/HEAD/exploratory/v2_medical_rescue.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e5591ca5aec42e12"}},{"code_sha256_prefix":"fbdd9328e963dfba","entry":"format_mcq_prompt","repo":"dafraile/SAE_mad","repo_kind":"found_in_text","path":"exploratory/v2_medical_pilot.py","file_url":"https://github.com/dafraile/SAE_mad/blob/HEAD/exploratory/v2_medical_pilot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fbdd9328e963dfba"}},{"code_sha256_prefix":"a76090ef34726ae9","entry":"get_layer","repo":"dafraile/SAE_mad","repo_kind":"found_in_text","path":"exploratory/v3_layer_sweep.py","file_url":"https://github.com/dafraile/SAE_mad/blob/HEAD/exploratory/v3_layer_sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a76090ef34726ae9"}},{"code_sha256_prefix":"b986da2a6ae24ffd","entry":"get_layer","repo":"dafraile/SAE_mad","repo_kind":"found_in_text","path":"exploratory/v2_medical_rescue.py","file_url":"https://github.com/dafraile/SAE_mad/blob/HEAD/exploratory/v2_medical_rescue.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b986da2a6ae24ffd"}},{"code_sha256_prefix":"885fee40242784c3","entry":"load_corpus","repo":"dafraile/SAE_mad","repo_kind":"found_in_text","path":"exploratory/v1_exploration.py","file_url":"https://github.com/dafraile/SAE_mad/blob/HEAD/exploratory/v1_exploration.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"885fee40242784c3"}},{"code_sha256_prefix":"d038d08a79dec798","entry":"load_paired","repo":"dafraile/SAE_mad","repo_kind":"found_in_text","path":"exploratory/v3_layer_sweep.py","file_url":"https://github.com/dafraile/SAE_mad/blob/HEAD/exploratory/v3_layer_sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d038d08a79dec798"}},{"code_sha256_prefix":"24c554922f0f09c4","entry":"load_paired","repo":"dafraile/SAE_mad","repo_kind":"found_in_text","path":"exploratory/v2_flip_distant_combined.py","file_url":"https://github.com/dafraile/SAE_mad/blob/HEAD/exploratory/v2_flip_distant_combined.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"24c554922f0f09c4"}},{"code_sha256_prefix":"f1b4dead7be35e73","entry":"load_paired","repo":"dafraile/SAE_mad","repo_kind":"found_in_text","path":"exploratory/v2_generalization.py","file_url":"https://github.com/dafraile/SAE_mad/blob/HEAD/exploratory/v2_generalization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f1b4dead7be35e73"}},{"code_sha256_prefix":"a48795862d9db663","entry":"make_hook","repo":"dafraile/SAE_mad","repo_kind":"found_in_text","path":"exploratory/hw2b_fallback.py","file_url":"https://github.com/dafraile/SAE_mad/blob/HEAD/exploratory/hw2b_fallback.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a48795862d9db663"}},{"code_sha256_prefix":"ab3ac37fe72ccbd7","entry":"collect_all_activations","repo":"dafraile/SAE_mad","repo_kind":"found_in_text","path":"exploratory/v1_exploration.py","file_url":"https://github.com/dafraile/SAE_mad/blob/HEAD/exploratory/v1_exploration.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab3ac37fe72ccbd7"}},{"code_sha256_prefix":"26f96736741840ca","entry":"evaluate_answer_likelihood","repo":"dafraile/SAE_mad","repo_kind":"found_in_text","path":"exploratory/v2_medical_pilot.py","file_url":"https://github.com/dafraile/SAE_mad/blob/HEAD/exploratory/v2_medical_pilot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"26f96736741840ca"}},{"code_sha256_prefix":"13d35c84e54379a6","entry":"extract_activations","repo":"dafraile/SAE_mad","repo_kind":"found_in_text","path":"exploratory/v1_exploration.py","file_url":"https://github.com/dafraile/SAE_mad/blob/HEAD/exploratory/v1_exploration.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"13d35c84e54379a6"}},{"code_sha256_prefix":"131434c85d33e76d","entry":"get_answer_probs","repo":"dafraile/SAE_mad","repo_kind":"found_in_text","path":"exploratory/v2_medical_rescue.py","file_url":"https://github.com/dafraile/SAE_mad/blob/HEAD/exploratory/v2_medical_rescue.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"131434c85d33e76d"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}