{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2512-15674","title":"Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers","arxiv_id":"2512.15674","date":"2025-12-17","proceeding":"ICML","authors":["Adam Karvonen","James Chua","Clément Dumas","Kit Fraser-Taliente","Subhash Kantamneni","Julian Minder","Euan Ong","Arnab Sen Sharma","Daniel Wen","Owain Evans","Samuel Marks"],"abstract":"Large language model (LLM) activations are notoriously difficult to understand, with most existing techniques using complex, specialized methods for interpreting them. Recent work has proposed a simpler approach known as LatentQA: training LLMs to directly accept LLM activations as inputs and answer arbitrary questions about them in natural language. However, prior work has focused on narrow task settings for both training and evaluation. In this paper, we instead take a generalist perspective. We evaluate LatentQA-trained models, which we call Activation Oracles (AOs), in far out-of-distribution settings and examine how performance scales with training data diversity. We find that AOs can recover information fine-tuned into a model (e.g., biographical knowledge or malign propensities) that does not appear in the input text, despite never being trained with activations from a fine-tuned model. Our main evaluations are four downstream tasks where we can compare to prior white- and black-box techniques. We find that even narrowly-trained LatentQA models can generalize well, and that adding additional training datasets (such as classification tasks and a self-supervised context prediction task) yields consistent further improvements. Our best AOs match or exceed white-box baselines on all four tasks and the best overall baseline on 3 of 4. These results suggest that diversified training to answer natural-language queries imparts a general capability to verbalize information about LLM activations.","url_abs":"https://arxiv.org/abs/2512.15674","url_pdf":"https://arxiv.org/pdf/2512.15674","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2512.15674","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2512.15674"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/cywinski/eliciting-secret-knowledge","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":25},"by_repo_kind":{"found_in_text":{"samples":25,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4d0dda598837be0b","entry":"calculate_prefill_statistics","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"ssc/calculate_auditing_metrics_gpt.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/ssc/calculate_auditing_metrics_gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d0dda598837be0b"}},{"code_sha256_prefix":"96d9d0b4f37999dd","entry":"decode_base64_partially","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"ssc/guess_secret_side_constraint_iterative.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/ssc/guess_secret_side_constraint_iterative.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"96d9d0b4f37999dd"}},{"code_sha256_prefix":"0f440ae1bd737171","entry":"extract_layer_probabilities","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"elicitation_methods/logit_lens.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/elicitation_methods/logit_lens.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f440ae1bd737171"}},{"code_sha256_prefix":"f56b76b763051ff9","entry":"extract_pairs","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"utils/guess_secret_word.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/utils/guess_secret_word.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f56b76b763051ff9"}},{"code_sha256_prefix":"9bc9d942b59f5d51","entry":"extract_residual_similarities","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"elicitation_methods/residual_tokens.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/elicitation_methods/residual_tokens.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9bc9d942b59f5d51"}},{"code_sha256_prefix":"69069a9ed77cfa7d","entry":"extract_ssc_positions_for_iteration","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"ssc/guess_secret_side_constraint_iterative.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/ssc/guess_secret_side_constraint_iterative.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"69069a9ed77cfa7d"}},{"code_sha256_prefix":"11cedf0738cf7a8b","entry":"fetch_sae_feature_top_tokens","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"ssc/guess_secret_side_constraint.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/ssc/guess_secret_side_constraint.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11cedf0738cf7a8b"}},{"code_sha256_prefix":"39dc68013194e9a1","entry":"get_inference_engine","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"sampling/inference_engine.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/sampling/inference_engine.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39dc68013194e9a1"}},{"code_sha256_prefix":"2a1f9a63b1a11968","entry":"get_prompt_preparer","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"sampling/prompt_preparers.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/sampling/prompt_preparers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a1f9a63b1a11968"}},{"code_sha256_prefix":"a8eb3ad40b635602","entry":"get_top_k_similar_tokens","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"elicitation_methods/residual_tokens.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/elicitation_methods/residual_tokens.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8eb3ad40b635602"}},{"code_sha256_prefix":"9fa782ec202e4857","entry":"get_top_k_tokens","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"elicitation_methods/logit_lens.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/elicitation_methods/logit_lens.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9fa782ec202e4857"}},{"code_sha256_prefix":"102d0f3e451d4908","entry":"get_top_k_tokens_ssc","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"elicitation_methods/residual_tokens.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/elicitation_methods/residual_tokens.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"102d0f3e451d4908"}},{"code_sha256_prefix":"087ae9d981b6505c","entry":"group_entries_by_key","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"ssc/calculate_auditing_metrics_gpt.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/ssc/calculate_auditing_metrics_gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"087ae9d981b6505c"}},{"code_sha256_prefix":"1253fb9a528e3524","entry":"group_entries_by_prefill","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"ssc/calculate_auditing_metrics_gpt.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/ssc/calculate_auditing_metrics_gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1253fb9a528e3524"}},{"code_sha256_prefix":"9c716e534c80d0fd","entry":"load_conversation_from_file","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"sampling/batch_inference.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/sampling/batch_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c716e534c80d0fd"}},{"code_sha256_prefix":"03ce8b49985247f6","entry":"load_densities_from_file","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"elicitation_methods/sae.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/elicitation_methods/sae.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03ce8b49985247f6"}},{"code_sha256_prefix":"1bc9d55c3d545619","entry":"load_density_tensor","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"elicitation_methods/sae.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/elicitation_methods/sae.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1bc9d55c3d545619"}},{"code_sha256_prefix":"4e066bd9279122ee","entry":"load_feature_tokens_file","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"ssc/guess_secret_side_constraint.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/ssc/guess_secret_side_constraint.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e066bd9279122ee"}},{"code_sha256_prefix":"e66de9305fea6490","entry":"load_lines_from_file","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"sampling/batch_inference.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/sampling/batch_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e66de9305fea6490"}},{"code_sha256_prefix":"7c099df440884000","entry":"load_model_and_tokenizer","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"sampling/utils.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/sampling/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c099df440884000"}},{"code_sha256_prefix":"7b71485921e1d112","entry":"load_prompt_template","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"utils/guess_secret_word.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/utils/guess_secret_word.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b71485921e1d112"}},{"code_sha256_prefix":"2557ed63984259cf","entry":"load_results_file","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"utils/guess_secret_word.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/utils/guess_secret_word.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2557ed63984259cf"}},{"code_sha256_prefix":"c59f849fa81317ce","entry":"parse_auditor_response","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"ssc/guess_secret_side_constraint_iterative.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/ssc/guess_secret_side_constraint_iterative.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c59f849fa81317ce"}},{"code_sha256_prefix":"43596bf917ab9a02","entry":"tfidf_feature_score","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"elicitation_methods/sae.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/elicitation_methods/sae.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43596bf917ab9a02"}},{"code_sha256_prefix":"3146c38bfcad5408","entry":"validate_and_determine_strategy","repo":"cywinski/eliciting-secret-knowledge","repo_kind":"found_in_text","path":"sampling/batch_inference.py","file_url":"https://github.com/cywinski/eliciting-secret-knowledge/blob/HEAD/sampling/batch_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3146c38bfcad5408"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_api"},"syntology_extracted_results":null}