{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-13337","title":"Where You Measure Decides What You Measure: Position Selection in Ablation-Based SAE Evaluation","arxiv_id":"2608.13337","date":"2026-08-13","proceeding":null,"authors":["Valentin Noël"],"abstract":"Sparse autoencoders are meant to name the things a language model computes, and the usual way to check that a latent matters is to switch it off and see what changes. But a latent fires at many tokens, and the effect has to be measured at one of them. The convention is to measure where the latent fires hardest. That choice is almost never reported, and it is not made by the experimenter: it is made by the dictionary under evaluation. Change the dictionary and the measurement moves to a different token. We show this is not a detail. Take two sparse autoencoders released by Google for the same model and match their latents by decoder similarity: even among the pairs the two dictionaries encode almost identically, they pick different tokens for a large share of them. Two dictionaries compared under the usual protocol are therefore very often compared at different places. To separate the convention from the dictionaries we train six autoencoders from one initialisation, differing only in fitting choices, so that a latent means the same thing in each. Most of the variance such a comparison reads as \"these dictionaries disagree about this latent\" turns out to be the position instead: it falls from 7.6% and 11.9% of variance to near zero once every dictionary is measured at the same token. More evaluation data does not rescue it. Across a sixteenfold range of corpus sizes the dictionaries agree less about where to measure, not more, so the problem grows with scale. The correction is one line of evaluation code. We give the protocol an ablation-based causal number must report to be comparable across papers, and an audit of five published papers against it. In short: a causal number reported without its position describes the token it was taken at as much as the latent it was taken from.","url_abs":"https://arxiv.org/abs/2608.13337","url_pdf":"https://arxiv.org/pdf/2608.13337","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2608.13337","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.13337"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/vcnoel/sae-artifact","reach":null}],"summary":{"ran":11,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":12,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"94299288126e6e85","entry":"align_of","repo":"vcnoel/sae-artifact","repo_kind":"found_in_text","path":"src/checks3.py","file_url":"https://github.com/vcnoel/sae-artifact/blob/HEAD/src/checks3.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"94299288126e6e85"}},{"code_sha256_prefix":"760a8402ab173f27","entry":"ci","repo":"vcnoel/sae-artifact","repo_kind":"found_in_text","path":"src/bootstrap.py","file_url":"https://github.com/vcnoel/sae-artifact/blob/HEAD/src/bootstrap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"760a8402ab173f27"}},{"code_sha256_prefix":"5693a44195686e85","entry":"find_table","repo":"vcnoel/sae-artifact","repo_kind":"found_in_text","path":"src/extract_target_tables.py","file_url":"https://github.com/vcnoel/sae-artifact/blob/HEAD/src/extract_target_tables.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5693a44195686e85"}},{"code_sha256_prefix":"8cc3b0c8e9b1fd7d","entry":"gated","repo":"vcnoel/sae-artifact","repo_kind":"found_in_text","path":"src/eval_arms.py","file_url":"https://github.com/vcnoel/sae-artifact/blob/HEAD/src/eval_arms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8cc3b0c8e9b1fd7d"}},{"code_sha256_prefix":"33306bb1ab9320d9","entry":"maxcos","repo":"vcnoel/sae-artifact","repo_kind":"found_in_text","path":"src/checks3.py","file_url":"https://github.com/vcnoel/sae-artifact/blob/HEAD/src/checks3.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"33306bb1ab9320d9"}},{"code_sha256_prefix":"ffa7809e8dba9ffa","entry":"norm","repo":"vcnoel/sae-artifact","repo_kind":"found_in_text","path":"src/extract_target_tables.py","file_url":"https://github.com/vcnoel/sae-artifact/blob/HEAD/src/extract_target_tables.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ffa7809e8dba9ffa"}},{"code_sha256_prefix":"87d039ffad070cbb","entry":"parse_rows","repo":"vcnoel/sae-artifact","repo_kind":"found_in_text","path":"src/extract_target_tables.py","file_url":"https://github.com/vcnoel/sae-artifact/blob/HEAD/src/extract_target_tables.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87d039ffad070cbb"}},{"code_sha256_prefix":"e560d33b44bcdb15","entry":"ratio_ci","repo":"vcnoel/sae-artifact","repo_kind":"found_in_text","path":"src/checks3.py","file_url":"https://github.com/vcnoel/sae-artifact/blob/HEAD/src/checks3.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e560d33b44bcdb15"}},{"code_sha256_prefix":"c311f961113ff969","entry":"resid_cache","repo":"vcnoel/sae-artifact","repo_kind":"found_in_text","path":"src/sae_rare.py","file_url":"https://github.com/vcnoel/sae-artifact/blob/HEAD/src/sae_rare.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c311f961113ff969"}},{"code_sha256_prefix":"d0b88d403f3db497","entry":"sae_encode","repo":"vcnoel/sae-artifact","repo_kind":"found_in_text","path":"src/sae_rare.py","file_url":"https://github.com/vcnoel/sae-artifact/blob/HEAD/src/sae_rare.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d0b88d403f3db497"}},{"code_sha256_prefix":"26ef5b16d6e6a6d9","entry":"spec","repo":"vcnoel/sae-artifact","repo_kind":"found_in_text","path":"src/controls.py","file_url":"https://github.com/vcnoel/sae-artifact/blob/HEAD/src/controls.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"26ef5b16d6e6a6d9"}},{"code_sha256_prefix":"9077b9c1acfcd453","entry":"corpus","repo":"vcnoel/sae-artifact","repo_kind":"found_in_text","path":"src/sae_rare.py","file_url":"https://github.com/vcnoel/sae-artifact/blob/HEAD/src/sae_rare.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9077b9c1acfcd453"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}