{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-08192","title":"Inside-Out: Measuring Generalization in Vision Transformers Through Inner Workings","arxiv_id":"2604.08192","date":"2026-04-09","proceeding":null,"authors":["Yunxiang Peng","Mengmeng Ma","Ziyu Yao","Xi Peng"],"abstract":"Reliable generalization metrics are fundamental to the evaluation of machine learning models. Especially in high-stakes applications where labeled target data are scarce, evaluation of models' generalization performance under distribution shift is a pressing need. We focus on two practical scenarios: (1) Before deployment, how to select the best model for unlabeled target data? (2) After deployment, how to monitor model performance under distribution shift? The central need in both cases is a reliable and label-free proxy metric. Yet existing proxy metrics, such as model confidence or accuracy-on-the-line, are often unreliable as they only assess model output while ignoring the internal mechanisms that produce them. We address this limitation by introducing a new perspective: using the inner workings of a model, i.e., circuits, as a predictive metric of generalization performance. Leveraging circuit discovery, we extract the causal interactions between internal representations as a circuit, from which we derive two metrics tailored to the two practical scenarios. (1) Before deployment, we introduce Dependency Depth Bias, which measures different models' generalization capability on target data. (2) After deployment, we propose Circuit Shift Score, which predicts a model's generalization under different distribution shifts. Across various tasks, both metrics demonstrate significantly improved correlation with generalization performance, outperforming existing proxies by an average of 13.4\\% and 34.1\\%, respectively. Our code is available at https://github.com/deep-real/GenCircuit.","url_abs":"https://arxiv.org/abs/2604.08192","url_pdf":"https://arxiv.org/pdf/2604.08192","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.08192"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/deep-real/GenCircuit","reach":null}],"summary":{"unverified":15},"by_repo_kind":{"found_in_text":{"samples":15,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"62435ebddf0bbc18","entry":"bernoulli_kl","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"dataset_utils.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"62435ebddf0bbc18"}},{"code_sha256_prefix":"f663999fde37d3ca","entry":"collate_EAP","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"dataset.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f663999fde37d3ca"}},{"code_sha256_prefix":"d74b7285b967fee4","entry":"compute_groupwise_accuracy","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"model_zoo/acc_eval.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/model_zoo/acc_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d74b7285b967fee4"}},{"code_sha256_prefix":"cbb0c42794d625dd","entry":"convert_hf_to_hookedvit","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"model_utils.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cbb0c42794d625dd"}},{"code_sha256_prefix":"e4314f14cae629b0","entry":"flatten_metric_output","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"compute_metrics.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/compute_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4314f14cae629b0"}},{"code_sha256_prefix":"7d7e38335fc8dce2","entry":"get_failure_list","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"dataset_utils.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d7e38335fc8dce2"}},{"code_sha256_prefix":"86e7460cce433ff9","entry":"get_layer_index","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"compute_metrics.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/compute_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"86e7460cce433ff9"}},{"code_sha256_prefix":"2732127dfe219441","entry":"get_transform","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"dataset_utils.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2732127dfe219441"}},{"code_sha256_prefix":"9f7585ff134e843d","entry":"jacobian","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"model_zoo/coupling_jacobian.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/model_zoo/coupling_jacobian.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f7585ff134e843d"}},{"code_sha256_prefix":"1a6f60a202e7f9a5","entry":"normalized_logit","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"metrics.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a6f60a202e7f9a5"}},{"code_sha256_prefix":"689bb797ea2eebd9","entry":"numel","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"model_zoo/cmnist_consistency.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/model_zoo/cmnist_consistency.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"689bb797ea2eebd9"}},{"code_sha256_prefix":"80901b7355cb5d7d","entry":"randomsvd","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"model_zoo/coupling_jacobian.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/model_zoo/coupling_jacobian.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80901b7355cb5d7d"}},{"code_sha256_prefix":"f1d6ff8b5691ff6a","entry":"svd","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"model_zoo/coupling_jacobian.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/model_zoo/coupling_jacobian.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f1d6ff8b5691ff6a"}},{"code_sha256_prefix":"0a359b702f5ab47d","entry":"topk_accuracy","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"metrics.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0a359b702f5ab47d"}},{"code_sha256_prefix":"8a9cf5ce54bd0f16","entry":"vit_hidden_states_timm","repo":"deep-real/GenCircuit","repo_kind":"found_in_text","path":"model_zoo/coupling_eval.py","file_url":"https://github.com/deep-real/GenCircuit/blob/HEAD/model_zoo/coupling_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a9cf5ce54bd0f16"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}