{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-08464","title":"The Geometric Structure of Models Learning Sparse Data","arxiv_id":"2605.08464","date":"2026-05-08","proceeding":null,"authors":["Thomas Walker","T. Mitchell Roddenberry","Ahmed Imtiaz Humayun","Randall Balestriero","Richard Baraniuk"],"abstract":"The manifold hypothesis (MH) is often used to explain how machine learning can overcome the curse of dimensionality. However, the MH is only applicable in regimes where the training data provides a sufficiently dense sample of the underlying low-dimensional data manifold, or where such a low-dimensional manifold is conceivably present. We describe the regimes where the MH is not applicable as sparse. In this paper, we demonstrate that models succeed in the sparse regime by exploiting a highly structured local geometry, a property we formalize as normal alignment. We prove that normal-aligned classifiers -- whose input-output Jacobians are rank-one and align perfectly with the training data -- minimize the training objective under norm constraints and achieve maximal local robustness under a non-zero Jacobian constraint. For continuous piecewise-affine deep networks, normal alignment manifests geometrically as centroid alignment within the network's induced power diagram partition and results from the feature-learning regime. Motivated by these theoretical insights, we introduce GrokAlign, a regularization strategy that actively induces normal alignment. We demonstrate that GrokAlign significantly accelerates the training dynamics of deep networks relevant to the grokking phenomenon. Furthermore, we apply the principle of normal alignment to Recursive Feature Machines (RFMs) to introduce Recursive Feature Alignment Machines (RFAMs). We show that RFAMs exhibit greater adversarial robustness compared to RFMs when trained on tabular data.","url_abs":"https://arxiv.org/abs/2605.08464","url_pdf":"https://arxiv.org/pdf/2605.08464","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.08464","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.08464"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ThomasWalker1/GrokAlign","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ThomasWalker1/RFAM","reach":null}],"summary":{"ran":1,"ran_draft_wrong":3,"ran_fixture":3,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":8,"ran":7,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"39a53f534f187f83","entry":"GrokAlign","repo":"ThomasWalker1/GrokAlign","repo_kind":"found_in_text","path":"regularization/grokalign.py","file_url":"https://github.com/ThomasWalker1/GrokAlign/blob/HEAD/regularization/grokalign.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39a53f534f187f83"}},{"code_sha256_prefix":"50468800d601109d","entry":"convert_one_hot","repo":"ThomasWalker1/RFAM","repo_kind":"found_in_text","path":"utils/rfm.py","file_url":"https://github.com/ThomasWalker1/RFAM/blob/HEAD/utils/rfm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50468800d601109d"}},{"code_sha256_prefix":"f9ff1cb6c9e8d20a","entry":"euclidean_distances_M","repo":"ThomasWalker1/RFAM","repo_kind":"found_in_text","path":"utils/rfm.py","file_url":"https://github.com/ThomasWalker1/RFAM/blob/HEAD/utils/rfm.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9ff1cb6c9e8d20a"}},{"code_sha256_prefix":"e1df35b5a8dddcb2","entry":"laplace_kernel_M","repo":"ThomasWalker1/RFAM","repo_kind":"found_in_text","path":"utils/rfm.py","file_url":"https://github.com/ThomasWalker1/RFAM/blob/HEAD/utils/rfm.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e1df35b5a8dddcb2"}},{"code_sha256_prefix":"8968b47683e94f61","entry":"laplacian_M","repo":"ThomasWalker1/RFAM","repo_kind":"found_in_text","path":"utils/rfm.py","file_url":"https://github.com/ThomasWalker1/RFAM/blob/HEAD/utils/rfm.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8968b47683e94f61"}},{"code_sha256_prefix":"a09edc93d11c9d93","entry":"pgd_attack","repo":"ThomasWalker1/RFAM","repo_kind":"found_in_text","path":"utils/rfm.py","file_url":"https://github.com/ThomasWalker1/RFAM/blob/HEAD/utils/rfm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a09edc93d11c9d93"}},{"code_sha256_prefix":"6ec0ce87d7ed07f5","entry":"train","repo":"ThomasWalker1/RFAM","repo_kind":"found_in_text","path":"utils/rfm.py","file_url":"https://github.com/ThomasWalker1/RFAM/blob/HEAD/utils/rfm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ec0ce87d7ed07f5"}},{"code_sha256_prefix":"d43a7623958fe7e6","entry":"get_grads","repo":"ThomasWalker1/RFAM","repo_kind":"found_in_text","path":"utils/rfm.py","file_url":"https://github.com/ThomasWalker1/RFAM/blob/HEAD/utils/rfm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d43a7623958fe7e6"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}