{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/linear-recursive-feature-machines-provably","title":"Linear Recursive Feature Machines provably recover low-rank matrices","arxiv_id":"2401.04553","date":"2024-01-09","proceeding":null,"authors":["Adityanarayanan Radhakrishnan","Mikhail Belkin","Dmitriy Drusvyatskiy"],"abstract":"A fundamental problem in machine learning is to understand how neural networks make accurate predictions, while seemingly bypassing the curse of dimensionality. A possible explanation is that common training algorithms for neural networks implicitly perform dimensionality reduction - a process called feature learning. Recent work posited that the effects of feature learning can be elicited from a classical statistical estimator called the average gradient outer product (AGOP). The authors proposed Recursive Feature Machines (RFMs) as an algorithm that explicitly performs feature learning by alternating between (1) reweighting the feature vectors by the AGOP and (2) learning the prediction function in the transformed space. In this work, we develop the first theoretical guarantees for how RFM performs dimensionality reduction by focusing on the class of overparametrized problems arising in sparse linear regression and low-rank matrix recovery. Specifically, we show that RFM restricted to linear models (lin-RFM) generalizes the well-studied Iteratively Reweighted Least Squares (IRLS) algorithm. Our results shed light on the connection between feature learning in neural networks and classical sparse recovery algorithms. In addition, we provide an implementation of lin-RFM that scales to matrices with millions of missing entries. Our implementation is faster than the standard IRLS algorithm as it is SVD-free. It also outperforms deep linear networks for sparse linear regression and low-rank matrix completion.","url_abs":"https://arxiv.org/abs/2401.04553v1","url_pdf":"https://arxiv.org/pdf/2401.04553v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"linear-recursive-feature-machines-provably","repo_url":"https://github.com/aradha/lin-rfm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"dimensionality-reduction","task_name":"Dimensionality Reduction"},{"task_slug":"low-rank-matrix-completion","task_name":"Low-Rank Matrix Completion"},{"task_slug":"matrix-completion","task_name":"Matrix Completion"},{"task_slug":"regression-1","task_name":"regression"}],"methods":[{"method_slug":"linear-regression","method_name":"Linear Regression"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.04553","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.04553"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aradha/lin-rfm","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"42418931deeb02ca","entry":"get_data","repo":"aradha/lin-rfm","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/aradha/lin-rfm/blob/HEAD/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42418931deeb02ca"}},{"code_sha256_prefix":"f9e908c952d2d920","entry":"linear_rfm","repo":"aradha/lin-rfm","repo_kind":"official","path":"linear_rfm.py","file_url":"https://github.com/aradha/lin-rfm/blob/HEAD/linear_rfm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9e908c952d2d920"}},{"code_sha256_prefix":"53d01cd28fb4f759","entry":"rfm","repo":"aradha/lin-rfm","repo_kind":"official","path":"svd_free_lin_rfm.py","file_url":"https://github.com/aradha/lin-rfm/blob/HEAD/svd_free_lin_rfm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"53d01cd28fb4f759"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}