{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-learning-high-precision-least-squares","title":"Towards Learning High-Precision Least Squares Algorithms with Sequence Models","arxiv_id":"2503.12295","date":"2025-03-15","proceeding":null,"authors":["Jerry Liu","Jessica Grogan","Owen Dugan","Ashish Rao","Simran Arora","Atri Rudra","Christopher Ré"],"abstract":"This paper investigates whether sequence models can learn to perform numerical algorithms, e.g. gradient descent, on the fundamental problem of least squares. Our goal is to inherit two properties of standard algorithms from numerical analysis: (1) machine precision, i.e. we want to obtain solutions that are accurate to near floating point error, and (2) numerical generality, i.e. we want them to apply broadly across problem instances. We find that prior approaches using Transformers fail to meet these criteria, and identify limitations present in existing architectures and training procedures. First, we show that softmax Transformers struggle to perform high-precision multiplications, which prevents them from precisely learning numerical algorithms. Second, we identify an alternate class of architectures, comprised entirely of polynomials, that can efficiently represent high-precision gradient descent iterates. Finally, we investigate precision bottlenecks during training and address them via a high-precision training recipe that reduces stochastic gradient noise. Our recipe enables us to train two polynomial architectures, gated convolutions and linear attention, to perform gradient descent iterates on least squares problems. For the first time, we demonstrate the ability to train to near machine precision. Applied iteratively, our models obtain 100,000x lower MSE than standard Transformers trained end-to-end and they incur a 10,000x smaller generalization gap on out-of-distribution problems. We make progress towards end-to-end learning of numerical algorithms for least squares.","url_abs":"https://arxiv.org/abs/2503.12295v1","url_pdf":"https://arxiv.org/pdf/2503.12295v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-learning-high-precision-least-squares","repo_url":"https://github.com/HazyResearch/precision-ls","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[],"methods":[{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.12295","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.12295"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/HazyResearch/precision-ls","reach":null}],"summary":{"ran":4,"unverified":6},"by_repo_kind":{"official":{"samples":10,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"099feb2cb11280c5","entry":"CausalSelfAttention","repo":"HazyResearch/precision-ls","repo_kind":"official","path":"src/models/gpt2.py","file_url":"https://github.com/HazyResearch/precision-ls/blob/HEAD/src/models/gpt2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"099feb2cb11280c5"}},{"code_sha256_prefix":"9e9cb8d5e05b5a61","entry":"GLU","repo":"HazyResearch/precision-ls","repo_kind":"official","path":"src/models/gpt2.py","file_url":"https://github.com/HazyResearch/precision-ls/blob/HEAD/src/models/gpt2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e9cb8d5e05b5a61"}},{"code_sha256_prefix":"57d4e2dacaca4be8","entry":"GPT2Config","repo":"HazyResearch/precision-ls","repo_kind":"official","path":"src/models/gpt2.py","file_url":"https://github.com/HazyResearch/precision-ls/blob/HEAD/src/models/gpt2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57d4e2dacaca4be8"}},{"code_sha256_prefix":"813516a2674fe31a","entry":"MLP","repo":"HazyResearch/precision-ls","repo_kind":"official","path":"src/models/gpt2.py","file_url":"https://github.com/HazyResearch/precision-ls/blob/HEAD/src/models/gpt2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"813516a2674fe31a"}},{"code_sha256_prefix":"6bf621cd35e95f81","entry":"BaseConv","repo":"HazyResearch/precision-ls","repo_kind":"official","path":"src/models/gpt2.py","file_url":"https://github.com/HazyResearch/precision-ls/blob/HEAD/src/models/gpt2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6bf621cd35e95f81"}},{"code_sha256_prefix":"d2f4bda71035892b","entry":"BaseConvLayer","repo":"HazyResearch/precision-ls","repo_kind":"official","path":"src/models/gpt2.py","file_url":"https://github.com/HazyResearch/precision-ls/blob/HEAD/src/models/gpt2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2f4bda71035892b"}},{"code_sha256_prefix":"a0745a160f92b244","entry":"Block","repo":"HazyResearch/precision-ls","repo_kind":"official","path":"src/models/gpt2.py","file_url":"https://github.com/HazyResearch/precision-ls/blob/HEAD/src/models/gpt2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0745a160f92b244"}},{"code_sha256_prefix":"d64f15e110eb25fe","entry":"GPT2Model","repo":"HazyResearch/precision-ls","repo_kind":"official","path":"src/models/gpt2.py","file_url":"https://github.com/HazyResearch/precision-ls/blob/HEAD/src/models/gpt2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d64f15e110eb25fe"}},{"code_sha256_prefix":"46aa5e22dbe3a170","entry":"TransformerModel","repo":"HazyResearch/precision-ls","repo_kind":"official","path":"src/models/gpt2.py","file_url":"https://github.com/HazyResearch/precision-ls/blob/HEAD/src/models/gpt2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"46aa5e22dbe3a170"}},{"code_sha256_prefix":"364a8c5607cc6f21","entry":"set_param","repo":"HazyResearch/precision-ls","repo_kind":"official","path":"src/models/gpt2.py","file_url":"https://github.com/HazyResearch/precision-ls/blob/HEAD/src/models/gpt2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"364a8c5607cc6f21"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}