{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/quasi-newton-methods-for-deep-learning-forget","title":"Quasi-Newton Methods for Machine Learning: Forget the Past, Just Sample","arxiv_id":"1901.09997","date":"2019-01-28","proceeding":null,"authors":["Albert S. Berahas","Majid Jahani","Peter Richtárik","Martin Takáč"],"abstract":"We present two sampled quasi-Newton methods (sampled LBFGS and sampled LSR1) for solving empirical risk minimization problems that arise in machine learning. Contrary to the classical variants of these methods that sequentially build Hessian or inverse Hessian approximations as the optimization progresses, our proposed methods sample points randomly around the current iterate at every iteration to produce these approximations. As a result, the approximations constructed make use of more reliable (recent and local) information, and do not depend on past iterate information that could be significantly stale. Our proposed algorithms are efficient in terms of accessed data points (epochs) and have enough concurrency to take advantage of parallel/distributed computing environments. We provide convergence guarantees for our proposed methods. Numerical tests on a toy classification problem as well as on popular benchmarking binary classification and neural network training tasks reveal that the methods outperform their classical variants.","url_abs":"https://arxiv.org/abs/1901.09997v5","url_pdf":"https://arxiv.org/pdf/1901.09997v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"quasi-newton-methods-for-deep-learning-forget","repo_url":"https://github.com/OptMLGroup/SQN","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"binary-classification","task_name":"Binary Classification"},{"task_slug":"distributed-computing","task_name":"Distributed Computing"},{"task_slug":"classification","task_name":"General Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1901.09997","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1901.09997"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/OptMLGroup/SQN","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7f5cb0801555fb37","entry":"getData","repo":"OptMLGroup/SQN","repo_kind":"official","path":"data_generation.py","file_url":"https://github.com/OptMLGroup/SQN/blob/HEAD/data_generation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f5cb0801555fb37"}},{"code_sha256_prefix":"ff04bb6f22f87337","entry":"sample_pairs_SY_SLBFGS","repo":"OptMLGroup/SQN","repo_kind":"official","path":"sampleSY.py","file_url":"https://github.com/OptMLGroup/SQN/blob/HEAD/sampleSY.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff04bb6f22f87337"}},{"code_sha256_prefix":"61060cd20bbc90c4","entry":"sample_pairs_SY_SLSR1","repo":"OptMLGroup/SQN","repo_kind":"official","path":"sampleSY.py","file_url":"https://github.com/OptMLGroup/SQN/blob/HEAD/sampleSY.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61060cd20bbc90c4"}},{"code_sha256_prefix":"242517e581c4f091","entry":"weight_variable","repo":"OptMLGroup/SQN","repo_kind":"official","path":"network.py","file_url":"https://github.com/OptMLGroup/SQN/blob/HEAD/network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"242517e581c4f091"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}