{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/accelerated-stochastic-power-iteration","title":"Accelerated Stochastic Power Iteration","arxiv_id":"1707.02670","date":"2017-07-10","proceeding":null,"authors":["Christopher De Sa","Bryan He","Ioannis Mitliagkas","Christopher Ré","Peng Xu"],"abstract":"Principal component analysis (PCA) is one of the most powerful tools in\nmachine learning. The simplest method for PCA, the power iteration, requires\n$\\mathcal O(1/\\Delta)$ full-data passes to recover the principal component of a\nmatrix with eigen-gap $\\Delta$. Lanczos, a significantly more complex method,\nachieves an accelerated rate of $\\mathcal O(1/\\sqrt{\\Delta})$ passes. Modern\napplications, however, motivate methods that only ingest a subset of available\ndata, known as the stochastic setting. In the online stochastic setting, simple\nalgorithms like Oja's iteration achieve the optimal sample complexity $\\mathcal\nO(\\sigma^2/\\Delta^2)$. Unfortunately, they are fully sequential, and also\nrequire $\\mathcal O(\\sigma^2/\\Delta^2)$ iterations, far from the $\\mathcal\nO(1/\\sqrt{\\Delta})$ rate of Lanczos. We propose a simple variant of the power\niteration with an added momentum term, that achieves both the optimal sample\nand iteration complexity. In the full-pass setting, standard analysis shows\nthat momentum achieves the accelerated rate, $\\mathcal O(1/\\sqrt{\\Delta})$. We\ndemonstrate empirically that naively applying momentum to a stochastic method,\ndoes not result in acceleration. We perform a novel, tight variance analysis\nthat reveals the \"breaking-point variance\" beyond which this acceleration does\nnot occur. By combining this insight with modern variance reduction techniques,\nwe construct stochastic PCA algorithms, for the online and offline setting,\nthat achieve an accelerated iteration complexity $\\mathcal O(1/\\sqrt{\\Delta})$.\nDue to the embarassingly parallel nature of our methods, this acceleration\ntranslates directly to wall-clock time if deployed in a parallel environment.\nOur approach is very general, and applies to many non-convex optimization\nproblems that can now be accelerated using the same technique.","url_abs":"http://arxiv.org/abs/1707.02670v1","url_pdf":"http://arxiv.org/pdf/1707.02670v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"accelerated-stochastic-power-iteration","repo_url":"https://github.com/ABadCandy/HessianDL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"accelerated-stochastic-power-iteration","repo_url":"https://github.com/noahgolmant/pytorch-hessian-eigenthings","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"dimensionality-reduction","task_name":"Dimensionality Reduction"}],"methods":[{"method_slug":"pca","method_name":"PCA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1707.02670","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1707.02670"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ABadCandy/HessianDL","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/noahgolmant/pytorch-hessian-eigenthings","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2},"by_repo_kind":{},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"4532c973ff280886","entry":"deflated_power_iteration","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"4532c973ff280886"}},{"code_sha256_prefix":"604f7701e0559dac","entry":"power_iteration","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"604f7701e0559dac"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}