{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/surfing-iterative-optimization-over","title":"Surfing: Iterative optimization over incrementally trained deep networks","arxiv_id":"1907.08653","date":"2019-07-19","proceeding":"NeurIPS 2019 12","authors":["Ganlin Song","Zhou Fan","John Lafferty"],"abstract":"We investigate a sequential optimization procedure to minimize the empirical risk functional $f_{\\hat\\theta}(x) = \\frac{1}{2}\\|G_{\\hat\\theta}(x) - y\\|^2$ for certain families of deep networks $G_{\\theta}(x)$. The approach is to optimize a sequence of objective functions that use network parameters obtained during different stages of the training process. When initialized with random parameters $\\theta_0$, we show that the objective $f_{\\theta_0}(x)$ is \"nice'' and easy to optimize with gradient descent. As learning is carried out, we obtain a sequence of generative networks $x \\mapsto G_{\\theta_t}(x)$ and associated risk functions $f_{\\theta_t}(x)$, where $t$ indicates a stage of stochastic gradient descent during training. Since the parameters of the network do not change by very much in each step, the surface evolves slowly and can be incrementally optimized. The algorithm is formalized and analyzed for a family of expansive networks. We call the procedure {\\it surfing} since it rides along the peak of the evolving (negative) empirical risk function, starting from a smooth surface at the beginning of learning and ending with a wavy nonconvex surface after learning is complete. Experiments show how surfing can be used to find the global optimum and for compressed sensing even when direct gradient descent on the final learned network fails.","url_abs":"https://arxiv.org/abs/1907.08653v1","url_pdf":"https://arxiv.org/pdf/1907.08653v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"surfing-iterative-optimization-over","repo_url":"https://github.com/jdlafferty/surfing","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"compressed-sensing","task_name":"compressed sensing"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1907.08653","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1907.08653"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jdlafferty/surfing","reach":null}],"summary":{"ran_honours":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"66fea652ddd0779c","entry":"clip","repo":"jdlafferty/surfing","repo_kind":"official","path":"codes/Fashion-MNIST/DCGAN/GD_surfing.py","file_url":"https://github.com/jdlafferty/surfing/blob/HEAD/codes/Fashion-MNIST/DCGAN/GD_surfing.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"66fea652ddd0779c"}},{"code_sha256_prefix":"c4d78cc22ede4945","entry":"dist","repo":"jdlafferty/surfing","repo_kind":"official","path":"codes/Fashion-MNIST/DCGAN/GD_surfing.py","file_url":"https://github.com/jdlafferty/surfing/blob/HEAD/codes/Fashion-MNIST/DCGAN/GD_surfing.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c4d78cc22ede4945"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}