{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-equivalence-between-neural-network-and","title":"On the Equivalence between Neural Network and Support Vector Machine","arxiv_id":"2111.06063","date":"2021-11-11","proceeding":"NeurIPS 2021 12","authors":["Yilan Chen","Wei Huang","Lam M. Nguyen","Tsui-Wei Weng"],"abstract":"Recent research shows that the dynamics of an infinitely wide neural network (NN) trained by gradient descent can be characterized by Neural Tangent Kernel (NTK) \\citep{jacot2018neural}. Under the squared loss, the infinite-width NN trained by gradient descent with an infinitely small learning rate is equivalent to kernel regression with NTK \\citep{arora2019exact}. However, the equivalence is only known for ridge regression currently \\citep{arora2019harnessing}, while the equivalence between NN and other kernel machines (KMs), e.g. support vector machine (SVM), remains unknown. Therefore, in this work, we propose to establish the equivalence between NN and SVM, and specifically, the infinitely wide NN trained by soft margin loss and the standard soft margin SVM with NTK trained by subgradient descent. Our main theoretical results include establishing the equivalences between NNs and a broad family of $\\ell_2$ regularized KMs with finite-width bounds, which cannot be handled by prior work, and showing that every finite-width NN trained by such regularized loss functions is approximately a KM. Furthermore, we demonstrate our theory can enable three practical applications, including (i) \\textit{non-vacuous} generalization bound of NN via the corresponding KM; (ii) \\textit{non-trivial} robustness certificate for the infinite-width NN (while existing robustness verification methods would provide vacuous bounds); (iii) intrinsically more robust infinite-width NNs than those from previous kernel regression. Our code for the experiments is available at \\url{https://github.com/leslie-CH/equiv-nn-svm}.","url_abs":"https://arxiv.org/abs/2111.06063v2","url_pdf":"https://arxiv.org/pdf/2111.06063v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-equivalence-between-neural-network-and","repo_url":"https://github.com/leslie-ch/equiv-nn-svm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"regression-1","task_name":"regression"}],"methods":[{"method_slug":"ntk","method_name":"NTK"},{"method_slug":"svm","method_name":"SVM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2111.06063","atlas_url":"https://app.syntology.ai/?focus=2111.06063","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.06063"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/leslie-CH/equiv-nn-svm","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leslie-ch/equiv-nn-svm","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":9,"unverified":1},"by_repo_kind":{"official":{"samples":10,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"69017b3a991e0815","entry":"activation_bound","repo":"leslie-CH/equiv-nn-svm","repo_kind":"official","path":"ibp.py","file_url":"https://github.com/leslie-CH/equiv-nn-svm/blob/HEAD/ibp.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"69017b3a991e0815"}},{"code_sha256_prefix":"5966a7c925b28912","entry":"binary_search","repo":"leslie-CH/equiv-nn-svm","repo_kind":"official","path":"robust_svm.py","file_url":"https://github.com/leslie-CH/equiv-nn-svm/blob/HEAD/robust_svm.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5966a7c925b28912"}},{"code_sha256_prefix":"70beef44814477b6","entry":"bound_ntk","repo":"leslie-CH/equiv-nn-svm","repo_kind":"official","path":"robust_svm.py","file_url":"https://github.com/leslie-CH/equiv-nn-svm/blob/HEAD/robust_svm.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"70beef44814477b6"}},{"code_sha256_prefix":"fc955f4c0c5be375","entry":"h_map","repo":"leslie-CH/equiv-nn-svm","repo_kind":"official","path":"robust_svm.py","file_url":"https://github.com/leslie-CH/equiv-nn-svm/blob/HEAD/robust_svm.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fc955f4c0c5be375"}},{"code_sha256_prefix":"cfb1c3dd218b92f0","entry":"initial_bounds","repo":"leslie-CH/equiv-nn-svm","repo_kind":"official","path":"ibp.py","file_url":"https://github.com/leslie-CH/equiv-nn-svm/blob/HEAD/ibp.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cfb1c3dd218b92f0"}},{"code_sha256_prefix":"81ac7e5de8c2e14b","entry":"load_hparam","repo":"leslie-CH/equiv-nn-svm","repo_kind":"official","path":"functions/hparams.py","file_url":"https://github.com/leslie-CH/equiv-nn-svm/blob/HEAD/functions/hparams.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"81ac7e5de8c2e14b"}},{"code_sha256_prefix":"bdd93f20e2c85cfa","entry":"make_layers","repo":"leslie-CH/equiv-nn-svm","repo_kind":"official","path":"models/model.py","file_url":"https://github.com/leslie-CH/equiv-nn-svm/blob/HEAD/models/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bdd93f20e2c85cfa"}},{"code_sha256_prefix":"0cfe2baee7e49f75","entry":"merge_dict","repo":"leslie-CH/equiv-nn-svm","repo_kind":"official","path":"functions/hparams.py","file_url":"https://github.com/leslie-CH/equiv-nn-svm/blob/HEAD/functions/hparams.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0cfe2baee7e49f75"}},{"code_sha256_prefix":"f928c5703d66d699","entry":"weighted_bound","repo":"leslie-CH/equiv-nn-svm","repo_kind":"official","path":"ibp.py","file_url":"https://github.com/leslie-CH/equiv-nn-svm/blob/HEAD/ibp.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f928c5703d66d699"}},{"code_sha256_prefix":"70da19149f22400f","entry":"load_hparam_str","repo":"leslie-CH/equiv-nn-svm","repo_kind":"official","path":"functions/hparams.py","file_url":"https://github.com/leslie-CH/equiv-nn-svm/blob/HEAD/functions/hparams.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"70da19149f22400f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}