{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neural-networks-fail-to-learn-periodic","title":"Neural Networks Fail to Learn Periodic Functions and How to Fix It","arxiv_id":"2006.08195","date":"2020-06-15","proceeding":"NeurIPS 2020 12","authors":["Liu Ziyin","Tilman Hartwig","Masahito Ueda"],"abstract":"Previous literature offers limited clues on how to learn a periodic function using modern neural networks. We start with a study of the extrapolation properties of neural networks; we prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a \"periodic\" inductive bias. As a fix of this problem, we propose a new activation, namely, $x + \\sin^2(x)$, which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations. Experimentally, we apply the proposed method to temperature and financial data prediction.","url_abs":"https://arxiv.org/abs/2006.08195v2","url_pdf":"https://arxiv.org/pdf/2006.08195v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neural-networks-fail-to-learn-periodic","repo_url":"https://github.com/AdenosHermes/NeurIPS_2020_Snake","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"neural-networks-fail-to-learn-periodic","repo_url":"https://github.com/yingkaisha/keras-unet-collection","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"neural-networks-fail-to-learn-periodic","repo_url":"https://github.com/mayurak47/Reproducibility_Challenge","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"inductive-bias","task_name":"Inductive Bias"}],"methods":[{"method_slug":"relu","method_name":"ReLU"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2006.08195","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.08195"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AdenosHermes/NeurIPS_2020_Snake","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yingkaisha/keras-unet-collection","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mayurak47/Reproducibility_Challenge","reach":null}],"summary":{"ran":2,"ran_honours":1},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"625b79d177e80e65","entry":"Snake","repo":"yingkaisha/keras-unet-collection","repo_kind":"listed","path":"keras_unet_collection/activations.py","file_url":"https://github.com/yingkaisha/keras-unet-collection/blob/HEAD/keras_unet_collection/activations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"625b79d177e80e65"}},{"code_sha256_prefix":"d4b4d2bd7dd9f9e0","entry":"XPlusSinActivation","repo":"mayurak47/Reproducibility_Challenge","repo_kind":"listed","path":"models/models.py","file_url":"https://github.com/mayurak47/Reproducibility_Challenge/blob/HEAD/models/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d4b4d2bd7dd9f9e0"}},{"code_sha256_prefix":"b591afd5922c4413","entry":"snake_","repo":"yingkaisha/keras-unet-collection","repo_kind":"listed","path":"keras_unet_collection/activations.py","file_url":"https://github.com/yingkaisha/keras-unet-collection/blob/HEAD/keras_unet_collection/activations.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b591afd5922c4413"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}