{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/random-projection-in-deep-neural-networks","title":"Random Projection in Deep Neural Networks","arxiv_id":"1812.09489","date":"2018-12-22","proceeding":null,"authors":["Piotr Iwo Wójcik"],"abstract":"This work investigates the ways in which deep learning methods can benefit\nfrom random projection (RP), a classic linear dimensionality reduction method.\nWe focus on two areas where, as we have found, employing RP techniques can\nimprove deep models: training neural networks on high-dimensional data and\ninitialization of network parameters. Training deep neural networks (DNNs) on\nsparse, high-dimensional data with no exploitable structure implies a network\narchitecture with an input layer that has a huge number of weights, which often\nmakes training infeasible. We show that this problem can be solved by\nprepending the network with an input layer whose weights are initialized with\nan RP matrix. We propose several modifications to the network architecture and\ntraining regime that makes it possible to efficiently train DNNs with learnable\nRP layer on data with as many as tens of millions of input features and\ntraining examples. In comparison to the state-of-the-art methods, neural\nnetworks with RP layer achieve competitive performance or improve the results\non several extremely high-dimensional real-world datasets. The second area\nwhere the application of RP techniques can be beneficial for training deep\nmodels is weight initialization. Setting the initial weights in DNNs to\nelements of various RP matrices enabled us to train residual deep networks to\nhigher levels of performance.","url_abs":"http://arxiv.org/abs/1812.09489v1","url_pdf":"http://arxiv.org/pdf/1812.09489v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"random-projection-in-deep-neural-networks","repo_url":"https://github.com/piotriwojcik/rpinit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"dimensionality-reduction","task_name":"Dimensionality Reduction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}