{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/injectivity-of-relu-networks-perspectives","title":"Injectivity of ReLU networks: perspectives from statistical physics","arxiv_id":"2302.14112","date":"2023-02-27","proceeding":null,"authors":["Antoine Maillard","Afonso S. Bandeira","David Belius","Ivan Dokmanić","Shuta Nakajima"],"abstract":"When can the input of a ReLU neural network be inferred from its output? In other words, when is the network injective? We consider a single layer, $x \\mapsto \\mathrm{ReLU}(Wx)$, with a random Gaussian $m \\times n$ matrix $W$, in a high-dimensional setting where $n, m \\to \\infty$. Recent work connects this problem to spherical integral geometry giving rise to a conjectured sharp injectivity threshold for $\\alpha = \\frac{m}{n}$ by studying the expected Euler characteristic of a certain random set. We adopt a different perspective and show that injectivity is equivalent to a property of the ground state of the spherical perceptron, an important spin glass model in statistical physics. By leveraging the (non-rigorous) replica symmetry-breaking theory, we derive analytical equations for the threshold whose solution is at odds with that from the Euler characteristic. Furthermore, we use Gordon's min--max theorem to prove that a replica-symmetric upper bound refutes the Euler characteristic prediction. Along the way we aim to give a tutorial-style introduction to key ideas from statistical physics in an effort to make the exposition accessible to a broad audience. Our analysis establishes a connection between spin glasses and integral geometry but leaves open the problem of explaining the discrepancies.","url_abs":"https://arxiv.org/abs/2302.14112v2","url_pdf":"https://arxiv.org/pdf/2302.14112v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"injectivity-of-relu-networks-perspectives","repo_url":"https://github.com/anmaillard/injectivity_relu_layer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}