{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tensor-switching-networks","title":"Tensor Switching Networks","arxiv_id":"1610.10087","date":"2016-10-31","proceeding":"NeurIPS 2016 12","authors":["Chuan-Yung Tsai","Andrew Saxe","David Cox"],"abstract":"We present a novel neural network algorithm, the Tensor Switching (TS)\nnetwork, which generalizes the Rectified Linear Unit (ReLU) nonlinearity to\ntensor-valued hidden units. The TS network copies its entire input vector to\ndifferent locations in an expanded representation, with the location determined\nby its hidden unit activity. In this way, even a simple linear readout from the\nTS representation can implement a highly expressive deep-network-like function.\nThe TS network hence avoids the vanishing gradient problem by construction, at\nthe cost of larger representation size. We develop several methods to train the\nTS network, including equivalent kernels for infinitely wide and deep TS\nnetworks, a one-pass linear learning algorithm, and two\nbackpropagation-inspired representation learning algorithms. Our experimental\nresults demonstrate that the TS network is indeed more expressive and\nconsistently learns faster than standard ReLU networks.","url_abs":"http://arxiv.org/abs/1610.10087v1","url_pdf":"http://arxiv.org/pdf/1610.10087v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tensor-switching-networks","repo_url":"https://github.com/coxlab/tsnet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"ts","method_name":"TS"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}