{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deep-learning-as-a-mixed-convex-combinatorial","title":"Deep Learning as a Mixed Convex-Combinatorial Optimization Problem","arxiv_id":"1710.11573","date":"2017-10-31","proceeding":"ICLR 2018 1","authors":["Abram L. Friesen","Pedro Domingos"],"abstract":"As neural networks grow deeper and wider, learning networks with\nhard-threshold activations is becoming increasingly important, both for network\nquantization, which can drastically reduce time and energy requirements, and\nfor creating large integrated systems of deep networks, which may have\nnon-differentiable components and must avoid vanishing and exploding gradients\nfor effective learning. However, since gradient descent is not applicable to\nhard-threshold functions, it is not clear how to learn networks of them in a\nprincipled way. We address this problem by observing that setting targets for\nhard-threshold hidden units in order to minimize loss is a discrete\noptimization problem, and can be solved as such. The discrete optimization goal\nis to find a set of targets such that each unit, including the output, has a\nlinearly separable problem to solve. Given these targets, the network\ndecomposes into individual perceptrons, which can then be learned with standard\nconvex approaches. Based on this, we develop a recursive mini-batch algorithm\nfor learning deep hard-threshold networks that includes the popular but poorly\njustified straight-through estimator as a special case. Empirically, we show\nthat our algorithm improves classification accuracy in a number of settings,\nincluding for AlexNet and ResNet-18 on ImageNet, when compared to the\nstraight-through estimator.","url_abs":"http://arxiv.org/abs/1710.11573v3","url_pdf":"http://arxiv.org/pdf/1710.11573v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deep-learning-as-a-mixed-convex-combinatorial","repo_url":"https://github.com/afriesen/ftprop","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"combinatorial-optimization","task_name":"Combinatorial Optimization"},{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"grouped-convolution","method_name":"Grouped Convolution"},{"method_slug":"local-response-normalization","method_name":"Local Response Normalization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}