{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-decision-boundary-of-deep-neural","title":"On the Decision Boundary of Deep Neural Networks","arxiv_id":"1808.05385","date":"2018-08-16","proceeding":null,"authors":["Yu Li","Lizhong Ding","Xin Gao"],"abstract":"While deep learning models and techniques have achieved great empirical\nsuccess, our understanding of the source of success in many aspects remains\nvery limited. In an attempt to bridge the gap, we investigate the decision\nboundary of a production deep learning architecture with weak assumptions on\nboth the training data and the model. We demonstrate, both theoretically and\nempirically, that the last weight layer of a neural network converges to a\nlinear SVM trained on the output of the last hidden layer, for both the binary\ncase and the multi-class case with the commonly used cross-entropy loss.\nFurthermore, we show empirically that training a neural network as a whole,\ninstead of only fine-tuning the last weight layer, may result in better bias\nconstant for the last weight layer, which is important for generalization. In\naddition to facilitating the understanding of deep learning, our result can be\nhelpful for solving a broad range of practical problems of deep learning, such\nas catastrophic forgetting and adversarial attacking. The experiment codes are\navailable at https://github.com/lykaust15/NN_decision_boundary","url_abs":"http://arxiv.org/abs/1808.05385v3","url_pdf":"http://arxiv.org/pdf/1808.05385v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-decision-boundary-of-deep-neural","repo_url":"https://github.com/lykaust15/NN_decision_boundary","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"}],"methods":[{"method_slug":"svm","method_name":"SVM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1808.05385","atlas_url":"https://app.syntology.ai/?focus=1808.05385","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}