{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/greedy-layerwise-learning-can-scale-to","title":"Greedy Layerwise Learning Can Scale to ImageNet","arxiv_id":"1812.11446","date":"2018-12-29","proceeding":null,"authors":["Eugene Belilovsky","Michael Eickenberg","Edouard Oyallon"],"abstract":"Shallow supervised 1-hidden layer neural networks have a number of favorable\nproperties that make them easier to interpret, analyze, and optimize than their\ndeep counterparts, but lack their representational power. Here we use 1-hidden\nlayer learning problems to sequentially build deep networks layer by layer,\nwhich can inherit properties from shallow networks. Contrary to previous\napproaches using shallow networks, we focus on problems where deep learning is\nreported as critical for success. We thus study CNNs on image classification\ntasks using the large-scale ImageNet dataset and the CIFAR-10 dataset. Using a\nsimple set of ideas for architecture and training we find that solving\nsequential 1-hidden-layer auxiliary problems lead to a CNN that exceeds AlexNet\nperformance on ImageNet. Extending this training methodology to construct\nindividual layers by solving 2-and-3-hidden layer auxiliary problems, we obtain\nan 11-layer network that exceeds several members of the VGG model family on\nImageNet, and can train a VGG-11 model to the same accuracy as end-to-end\nlearning. To our knowledge, this is the first competitive alternative to\nend-to-end training of CNNs that can scale to ImageNet. We illustrate several\ninteresting properties of these models theoretically and conduct a range of\nexperiments to study the properties this training induces on the intermediate\nlayers.","url_abs":"http://arxiv.org/abs/1812.11446v3","url_pdf":"http://arxiv.org/pdf/1812.11446v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"greedy-layerwise-learning-can-scale-to","repo_url":"https://github.com/eugenium/layerCNN","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1812.11446","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}