{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diving-deeper-into-mentee-networks","title":"Diving deeper into mentee networks","arxiv_id":"1604.08220","date":"2016-04-27","proceeding":null,"authors":["Ragav Venkatesan","Baoxin Li"],"abstract":"Modern computer vision is all about the possession of powerful image\nrepresentations. Deeper and deeper convolutional neural networks have been\nbuilt using larger and larger datasets and are made publicly available. A large\nswath of computer vision scientists use these pre-trained networks with varying\ndegrees of successes in various tasks. Even though there is tremendous success\nin copying these networks, the representational space is not learnt from the\ntarget dataset in a traditional manner. One of the reasons for opting to use a\npre-trained network over a network learnt from scratch is that small datasets\nprovide less supervision and require meticulous regularization, smaller and\ncareful tweaking of learning rates to even achieve stable learning without\nweight explosion. It is often the case that large deep networks are not\nportable, which necessitates the ability to learn mid-sized networks from\nscratch.\n  In this article, we dive deeper into training these mid-sized networks on\nsmall datasets from scratch by drawing additional supervision from a large\npre-trained network. Such learning also provides better generalization\naccuracies than networks trained with common regularization techniques such as\nl2, l1 and dropouts. We show that features learnt thus, are more general than\nthose learnt independently. We studied various characteristics of such networks\nand found some interesting behaviors.","url_abs":"http://arxiv.org/abs/1604.08220v1","url_pdf":"http://arxiv.org/pdf/1604.08220v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diving-deeper-into-mentee-networks","repo_url":"https://github.com/ragavvenkatesan/regularizer-network","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}