{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/knowledge-matters-importance-of-prior","title":"Knowledge Matters: Importance of Prior Information for Optimization","arxiv_id":"1301.4083","date":"2013-01-17","proceeding":null,"authors":["Çağlar Gülçehre","Yoshua Bengio"],"abstract":"We explore the effect of introducing prior information into the intermediate\nlevel of neural networks for a learning task on which all the state-of-the-art\nmachine learning algorithms tested failed to learn. We motivate our work from\nthe hypothesis that humans learn such intermediate concepts from other\nindividuals via a form of supervision or guidance using a curriculum. The\nexperiments we have conducted provide positive evidence in favor of this\nhypothesis. In our experiments, a two-tiered MLP architecture is trained on a\ndataset with 64x64 binary inputs images, each image with three sprites. The\nfinal task is to decide whether all the sprites are the same or one of them is\ndifferent. Sprites are pentomino tetris shapes and they are placed in an image\nwith different locations using scaling and rotation transformations. The first\npart of the two-tiered MLP is pre-trained with intermediate-level targets being\nthe presence of sprites at each location, while the second part takes the\noutput of the first part as input and predicts the final task's target binary\nevent. The two-tiered MLP architecture, with a few tens of thousand examples,\nwas able to learn the task perfectly, whereas all other algorithms (include\nunsupervised pre-training, but also traditional algorithms like SVMs, decision\ntrees and boosting) all perform no better than chance. We hypothesize that the\noptimization difficulty involved when the intermediate pre-training is not\nperformed is due to the {\\em composition} of two highly non-linear tasks. Our\nfindings are also consistent with hypotheses on cultural learning inspired by\nthe observations of optimization problems with deep learning, presumably\nbecause of effective local minima.","url_abs":"http://arxiv.org/abs/1301.4083v6","url_pdf":"http://arxiv.org/pdf/1301.4083v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"knowledge-matters-importance-of-prior","repo_url":"https://github.com/caglar/structured_mlp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"unsupervised-pre-training","task_name":"Unsupervised Pre-training"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1301.4083","atlas_url":"https://app.syntology.ai/?focus=1301.4083","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}