{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unsupervised-learning-of-visual-structure","title":"Unsupervised Learning of Visual Structure using Predictive Generative Networks","arxiv_id":"1511.06380","date":"2015-11-19","proceeding":null,"authors":["William Lotter","Gabriel Kreiman","David Cox"],"abstract":"The ability to predict future states of the environment is a central pillar\nof intelligence. At its core, effective prediction requires an internal model\nof the world and an understanding of the rules by which the world changes.\nHere, we explore the internal models developed by deep neural networks trained\nusing a loss based on predicting future frames in synthetic video sequences,\nusing a CNN-LSTM-deCNN framework. We first show that this architecture can\nachieve excellent performance in visual sequence prediction tasks, including\nstate-of-the-art performance in a standard 'bouncing balls' dataset (Sutskever\net al., 2009). Using a weighted mean-squared error and adversarial loss\n(Goodfellow et al., 2014), the same architecture successfully extrapolates\nout-of-the-plane rotations of computer-generated faces. Furthermore, despite\nbeing trained end-to-end to predict only pixel-level information, our\nPredictive Generative Networks learn a representation of the latent structure\nof the underlying three-dimensional objects themselves. Importantly, we find\nthat this representation is naturally tolerant to object transformations, and\ngeneralizes well to new tasks, such as classification of static images. Similar\nmodels trained solely with a reconstruction loss fail to generalize as\neffectively. We argue that prediction can serve as a powerful unsupervised loss\nfor learning rich internal representations of high-level object features.","url_abs":"http://arxiv.org/abs/1511.06380v2","url_pdf":"http://arxiv.org/pdf/1511.06380v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unsupervised-learning-of-visual-structure","repo_url":"https://github.com/kunimasa-kawasaki/chainer-prednet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"unsupervised-learning-of-visual-structure","repo_url":"https://github.com/kunimasa-kawasaki/keras-prednet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1511.06380","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}