{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-supervised-visual-planning-with-temporal","title":"Self-Supervised Visual Planning with Temporal Skip Connections","arxiv_id":"1710.05268","date":"2017-10-15","proceeding":null,"authors":["Frederik Ebert","Chelsea Finn","Alex X. Lee","Sergey Levine"],"abstract":"In order to autonomously learn wide repertoires of complex skills, robots\nmust be able to learn from their own autonomously collected data, without human\nsupervision. One learning signal that is always available for autonomously\ncollected data is prediction: if a robot can learn to predict the future, it\ncan use this predictive model to take actions to produce desired outcomes, such\nas moving an object to a particular location. However, in complex open-world\nscenarios, designing a representation for prediction is difficult. In this\nwork, we instead aim to enable self-supervised robotic learning through direct\nvideo prediction: instead of attempting to design a good representation, we\ndirectly predict what the robot will see next, and then use this model to\nachieve desired goals. A key challenge in video prediction for robotic\nmanipulation is handling complex spatial arrangements such as occlusions. To\nthat end, we introduce a video prediction model that can keep track of objects\nthrough occlusion by incorporating temporal skip-connections. Together with a\nnovel planning criterion and action space formulation, we demonstrate that this\nmodel substantially outperforms prior work on video prediction-based control.\nOur results show manipulation of objects not seen during training, handling\nmultiple objects, and pushing objects around obstructions. These results\nrepresent a significant advance in the range and complexity of skills that can\nbe performed entirely with self-supervised robotic learning.","url_abs":"http://arxiv.org/abs/1710.05268v1","url_pdf":"http://arxiv.org/pdf/1710.05268v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-supervised-visual-planning-with-temporal","repo_url":"https://github.com/CompVis/image2video-synthesis-using-cINNs","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"self-supervised-visual-planning-with-temporal","repo_url":"https://github.com/joelouismarino/amortized-variational-filtering","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"self-supervised-visual-planning-with-temporal","repo_url":"https://github.com/kamran0153/impact-of-data-freshness-in-learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"video-prediction","task_name":"Video Prediction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1710.05268","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}