{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unsupervised-learning-for-physical","title":"Unsupervised Learning for Physical Interaction through Video Prediction","arxiv_id":"1605.07157","date":"2016-05-23","proceeding":"NeurIPS 2016 12","authors":["Chelsea Finn","Ian Goodfellow","Sergey Levine"],"abstract":"A core challenge for an agent learning to interact with the world is to\npredict how its actions affect objects in its environment. Many existing\nmethods for learning the dynamics of physical interactions require labeled\nobject information. However, to scale real-world interaction learning to a\nvariety of scenes and objects, acquiring labeled data becomes increasingly\nimpractical. To learn about physical object motion without labels, we develop\nan action-conditioned video prediction model that explicitly models pixel\nmotion, by predicting a distribution over pixel motion from previous frames.\nBecause our model explicitly predicts motion, it is partially invariant to\nobject appearance, enabling it to generalize to previously unseen objects. To\nexplore video prediction for real-world interactive agents, we also introduce a\ndataset of 59,000 robot interactions involving pushing motions, including a\ntest set with novel objects. In this dataset, accurate prediction of videos\nconditioned on the robot's future actions amounts to learning a \"visual\nimagination\" of different futures based on different courses of action. Our\nexperiments show that our proposed method produces more accurate video\npredictions both quantitatively and qualitatively, when compared to prior\nmethods.","url_abs":"http://arxiv.org/abs/1605.07157v4","url_pdf":"http://arxiv.org/pdf/1605.07157v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unsupervised-learning-for-physical","repo_url":"https://github.com/tensorflow/models/tree/master/research/video_prediction","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"unsupervised-learning-for-physical","repo_url":"https://github.com/Xiaohui9607/physical_interaction_video_prediction_pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"object","task_name":"Object"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"video-generation","task_name":"Video Generation"},{"task_slug":"video-prediction","task_name":"Video Prediction"}],"methods":[],"datasets_introduced":[{"slug":"robotic-pushing","name":"Robotic Pushing","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-generation-on-bair-robot-pushing","task":"Video Generation","dataset":"BAIR Robot Pushing","model":"CDNA (from FVD)","rank_in_archive_order":26,"of":31,"metrics":{"Cond":"2","FVD score":"296.5","Pred":"14","Train":"14"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1605.07157","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1605.07157"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Xiaohui9607/physical_interaction_video_prediction_pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tensorflow/models/tree/master/research/video_prediction","reach":null}],"summary":{"ran_draft_wrong":1,"ran_honours":1},"by_repo_kind":{"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"14b3895b30d43a71","entry":"make_dataset","repo":"Xiaohui9607/physical_interaction_video_prediction_pytorch","repo_kind":"listed","path":"data.py","file_url":"https://github.com/Xiaohui9607/physical_interaction_video_prediction_pytorch/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14b3895b30d43a71"}},{"code_sha256_prefix":"3425012359c71b0b","entry":"npy_loader","repo":"Xiaohui9607/physical_interaction_video_prediction_pytorch","repo_kind":"listed","path":"data.py","file_url":"https://github.com/Xiaohui9607/physical_interaction_video_prediction_pytorch/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3425012359c71b0b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}