{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-intuitive-physics-with-multimodal","title":"Learning Intuitive Physics with Multimodal Generative Models","arxiv_id":"2101.04454","date":"2021-01-12","proceeding":null,"authors":["Sahand Rezaei-Shoshtari","Francois Robert Hogan","Michael Jenkin","David Meger","Gregory Dudek"],"abstract":"Predicting the future interaction of objects when they come into contact with their environment is key for autonomous agents to take intelligent and anticipatory actions. This paper presents a perception framework that fuses visual and tactile feedback to make predictions about the expected motion of objects in dynamic scenes. Visual information captures object properties such as 3D shape and location, while tactile information provides critical cues about interaction forces and resulting object motion when it makes contact with the environment. Utilizing a novel See-Through-your-Skin (STS) sensor that provides high resolution multimodal sensing of contact surfaces, our system captures both the visual appearance and the tactile properties of objects. We interpret the dual stream signals from the sensor using a Multimodal Variational Autoencoder (MVAE), allowing us to capture both modalities of contacting objects and to develop a mapping from visual to tactile interaction and vice-versa. Additionally, the perceptual system can be used to infer the outcome of future physical interactions, which we validate through simulated and real-world experiments in which the resting state of an object is predicted from given initial conditions.","url_abs":"https://arxiv.org/abs/2101.04454v2","url_pdf":"https://arxiv.org/pdf/2101.04454v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-intuitive-physics-with-multimodal","repo_url":"https://github.com/SAIC-MONTREAL/multimodal-dynamics","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"object","task_name":"Object"},{"task_slug":"sts","task_name":"STS"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2101.04454","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2101.04454"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/SAIC-MONTREAL/multimodal-dynamics","reach":null}],"summary":{"ran":3,"ran_draft_wrong":1,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"6c4896b8886e1580","entry":"Encoder","repo":"SAIC-MONTREAL/multimodal-dynamics","repo_kind":"official","path":"mmdyn/pytorch/models/vae.py","file_url":"https://github.com/SAIC-MONTREAL/multimodal-dynamics/blob/HEAD/mmdyn/pytorch/models/vae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6c4896b8886e1580"}},{"code_sha256_prefix":"a19dc1753fe80221","entry":"ProductOfExperts","repo":"SAIC-MONTREAL/multimodal-dynamics","repo_kind":"official","path":"mmdyn/pytorch/models/vae.py","file_url":"https://github.com/SAIC-MONTREAL/multimodal-dynamics/blob/HEAD/mmdyn/pytorch/models/vae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a19dc1753fe80221"}},{"code_sha256_prefix":"14e06e2db08f4665","entry":"Swish","repo":"SAIC-MONTREAL/multimodal-dynamics","repo_kind":"official","path":"mmdyn/pytorch/models/vae.py","file_url":"https://github.com/SAIC-MONTREAL/multimodal-dynamics/blob/HEAD/mmdyn/pytorch/models/vae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"14e06e2db08f4665"}},{"code_sha256_prefix":"a5ea53bdbdb19025","entry":"idx2onehot","repo":"SAIC-MONTREAL/multimodal-dynamics","repo_kind":"official","path":"mmdyn/pytorch/models/vae.py","file_url":"https://github.com/SAIC-MONTREAL/multimodal-dynamics/blob/HEAD/mmdyn/pytorch/models/vae.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a5ea53bdbdb19025"}},{"code_sha256_prefix":"60b091a9b04427a1","entry":"prior_expert","repo":"SAIC-MONTREAL/multimodal-dynamics","repo_kind":"official","path":"mmdyn/pytorch/models/vae.py","file_url":"https://github.com/SAIC-MONTREAL/multimodal-dynamics/blob/HEAD/mmdyn/pytorch/models/vae.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"60b091a9b04427a1"}},{"code_sha256_prefix":"f92a80324a155247","entry":"Autoencoder","repo":"SAIC-MONTREAL/multimodal-dynamics","repo_kind":"official","path":"mmdyn/pytorch/models/vae.py","file_url":"https://github.com/SAIC-MONTREAL/multimodal-dynamics/blob/HEAD/mmdyn/pytorch/models/vae.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f92a80324a155247"}},{"code_sha256_prefix":"0ce12c98601d5270","entry":"Decoder","repo":"SAIC-MONTREAL/multimodal-dynamics","repo_kind":"official","path":"mmdyn/pytorch/models/vae.py","file_url":"https://github.com/SAIC-MONTREAL/multimodal-dynamics/blob/HEAD/mmdyn/pytorch/models/vae.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0ce12c98601d5270"}},{"code_sha256_prefix":"5e558ee407bbab04","entry":"MVAE","repo":"SAIC-MONTREAL/multimodal-dynamics","repo_kind":"official","path":"mmdyn/pytorch/models/vae.py","file_url":"https://github.com/SAIC-MONTREAL/multimodal-dynamics/blob/HEAD/mmdyn/pytorch/models/vae.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5e558ee407bbab04"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}