{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/memory-in-memory-a-predictive-neural-network","title":"Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics","arxiv_id":"1811.07490","date":"2018-11-19","proceeding":"CVPR 2019 6","authors":["Yunbo Wang","Jianjin Zhang","Hongyu Zhu","Mingsheng Long","Jian-Min Wang","Philip S. Yu"],"abstract":"Natural spatiotemporal processes can be highly non-stationary in many ways,\ne.g. the low-level non-stationarity such as spatial correlations or temporal\ndependencies of local pixel values; and the high-level variations such as the\naccumulation, deformation or dissipation of radar echoes in precipitation\nforecasting. From Cramer's Decomposition, any non-stationary process can be\ndecomposed into deterministic, time-variant polynomials, plus a zero-mean\nstochastic term. By applying differencing operations appropriately, we may turn\ntime-variant polynomials into a constant, making the deterministic component\npredictable. However, most previous recurrent neural networks for\nspatiotemporal prediction do not use the differential signals effectively, and\ntheir relatively simple state transition functions prevent them from learning\ntoo complicated variations in spacetime. We propose the Memory In Memory (MIM)\nnetworks and corresponding recurrent blocks for this purpose. The MIM blocks\nexploit the differential signals between adjacent recurrent states to model the\nnon-stationary and approximately stationary properties in spatiotemporal\ndynamics with two cascaded, self-renewed memory modules. By stacking multiple\nMIM blocks, we could potentially handle higher-order non-stationarity. The MIM\nnetworks achieve the state-of-the-art results on four spatiotemporal prediction\ntasks across both synthetic and real-world datasets. We believe that the\ngeneral idea of this work can be potentially applied to other time-series\nforecasting tasks.","url_abs":"http://arxiv.org/abs/1811.07490v3","url_pdf":"http://arxiv.org/pdf/1811.07490v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"memory-in-memory-a-predictive-neural-network","repo_url":"https://github.com/Yunbo426/MIM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"memory-in-memory-a-predictive-neural-network","repo_url":"https://github.com/chengtan9907/simvpv2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"memory-in-memory-a-predictive-neural-network","repo_url":"https://github.com/kyafuk/3d-mim","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"memory-in-memory-a-predictive-neural-network","repo_url":"https://github.com/tianhai123/MIM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"precipitation-forecasting","task_name":"Precipitation Forecasting"},{"task_slug":"time-series","task_name":"Time Series Analysis"},{"task_slug":"time-series-forecasting","task_name":"Time Series Forecasting"},{"task_slug":"video-prediction","task_name":"Video Prediction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-prediction-on-human36m","task":"Video Prediction","dataset":"Human3.6M","model":"MIM","rank_in_archive_order":7,"of":9,"metrics":{"MAE":"1782.8","MSE":"429.9","SSIM":"0.790"},"uses_additional_data":false},{"leaderboard":"/sota/video-prediction-on-moving-mnist","task":"Video Prediction","dataset":"Moving MNIST","model":"MIM*","rank_in_archive_order":26,"of":31,"metrics":{"MAE":"101.1","MSE":"44.2","SSIM":"0.910"},"uses_additional_data":false},{"leaderboard":"/sota/video-prediction-on-moving-mnist","task":"Video Prediction","dataset":"Moving MNIST","model":"MIM","rank_in_archive_order":29,"of":31,"metrics":{"MAE":"116.5","MSE":"52.0","SSIM":"0.874"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1811.07490","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}