{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/discrete-event-continuous-time-rnns","title":"Discrete Event, Continuous Time RNNs","arxiv_id":"1710.04110","date":"2017-10-11","proceeding":null,"authors":["Michael C. Mozer","Denis Kazakov","Robert V. Lindsey"],"abstract":"We investigate recurrent neural network architectures for event-sequence\nprocessing. Event sequences, characterized by discrete observations stamped\nwith continuous-valued times of occurrence, are challenging due to the\npotentially wide dynamic range of relevant time scales as well as interactions\nbetween time scales. We describe four forms of inductive bias that should\nbenefit architectures for event sequences: temporal locality, position and\nscale homogeneity, and scale interdependence. We extend the popular gated\nrecurrent unit (GRU) architecture to incorporate these biases via intrinsic\ntemporal dynamics, obtaining a continuous-time GRU. The CT-GRU arises by\ninterpreting the gates of a GRU as selecting a time scale of memory, and the\nCT-GRU generalizes the GRU by incorporating multiple time scales of memory and\nperforming context-dependent selection of time scales for information storage\nand retrieval. Event time-stamps drive decay dynamics of the CT-GRU, whereas\nthey serve as generic additional inputs to the GRU. Despite the very different\nmanner in which the two models consider time, their performance on eleven data\nsets we examined is essentially identical. Our surprising results point both to\nthe robustness of GRU and LSTM architectures for handling continuous time, and\nto the potency of incorporating continuous dynamics into neural architectures.","url_abs":"http://arxiv.org/abs/1710.04110v1","url_pdf":"http://arxiv.org/pdf/1710.04110v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"discrete-event-continuous-time-rnns","repo_url":"https://github.com/raminmh/liquid_time_constant_networks","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"gru","method_name":"GRU"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1710.04110","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}