{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hierarchical-multiscale-recurrent-neural","title":"Hierarchical Multiscale Recurrent Neural Networks","arxiv_id":"1609.01704","date":"2016-09-06","proceeding":null,"authors":["Junyoung Chung","Sungjin Ahn","Yoshua Bengio"],"abstract":"Learning both hierarchical and temporal representation has been among the\nlong-standing challenges of recurrent neural networks. Multiscale recurrent\nneural networks have been considered as a promising approach to resolve this\nissue, yet there has been a lack of empirical evidence showing that this type\nof models can actually capture the temporal dependencies by discovering the\nlatent hierarchical structure of the sequence. In this paper, we propose a\nnovel multiscale approach, called the hierarchical multiscale recurrent neural\nnetworks, which can capture the latent hierarchical structure in the sequence\nby encoding the temporal dependencies with different timescales using a novel\nupdate mechanism. We show some evidence that our proposed multiscale\narchitecture can discover underlying hierarchical structure in the sequences\nwithout using explicit boundary information. We evaluate our proposed model on\ncharacter-level language modelling and handwriting sequence modelling.","url_abs":"http://arxiv.org/abs/1609.01704v7","url_pdf":"http://arxiv.org/pdf/1609.01704v7.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hierarchical-multiscale-recurrent-neural","repo_url":"https://github.com/bolducp/hierarchical-rnn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"hierarchical-multiscale-recurrent-neural","repo_url":"https://github.com/kaiu85/hm-rnn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"hierarchical-multiscale-recurrent-neural","repo_url":"https://github.com/nikolasthuesen/HMLSTM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-text8","task":"Language Modelling","dataset":"Text8","model":"LayerNorm HM-LSTM","rank_in_archive_order":19,"of":24,"metrics":{"Bit per Character (BPC)":"1.29","Number of params":"35M"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-enwiki8","task":"Language Modelling","dataset":"enwik8","model":"LN HM-LSTM","rank_in_archive_order":38,"of":42,"metrics":{"Bit per Character (BPC)":"1.32","Number of params":"35M"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1609.01704","atlas_url":"https://app.syntology.ai/?focus=1609.01704","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}