{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/focused-hierarchical-rnns-for-conditional","title":"Focused Hierarchical RNNs for Conditional Sequence Processing","arxiv_id":"1806.04342","date":"2018-06-12","proceeding":"ICML 2018 7","authors":["Nan Rosemary Ke","Konrad Zolna","Alessandro Sordoni","Zhouhan Lin","Adam Trischler","Yoshua Bengio","Joelle Pineau","Laurent Charlin","Chris Pal"],"abstract":"Recurrent Neural Networks (RNNs) with attention mechanisms have obtained\nstate-of-the-art results for many sequence processing tasks. Most of these\nmodels use a simple form of encoder with attention that looks over the entire\nsequence and assigns a weight to each token independently. We present a\nmechanism for focusing RNN encoders for sequence modelling tasks which allows\nthem to attend to key parts of the input as needed. We formulate this using a\nmulti-layer conditional sequence encoder that reads in one token at a time and\nmakes a discrete decision on whether the token is relevant to the context or\nquestion being asked. The discrete gating mechanism takes in the context\nembedding and the current hidden state as inputs and controls information flow\ninto the layer above. We train it using policy gradient methods. We evaluate\nthis method on several types of tasks with different attributes. First, we\nevaluate the method on synthetic tasks which allow us to evaluate the model for\nits generalization ability and probe the behavior of the gates in more\ncontrolled settings. We then evaluate this approach on large scale Question\nAnswering tasks including the challenging MS MARCO and SearchQA tasks. Our\nmodels shows consistent improvements for both tasks over prior work and our\nbaselines. It has also shown to generalize significantly better on synthetic\ntasks as compared to the baselines.","url_abs":"http://arxiv.org/abs/1806.04342v1","url_pdf":"http://arxiv.org/pdf/1806.04342v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"open-domain-question-answering","task_name":"Open-Domain Question Answering"},{"task_slug":"policy-gradient-methods","task_name":"Policy Gradient Methods"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/open-domain-question-answering-on-searchqa","task":"Open-Domain Question Answering","dataset":"SearchQA","model":"Focused Hierarchical RNN","rank_in_archive_order":12,"of":14,"metrics":{"N-gram F1":"53.4","Unigram Acc":"46.8"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}