{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-is-the-role-of-recurrent-neural-networks","title":"What is the Role of Recurrent Neural Networks (RNNs) in an Image Caption Generator?","arxiv_id":"1708.02043","date":"2017-08-07","proceeding":"WS 2017 9","authors":["Marc Tanti","Albert Gatt","Kenneth P. Camilleri"],"abstract":"In neural image captioning systems, a recurrent neural network (RNN) is\ntypically viewed as the primary `generation' component. This view suggests that\nthe image features should be `injected' into the RNN. This is in fact the\ndominant view in the literature. Alternatively, the RNN can instead be viewed\nas only encoding the previously generated words. This view suggests that the\nRNN should only be used to encode linguistic features and that only the final\nrepresentation should be `merged' with the image features at a later stage.\nThis paper compares these two architectures. We find that, in general, late\nmerging outperforms injection, suggesting that RNNs are better viewed as\nencoders, rather than generators.","url_abs":"http://arxiv.org/abs/1708.02043v2","url_pdf":"http://arxiv.org/pdf/1708.02043v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-is-the-role-of-recurrent-neural-networks","repo_url":"https://github.com/mtanti/rnn-role","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"what-is-the-role-of-recurrent-neural-networks","repo_url":"https://github.com/jishubasak/Punny-Caption--Exploring-Image-Captioning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"what-is-the-role-of-recurrent-neural-networks","repo_url":"https://github.com/mrityunjayojha10/Image-Caption-Generation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"what-is-the-role-of-recurrent-neural-networks","repo_url":"https://github.com/sinazarriess/zero_shot_reg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}