{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/knowing-when-to-look-adaptive-attention-via-a","title":"Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning","arxiv_id":"1612.01887","date":"2016-12-06","proceeding":"CVPR 2017 7","authors":["Jiasen Lu","Caiming Xiong","Devi Parikh","Richard Socher"],"abstract":"Attention-based neural encoder-decoder frameworks have been widely adopted\nfor image captioning. Most methods force visual attention to be active for\nevery generated word. However, the decoder likely requires little to no visual\ninformation from the image to predict non-visual words such as \"the\" and \"of\".\nOther words that may seem visual can often be predicted reliably just from the\nlanguage model e.g., \"sign\" after \"behind a red stop\" or \"phone\" following\n\"talking on a cell\". In this paper, we propose a novel adaptive attention model\nwith a visual sentinel. At each time step, our model decides whether to attend\nto the image (and if so, to which regions) or to the visual sentinel. The model\ndecides whether to attend to the image and where, in order to extract\nmeaningful information for sequential word generation. We test our method on\nthe COCO image captioning 2015 challenge dataset and Flickr30K. Our approach\nsets the new state-of-the-art by a significant margin.","url_abs":"http://arxiv.org/abs/1612.01887v2","url_pdf":"http://arxiv.org/pdf/1612.01887v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/jiasenlu/AdaptiveAttention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"torch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/BJennWare/ImcaptionNet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/Chloejay/image_caption_app","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/abcSup/captionamerica","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/fawazsammani/knowing-when-to-look-adaptive-attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/miroblog/AdaptiveAttention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/yufengm/Adaptive","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/MindSpore-scientific-2/code-5/tree/main/Knowing-When-to-Look-Adaptive-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/MindSpore-scientific-2/code-9/tree/main/Knowing-When-to-Look-Adaptive-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/MindSpore-scientific/code-1/tree/main/Knowing-When-to-Look-Adaptive-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/MindSpore-scientific/code-10/tree/main/Knowing-When-to-Look-Adaptive-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/MindSpore-scientific/code-11/tree/main/Knowing-When-to-Look-Adaptive-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/MindSpore-scientific/code-13/tree/main/Knowing-When-to-Look-Adaptive-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/MindSpore-scientific/code-4/tree/main/Knowing-When-to-Look-Adaptive-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/MindSpore-scientific/code-8/tree/main/Knowing-When-to-Look-Adaptive-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/mindspore-ai/contrib/tree/master/intern/Knowing-When-to-Look-Adaptive-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"knowing-when-to-look-adaptive-attention-via-a","repo_url":"https://github.com/pwc-1/Paper-9/tree/main/4/Knowing-When-to-Look-Adaptive-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1612.01887","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1612.01887"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/miroblog/AdaptiveAttention","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Chloejay/image_caption_app","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abcSup/captionamerica","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fawazsammani/knowing-when-to-look-adaptive-attention","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-4/tree/main/Knowing-When-to-Look-Adaptive-Attention","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pwc-1/Paper-9/tree/main/4/Knowing-When-to-Look-Adaptive-Attention","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-11/tree/main/Knowing-When-to-Look-Adaptive-Attention","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-1/tree/main/Knowing-When-to-Look-Adaptive-Attention","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific-2/code-9/tree/main/Knowing-When-to-Look-Adaptive-Attention","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/BJennWare/ImcaptionNet","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yufengm/Adaptive","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-13/tree/main/Knowing-When-to-Look-Adaptive-Attention","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-8/tree/main/Knowing-When-to-Look-Adaptive-Attention","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific-2/code-5/tree/main/Knowing-When-to-Look-Adaptive-Attention","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-10/tree/main/Knowing-When-to-Look-Adaptive-Attention","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mindspore-ai/contrib/tree/master/intern/Knowing-When-to-Look-Adaptive-Attention","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jiasenlu/AdaptiveAttention","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_honours":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"d0ee764129b96d9d","entry":"get_start_states","repo":"miroblog/AdaptiveAttention","repo_kind":"listed","path":"adaptive_attention.py","file_url":"https://github.com/miroblog/AdaptiveAttention/blob/HEAD/adaptive_attention.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d0ee764129b96d9d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}