{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lifelong-learning-for-image-captioning-by","title":"Learning to Caption Images through a Lifetime by Asking Questions","arxiv_id":"1812.00235","date":"2018-12-01","proceeding":null,"authors":["Kevin Shen","Amlan Kar","Sanja Fidler"],"abstract":"In order to bring artificial agents into our lives, we will need to go beyond\nsupervised learning on closed datasets to having the ability to continuously\nexpand knowledge. Inspired by a student learning in a classroom, we present an\nagent that can continuously learn by posing natural language questions to\nhumans. Our agent is composed of three interacting modules, one that performs\ncaptioning, another that generates questions and a decision maker that learns\nwhen to ask questions by implicitly reasoning about the uncertainty of the\nagent and expertise of the teacher. As compared to current active learning\nmethods which query images for full captions, our agent is able to ask pointed\nquestions to improve the generated captions. The agent trains on the improved\ncaptions, expanding its knowledge. We show that our approach achieves better\nperformance using less human supervision than the baselines on the challenging\nMSCOCO dataset.","url_abs":"http://arxiv.org/abs/1812.00235v3","url_pdf":"http://arxiv.org/pdf/1812.00235v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lifelong-learning-for-image-captioning-by","repo_url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"active-learning","task_name":"Active Learning"},{"task_slug":"image-captioning","task_name":"Image Captioning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1812.00235","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1812.00235"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":13},"by_repo_kind":{"official":{"samples":13,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"758ce8c8ac1b4c71","entry":"cook_refs","repo":"shenkev/Caption-Lifetime-by-Asking-Questions","repo_kind":"official","path":"Utils/preprocess/preprocess_cider.py","file_url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions/blob/HEAD/Utils/preprocess/preprocess_cider.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"758ce8c8ac1b4c71"}},{"code_sha256_prefix":"1452c9f3906105d1","entry":"create_crefs","repo":"shenkev/Caption-Lifetime-by-Asking-Questions","repo_kind":"official","path":"Utils/preprocess/preprocess_cider.py","file_url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions/blob/HEAD/Utils/preprocess/preprocess_cider.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1452c9f3906105d1"}},{"code_sha256_prefix":"09204bdcd67a5309","entry":"init_hidden","repo":"shenkev/Caption-Lifetime-by-Asking-Questions","repo_kind":"official","path":"Models/attention_captioner.py","file_url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions/blob/HEAD/Models/attention_captioner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09204bdcd67a5309"}},{"code_sha256_prefix":"9e7cb7f7bef2c76f","entry":"linear_reward_weighting","repo":"shenkev/Caption-Lifetime-by-Asking-Questions","repo_kind":"official","path":"Losses/metrics.py","file_url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions/blob/HEAD/Losses/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e7cb7f7bef2c76f"}},{"code_sha256_prefix":"fd3efd292071801e","entry":"masked_CE","repo":"shenkev/Caption-Lifetime-by-Asking-Questions","repo_kind":"official","path":"Losses/loss.py","file_url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions/blob/HEAD/Losses/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd3efd292071801e"}},{"code_sha256_prefix":"ca62ed9709db383e","entry":"masked_PG","repo":"shenkev/Caption-Lifetime-by-Asking-Questions","repo_kind":"official","path":"Losses/loss.py","file_url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions/blob/HEAD/Losses/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca62ed9709db383e"}},{"code_sha256_prefix":"3ae4fdcfc43afe37","entry":"precook","repo":"shenkev/Caption-Lifetime-by-Asking-Questions","repo_kind":"official","path":"Utils/preprocess/preprocess_cider.py","file_url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions/blob/HEAD/Utils/preprocess/preprocess_cider.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ae4fdcfc43afe37"}},{"code_sha256_prefix":"0a720b01afd3f4ca","entry":"remove_datasamples","repo":"shenkev/Caption-Lifetime-by-Asking-Questions","repo_kind":"official","path":"Utils/preprocess/preprocess_llsplits.py","file_url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions/blob/HEAD/Utils/preprocess/preprocess_llsplits.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a720b01afd3f4ca"}},{"code_sha256_prefix":"a811ae08639f7bae","entry":"seq_max_and_mask","repo":"shenkev/Caption-Lifetime-by-Asking-Questions","repo_kind":"official","path":"Losses/loss.py","file_url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions/blob/HEAD/Losses/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a811ae08639f7bae"}},{"code_sha256_prefix":"aa73043b661acaa4","entry":"symbol_to_string","repo":"shenkev/Caption-Lifetime-by-Asking-Questions","repo_kind":"official","path":"Losses/metrics.py","file_url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions/blob/HEAD/Losses/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa73043b661acaa4"}},{"code_sha256_prefix":"bc93b2a653456596","entry":"time_remaining","repo":"shenkev/Caption-Lifetime-by-Asking-Questions","repo_kind":"official","path":"Utils/util.py","file_url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions/blob/HEAD/Utils/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc93b2a653456596"}},{"code_sha256_prefix":"91f76724f936bf4c","entry":"update_lr","repo":"shenkev/Caption-Lifetime-by-Asking-Questions","repo_kind":"official","path":"Utils/util.py","file_url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions/blob/HEAD/Utils/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91f76724f936bf4c"}},{"code_sha256_prefix":"7b76618e10e9cb85","entry":"update_scheduled_sampling","repo":"shenkev/Caption-Lifetime-by-Asking-Questions","repo_kind":"official","path":"Utils/util.py","file_url":"https://github.com/shenkev/Caption-Lifetime-by-Asking-Questions/blob/HEAD/Utils/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b76618e10e9cb85"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}