{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/decoupled-novel-object-captioner","title":"Decoupled Novel Object Captioner","arxiv_id":"1804.03803","date":"2018-04-11","proceeding":null,"authors":["Yu Wu","Linchao Zhu","Lu Jiang","Yi Yang"],"abstract":"Image captioning is a challenging task where the machine automatically\ndescribes an image by sentences or phrases. It often requires a large number of\npaired image-sentence annotations for training. However, a pre-trained\ncaptioning model can hardly be applied to a new domain in which some novel\nobject categories exist, i.e., the objects and their description words are\nunseen during model training. To correctly caption the novel object, it\nrequires professional human workers to annotate the images by sentences with\nthe novel words. It is labor expensive and thus limits its usage in real-world\napplications.\n  In this paper, we introduce the zero-shot novel object captioning task where\nthe machine generates descriptions without extra sentences about the novel\nobject. To tackle the challenging problem, we propose a Decoupled Novel Object\nCaptioner (DNOC) framework that can fully decouple the language sequence model\nfrom the object descriptions. DNOC has two components. 1) A Sequence Model with\nthe Placeholder (SM-P) generates a sentence containing placeholders. The\nplaceholder represents an unseen novel object. Thus, the sequence model can be\ndecoupled from the novel object descriptions. 2) A key-value object memory\nbuilt upon the freely available detection model, contains the visual\ninformation and the corresponding word for each object. The SM-P will generate\na query to retrieve the words from the object memory. The placeholder will then\nbe filled with the correct word, resulting in a caption with novel object\ndescriptions. The experimental results on the held-out MSCOCO dataset\ndemonstrate the ability of DNOC in describing novel concepts in the zero-shot\nnovel object captioning task.","url_abs":"http://arxiv.org/abs/1804.03803v2","url_pdf":"http://arxiv.org/pdf/1804.03803v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"decoupled-novel-object-captioner","repo_url":"https://github.com/Pranav21091996/Semantic_Fidelity-and-Egoshots","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"novel-concepts","task_name":"Novel Concepts"},{"task_slug":"object","task_name":"Object"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1804.03803","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}