{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improved-image-captioning-via-policy-gradient","title":"Improved Image Captioning via Policy Gradient optimization of SPIDEr","arxiv_id":"1612.00370","date":"2016-12-01","proceeding":"ICCV 2017 10","authors":["Si-Qi Liu","Zhenhai Zhu","Ning Ye","Sergio Guadarrama","Kevin Murphy"],"abstract":"Current image captioning methods are usually trained via (penalized) maximum\nlikelihood estimation. However, the log-likelihood score of a caption does not\ncorrelate well with human assessments of quality. Standard syntactic evaluation\nmetrics, such as BLEU, METEOR and ROUGE, are also not well correlated. The\nnewer SPICE and CIDEr metrics are better correlated, but have traditionally\nbeen hard to optimize for. In this paper, we show how to use a policy gradient\n(PG) method to directly optimize a linear combination of SPICE and CIDEr (a\ncombination we call SPIDEr): the SPICE score ensures our captions are\nsemantically faithful to the image, while CIDEr score ensures our captions are\nsyntactically fluent. The PG method we propose improves on the prior MIXER\napproach, by using Monte Carlo rollouts instead of mixing MLE training with PG.\nWe show empirically that our algorithm leads to easier optimization and\nimproved results compared to MIXER. Finally, we show that using our PG method\nwe can optimize any of the metrics, including the proposed SPIDEr metric which\nresults in image captions that are strongly preferred by human raters compared\nto captions generated by the same model but trained to optimize MLE or the COCO\nmetrics.","url_abs":"http://arxiv.org/abs/1612.00370v4","url_pdf":"http://arxiv.org/pdf/1612.00370v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improved-image-captioning-via-policy-gradient","repo_url":"https://github.com/peteanderson80/SPICE","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"AGPL-3.0"}},{"paper_slug":"improved-image-captioning-via-policy-gradient","repo_url":"https://github.com/audio-captioning/caption-evaluation-tools","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1612.00370","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}