{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/beyond-narrative-description-generating","title":"Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training","arxiv_id":"1804.08473","date":"2018-04-23","proceeding":null,"authors":["Bei Liu","Jianlong Fu","Makoto P. Kato","Masatoshi Yoshikawa"],"abstract":"Automatic generation of natural language from images has attracted extensive\nattention. In this paper, we take one step further to investigate generation of\npoetic language (with multiple lines) to an image for automatic poetry\ncreation. This task involves multiple challenges, including discovering poetic\nclues from the image (e.g., hope from green), and generating poems to satisfy\nboth relevance to the image and poeticness in language level. To solve the\nabove challenges, we formulate the task of poem generation into two correlated\nsub-tasks by multi-adversarial training via policy gradient, through which the\ncross-modal relevance and poetic language style can be ensured. To extract\npoetic clues from images, we propose to learn a deep coupled visual-poetic\nembedding, in which the poetic representation from objects, sentiments and\nscenes in an image can be jointly learned. Two discriminative networks are\nfurther introduced to guide the poem generation, including a multi-modal\ndiscriminator and a poem-style discriminator. To facilitate the research, we\nhave released two poem datasets by human annotators with two distinct\nproperties: 1) the first human annotated image-to-poem pair dataset (with 8,292\npairs in total), and 2) to-date the largest public English poem corpus dataset\n(with 92,265 different poems in total). Extensive experiments are conducted\nwith 8K images, among which 1.5K image are randomly picked for evaluation. Both\nobjective and subjective evaluations show the superior performances against the\nstate-of-the-art methods for poem generation from images. Turing test carried\nout with over 500 human subjects, among which 30 evaluators are poetry experts,\ndemonstrates the effectiveness of our approach.","url_abs":"http://arxiv.org/abs/1804.08473v4","url_pdf":"http://arxiv.org/pdf/1804.08473v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"beyond-narrative-description-generating","repo_url":"https://github.com/BruceZoom/EECourse-Poem-Project","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"beyond-narrative-description-generating","repo_url":"https://github.com/alex-calderwood/all_is_all_poetry","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"beyond-narrative-description-generating","repo_url":"https://github.com/researchmm/img2poem","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"mxnet","reach":null}],"tasks":[{"task_slug":null,"task_name":"8k"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1804.08473","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}