{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diverse-and-relevant-visual-storytelling-with","title":"Diverse and Relevant Visual Storytelling with Scene Graph Embeddings","arxiv_id":null,"date":"2020-11-01","proceeding":"CONLL 2020","authors":["Xudong Hong","Rakshith Shetty","Asad Sayeed","Khushboo Mehra","Vera Demberg","Bernt Schiele"],"abstract":"A problem in automatically generated stories for image sequences is that they use overly generic vocabulary and phrase structure and fail to match the distributional characteristics of human-generated text. We address this problem by introducing explicit representations for objects and their relations by extracting scene graphs from the images. Utilizing an embedding of this scene graph enables our model to more explicitly reason over objects and their relations during story generation, compared to the global features from an object classifier used in previous work. We apply metrics that account for the diversity of words and phrases of generated stories as well as for reference to narratively-salient image features and show that our approach outperforms previous systems. Our experiments also indicate that our models obtain competitive results on reference-based metrics.","url_abs":"https://aclanthology.org/2020.conll-1.34","url_pdf":"https://aclanthology.org/2020.conll-1.34.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"story-generation","task_name":"Story Generation"},{"task_slug":"visual-storytelling","task_name":"Visual Storytelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-storytelling-on-vist","task":"Visual Storytelling","dataset":"VIST","model":"SGEmb","rank_in_archive_order":6,"of":33,"metrics":{"BLEU-1":"62.2","BLEU-2":"38.7","BLEU-3":"23.5","BLEU-4":"14.8","CIDEr":"8.6","METEOR":"35.6","ROUGE-L":"30.2"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}