{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/aligning-books-and-movies-towards-story-like","title":"Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books","arxiv_id":"1506.06724","date":"2015-06-22","proceeding":"ICCV 2015 12","authors":["Yukun Zhu","Ryan Kiros","Richard Zemel","Ruslan Salakhutdinov","Raquel Urtasun","Antonio Torralba","Sanja Fidler"],"abstract":"Books are a rich source of both fine-grained information, how a character, an\nobject or a scene looks like, as well as high-level semantics, what someone is\nthinking, feeling and how these states evolve through a story. This paper aims\nto align books to their movie releases in order to provide rich descriptive\nexplanations for visual content that go semantically far beyond the captions\navailable in current datasets. To align movies and books we exploit a neural\nsentence embedding that is trained in an unsupervised way from a large corpus\nof books, as well as a video-text neural embedding for computing similarities\nbetween movie clips and sentences in the book. We propose a context-aware CNN\nto combine information from multiple sources. We demonstrate good quantitative\nperformance for movie/book alignment and show several qualitative examples that\nshowcase the diversity of tasks our model can be used for.","url_abs":"http://arxiv.org/abs/1506.06724v1","url_pdf":"http://arxiv.org/pdf/1506.06724v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"aligning-books-and-movies-towards-story-like","repo_url":"https://github.com/altsoph/paranoid_transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"aligning-books-and-movies-towards-story-like","repo_url":"https://github.com/soskek/bookcorpus","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"aligning-books-and-movies-towards-story-like","repo_url":"https://github.com/soskek/homemade_bookcorpus","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"sentence-embedding","task_name":"Sentence Embedding"},{"task_slug":"sentence-embedding-1","task_name":"Sentence-Embedding"}],"methods":[],"datasets_introduced":[{"slug":"bookcorpus","name":"BookCorpus","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1506.06724","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}