{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visual-reasoning-with-multi-hop-feature","title":"Visual Reasoning with Multi-hop Feature Modulation","arxiv_id":"1808.04446","date":"2018-08-03","proceeding":"ECCV 2018 9","authors":["Florian Strub","Mathieu Seurin","Ethan Perez","Harm de Vries","Jérémie Mary","Philippe Preux","Aaron Courville","Olivier Pietquin"],"abstract":"Recent breakthroughs in computer vision and natural language processing have\nspurred interest in challenging multi-modal tasks such as visual\nquestion-answering and visual dialogue. For such tasks, one successful approach\nis to condition image-based convolutional network computation on language via\nFeature-wise Linear Modulation (FiLM) layers, i.e., per-channel scaling and\nshifting. We propose to generate the parameters of FiLM layers going up the\nhierarchy of a convolutional network in a multi-hop fashion rather than all at\nonce, as in prior work. By alternating between attending to the language input\nand generating FiLM layer parameters, this approach is better able to scale to\nsettings with longer input sequences such as dialogue. We demonstrate that\nmulti-hop FiLM generation achieves state-of-the-art for the short input\nsequence task ReferIt --- on-par with single-hop FiLM generation --- while also\nsignificantly outperforming prior state-of-the-art and single-hop FiLM\ngeneration on the GuessWhat?! visual dialogue task.","url_abs":"http://arxiv.org/abs/1808.04446v2","url_pdf":"http://arxiv.org/pdf/1808.04446v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visual-reasoning-with-multi-hop-feature","repo_url":"https://github.com/GuessWhatGame/referit","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-dialogue","task_name":"Visual Dialog"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1808.04446","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}