{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/modulating-early-visual-processing-by","title":"Modulating early visual processing by language","arxiv_id":"1707.00683","date":"2017-07-02","proceeding":"NeurIPS 2017 12","authors":["Harm de Vries","Florian Strub","Jérémie Mary","Hugo Larochelle","Olivier Pietquin","Aaron Courville"],"abstract":"It is commonly assumed that language refers to high-level visual concepts\nwhile leaving low-level visual processing unaffected. This view dominates the\ncurrent literature in computational models for language-vision tasks, where\nvisual and linguistic input are mostly processed independently before being\nfused into a single representation. In this paper, we deviate from this classic\npipeline and propose to modulate the \\emph{entire visual processing} by\nlinguistic input. Specifically, we condition the batch normalization parameters\nof a pretrained residual network (ResNet) on a language embedding. This\napproach, which we call MOdulated RESnet (\\MRN), significantly improves strong\nbaselines on two visual question answering tasks. Our ablation study shows that\nmodulating from the early stages of the visual processing is beneficial.","url_abs":"http://arxiv.org/abs/1707.00683v3","url_pdf":"http://arxiv.org/pdf/1707.00683v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"modulating-early-visual-processing-by","repo_url":"https://github.com/GuessWhatGame/clevr","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"modulating-early-visual-processing-by","repo_url":"https://github.com/ap229997/Conditional-Batch-Norm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"conditional-batch-normalization","method_name":"Conditional Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"modern","method_name":"MODERN"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[{"slug":"conditional-batch-normalization","name":"Conditional Batch Normalization","full_name":"Conditional Batch Normalization"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1707.00683","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}