{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/devil-decoding-vision-features-into-language","title":"DeViL: Decoding Vision features into Language","arxiv_id":"2309.01617","date":"2023-09-04","proceeding":null,"authors":["Meghal Dani","Isabel Rio-Torto","Stephan Alaniz","Zeynep Akata"],"abstract":"Post-hoc explanation methods have often been criticised for abstracting away the decision-making process of deep neural networks. In this work, we would like to provide natural language descriptions for what different layers of a vision backbone have learned. Our DeViL method decodes vision features into language, not only highlighting the attribution locations but also generating textual descriptions of visual features at different layers of the network. We train a transformer network to translate individual image features of any vision layer into a prompt that a separate off-the-shelf language model decodes into natural language. By employing dropout both per-layer and per-spatial-location, our model can generalize training on image-text pairs to generate localized explanations. As it uses a pre-trained language model, our approach is fast to train, can be applied to any vision backbone, and produces textual descriptions at different layers of the vision network. Moreover, DeViL can create open-vocabulary attribution maps corresponding to words or phrases even outside the training scope of the vision model. We demonstrate that DeViL generates textual descriptions relevant to the image content on CC3M surpassing previous lightweight captioning models and attribution maps uncovering the learned concepts of the vision backbone. Finally, we show DeViL also outperforms the current state-of-the-art on the neuron-wise descriptions of the MILANNOTATIONS dataset. Code available at https://github.com/ExplainableML/DeViL","url_abs":"https://arxiv.org/abs/2309.01617v1","url_pdf":"https://arxiv.org/pdf/2309.01617v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"devil-decoding-vision-features-into-language","repo_url":"https://github.com/ExplainableML/DeViL","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"dropout","method_name":"Dropout"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.01617","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.01617"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ExplainableML/DeViL","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"9154e87fce3eb896","entry":"clip_resnet_forward","repo":"ExplainableML/DeViL","repo_kind":"official","path":"src/vision_models.py","file_url":"https://github.com/ExplainableML/DeViL/blob/HEAD/src/vision_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"9154e87fce3eb896"}},{"code_sha256_prefix":"094b32dc22e046bb","entry":"filter_no_caption_or_no_image","repo":"ExplainableML/DeViL","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/ExplainableML/DeViL/blob/HEAD/src/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"094b32dc22e046bb"}},{"code_sha256_prefix":"ad4528925f4a4d1f","entry":"get_resnet_dims","repo":"ExplainableML/DeViL","repo_kind":"official","path":"src/vision_models.py","file_url":"https://github.com/ExplainableML/DeViL/blob/HEAD/src/vision_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"ad4528925f4a4d1f"}},{"code_sha256_prefix":"6c584e98426ef212","entry":"grid","repo":"ExplainableML/DeViL","repo_kind":"official","path":"src/evaluation.py","file_url":"https://github.com/ExplainableML/DeViL/blob/HEAD/src/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"6c584e98426ef212"}},{"code_sha256_prefix":"46595d79c0a67ea8","entry":"chunkIt","repo":"ExplainableML/DeViL","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/ExplainableML/DeViL/blob/HEAD/src/dataset.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"46595d79c0a67ea8"}},{"code_sha256_prefix":"ac6140b208d4d104","entry":"get_language_model","repo":"ExplainableML/DeViL","repo_kind":"official","path":"src/language_model.py","file_url":"https://github.com/ExplainableML/DeViL/blob/HEAD/src/language_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"ac6140b208d4d104"}},{"code_sha256_prefix":"7d604a210ad0a3d9","entry":"get_tokenizer","repo":"ExplainableML/DeViL","repo_kind":"official","path":"src/language_model.py","file_url":"https://github.com/ExplainableML/DeViL/blob/HEAD/src/language_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"7d604a210ad0a3d9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}