{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-do-deep-saliency-models-learn-about-1","title":"What Do Deep Saliency Models Learn about Visual Attention?","arxiv_id":"2310.09679","date":"2023-10-14","proceeding":"NeurIPS 2023 11","authors":["Shi Chen","Ming Jiang","Qi Zhao"],"abstract":"In recent years, deep saliency models have made significant progress in predicting human visual attention. However, the mechanisms behind their success remain largely unexplained due to the opaque nature of deep neural networks. In this paper, we present a novel analytic framework that sheds light on the implicit features learned by saliency models and provides principled interpretation and quantification of their contributions to saliency prediction. Our approach decomposes these implicit features into interpretable bases that are explicitly aligned with semantic attributes and reformulates saliency prediction as a weighted combination of probability maps connecting the bases and saliency. By applying our framework, we conduct extensive analyses from various perspectives, including the positive and negative weights of semantics, the impact of training data and architectural designs, the progressive influences of fine-tuning, and common failure patterns of state-of-the-art deep saliency models. Additionally, we demonstrate the effectiveness of our framework by exploring visual attention characteristics in various application scenarios, such as the atypical attention of people with autism spectrum disorder, attention to emotion-eliciting stimuli, and attention evolution over time. Our code is publicly available at \\url{https://github.com/szzexpoi/saliency_analysis}.","url_abs":"https://arxiv.org/abs/2310.09679v1","url_pdf":"https://arxiv.org/pdf/2310.09679v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-do-deep-saliency-models-learn-about-1","repo_url":"https://github.com/szzexpoi/saliency_analysis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"saliency-prediction","task_name":"Saliency Prediction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2310.09679","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.09679"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/szzexpoi/saliency_analysis","reach":null}],"summary":{"ran":7,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":14,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"bbf87470b090a5d9","entry":"Inception_Encoder","repo":"szzexpoi/saliency_analysis","repo_kind":"official","path":"prototype_dissection.py","file_url":"https://github.com/szzexpoi/saliency_analysis/blob/HEAD/prototype_dissection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bbf87470b090a5d9"}},{"code_sha256_prefix":"ae7e55b26b6c63fa","entry":"MLP","repo":"szzexpoi/saliency_analysis","repo_kind":"official","path":"prototype_dissection.py","file_url":"https://github.com/szzexpoi/saliency_analysis/blob/HEAD/prototype_dissection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae7e55b26b6c63fa"}},{"code_sha256_prefix":"67bbc52e2db5c13f","entry":"PatchEncoder","repo":"szzexpoi/saliency_analysis","repo_kind":"official","path":"prototype_dissection.py","file_url":"https://github.com/szzexpoi/saliency_analysis/blob/HEAD/prototype_dissection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"67bbc52e2db5c13f"}},{"code_sha256_prefix":"7bf51d5279e351ea","entry":"Proto_Decoder","repo":"szzexpoi/saliency_analysis","repo_kind":"official","path":"prototype_dissection.py","file_url":"https://github.com/szzexpoi/saliency_analysis/blob/HEAD/prototype_dissection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7bf51d5279e351ea"}},{"code_sha256_prefix":"6cade961c640cc5e","entry":"ResNet","repo":"szzexpoi/saliency_analysis","repo_kind":"official","path":"prototype_dissection.py","file_url":"https://github.com/szzexpoi/saliency_analysis/blob/HEAD/prototype_dissection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6cade961c640cc5e"}},{"code_sha256_prefix":"82100b664e3f65ec","entry":"Simple_Decoder","repo":"szzexpoi/saliency_analysis","repo_kind":"official","path":"prototype_dissection.py","file_url":"https://github.com/szzexpoi/saliency_analysis/blob/HEAD/prototype_dissection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"82100b664e3f65ec"}},{"code_sha256_prefix":"cee2ce9baa77dd88","entry":"Trans_encoder","repo":"szzexpoi/saliency_analysis","repo_kind":"official","path":"prototype_dissection.py","file_url":"https://github.com/szzexpoi/saliency_analysis/blob/HEAD/prototype_dissection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cee2ce9baa77dd88"}},{"code_sha256_prefix":"6b45190dae2942f6","entry":"resnet50","repo":"szzexpoi/saliency_analysis","repo_kind":"official","path":"prototype_dissection.py","file_url":"https://github.com/szzexpoi/saliency_analysis/blob/HEAD/prototype_dissection.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b45190dae2942f6"}},{"code_sha256_prefix":"0dacde37ea08e367","entry":"DINet","repo":"szzexpoi/saliency_analysis","repo_kind":"official","path":"prototype_dissection.py","file_url":"https://github.com/szzexpoi/saliency_analysis/blob/HEAD/prototype_dissection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0dacde37ea08e367"}},{"code_sha256_prefix":"630eb55dff774d6e","entry":"SALICON","repo":"szzexpoi/saliency_analysis","repo_kind":"official","path":"prototype_dissection.py","file_url":"https://github.com/szzexpoi/saliency_analysis/blob/HEAD/prototype_dissection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"630eb55dff774d6e"}},{"code_sha256_prefix":"5f960bc5b10ad788","entry":"TranSalNet","repo":"szzexpoi/saliency_analysis","repo_kind":"official","path":"prototype_dissection.py","file_url":"https://github.com/szzexpoi/saliency_analysis/blob/HEAD/prototype_dissection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f960bc5b10ad788"}},{"code_sha256_prefix":"46e9fb4dced5b47f","entry":"VG_generator","repo":"szzexpoi/saliency_analysis","repo_kind":"official","path":"prototype_dissection.py","file_url":"https://github.com/szzexpoi/saliency_analysis/blob/HEAD/prototype_dissection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46e9fb4dced5b47f"}},{"code_sha256_prefix":"423681206c2b5298","entry":"compute_iou","repo":"szzexpoi/saliency_analysis","repo_kind":"official","path":"prototype_dissection.py","file_url":"https://github.com/szzexpoi/saliency_analysis/blob/HEAD/prototype_dissection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"423681206c2b5298"}},{"code_sha256_prefix":"289a8337ec2a6d9d","entry":"prototype_dissection","repo":"szzexpoi/saliency_analysis","repo_kind":"official","path":"prototype_dissection.py","file_url":"https://github.com/szzexpoi/saliency_analysis/blob/HEAD/prototype_dissection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"289a8337ec2a6d9d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}