{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visual-saliency-transformer","title":"Visual Saliency Transformer","arxiv_id":"2104.12099","date":"2021-04-25","proceeding":"ICCV 2021 10","authors":["Nian Liu","Ni Zhang","Kaiyuan Wan","Ling Shao","Junwei Han"],"abstract":"Existing state-of-the-art saliency detection methods heavily rely on CNN-based architectures. Alternatively, we rethink this task from a convolution-free sequence-to-sequence perspective and predict saliency by modeling long-range dependencies, which can not be achieved by convolution. Specifically, we develop a novel unified model based on a pure transformer, namely, Visual Saliency Transformer (VST), for both RGB and RGB-D salient object detection (SOD). It takes image patches as inputs and leverages the transformer to propagate global contexts among image patches. Unlike conventional architectures used in Vision Transformer (ViT), we leverage multi-level token fusion and propose a new token upsampling method under the transformer framework to get high-resolution detection results. We also develop a token-based multi-task decoder to simultaneously perform saliency and boundary detection by introducing task-related tokens and a novel patch-task-attention mechanism. Experimental results show that our model outperforms existing methods on both RGB and RGB-D SOD benchmark datasets. Most importantly, our whole framework not only provides a new perspective for the SOD field but also shows a new paradigm for transformer-based dense prediction models. Code is available at https://github.com/nnizhang/VST.","url_abs":"https://arxiv.org/abs/2104.12099v2","url_pdf":"https://arxiv.org/pdf/2104.12099v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visual-saliency-transformer","repo_url":"https://github.com/nnizhang/VST","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"visual-saliency-transformer","repo_url":"https://github.com/fhshen2022/prunerepaint","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"boundary-detection","task_name":"Boundary Detection"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"rgb-d-salient-object-detection","task_name":"RGB-D Salient Object Detection"},{"task_slug":"saliency-detection","task_name":"Saliency Detection"},{"task_slug":"salient-object-detection-1","task_name":"Salient Object Detection"},{"task_slug":"thermal-image-segmentation","task_name":"Thermal Image Segmentation"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/rgb-d-salient-object-detection-on-njud","task":"RGB-D Salient Object Detection","dataset":"NJUD","model":"VST","rank_in_archive_order":1,"of":1,"metrics":{"S-Measure":"0.922"},"uses_additional_data":false},{"leaderboard":"/sota/rgb-d-salient-object-detection-on-nlpr","task":"RGB-D Salient Object Detection","dataset":"NLPR","model":"VST","rank_in_archive_order":14,"of":14,"metrics":{"S-Measure":"0.932"},"uses_additional_data":false},{"leaderboard":"/sota/rgb-d-salient-object-detection-on-sip","task":"RGB-D Salient Object Detection","dataset":"SIP","model":"VST","rank_in_archive_order":2,"of":16,"metrics":{"Average MAE":"0.040","S-Measure":"90.4","max E-Measure":"94.4","max F-Measure":"91.5"},"uses_additional_data":false},{"leaderboard":"/sota/thermal-image-segmentation-on-rgb-t-glass","task":"Thermal Image Segmentation","dataset":"RGB-T-Glass-Segmentation","model":"VST","rank_in_archive_order":7,"of":22,"metrics":{"MAE":"0.044"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.12099","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.12099"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fhshen2022/prunerepaint","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/nnizhang/VST","reach":null}],"summary":{"ran":6,"ran_honours":1,"unverified":4},"by_repo_kind":{"named_in_paper":{"samples":4,"ran":3,"repositories":1},"listed":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"e518414e453d6bd4","entry":"Attention","repo":"nnizhang/VST","repo_kind":"named_in_paper","path":"RGBD_VST/Models/Transformer.py","file_url":"https://github.com/nnizhang/VST/blob/HEAD/RGBD_VST/Models/Transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e518414e453d6bd4"}},{"code_sha256_prefix":"f19483886b6caa8a","entry":"MutualAttention","repo":"nnizhang/VST","repo_kind":"named_in_paper","path":"RGBD_VST/Models/Transformer.py","file_url":"https://github.com/nnizhang/VST/blob/HEAD/RGBD_VST/Models/Transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f19483886b6caa8a"}},{"code_sha256_prefix":"ad3ceaa863c245d9","entry":"MutualSelfBlock","repo":"nnizhang/VST","repo_kind":"named_in_paper","path":"RGBD_VST/Models/Transformer.py","file_url":"https://github.com/nnizhang/VST/blob/HEAD/RGBD_VST/Models/Transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad3ceaa863c245d9"}},{"code_sha256_prefix":"8f4eb7ec658f7737","entry":"T2T_module","repo":"fhshen2022/prunerepaint","repo_kind":"listed","path":"RGB_VST/Models/t2t_vit.py","file_url":"https://github.com/fhshen2022/prunerepaint/blob/HEAD/RGB_VST/Models/t2t_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f4eb7ec658f7737"}},{"code_sha256_prefix":"43b5c1f0c5692aae","entry":"Token_performer","repo":"fhshen2022/prunerepaint","repo_kind":"listed","path":"RGB_VST/Models/t2t_vit.py","file_url":"https://github.com/fhshen2022/prunerepaint/blob/HEAD/RGB_VST/Models/t2t_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"43b5c1f0c5692aae"}},{"code_sha256_prefix":"a38451f0ca6db8b5","entry":"Token_transformer","repo":"fhshen2022/prunerepaint","repo_kind":"listed","path":"RGB_VST/Models/t2t_vit.py","file_url":"https://github.com/fhshen2022/prunerepaint/blob/HEAD/RGB_VST/Models/t2t_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a38451f0ca6db8b5"}},{"code_sha256_prefix":"ced6bc2bfdfe0059","entry":"get_sinusoid_encoding","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"ced6bc2bfdfe0059"}},{"code_sha256_prefix":"f6f001c7931d49f2","entry":"Attention","repo":"fhshen2022/prunerepaint","repo_kind":"listed","path":"RGB_VST/Models/t2t_vit.py","file_url":"https://github.com/fhshen2022/prunerepaint/blob/HEAD/RGB_VST/Models/t2t_vit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f6f001c7931d49f2"}},{"code_sha256_prefix":"7c18bbee6c5d2255","entry":"Block","repo":"fhshen2022/prunerepaint","repo_kind":"listed","path":"RGB_VST/Models/t2t_vit.py","file_url":"https://github.com/fhshen2022/prunerepaint/blob/HEAD/RGB_VST/Models/t2t_vit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c18bbee6c5d2255"}},{"code_sha256_prefix":"e1a7b5ac434bab26","entry":"T2T_ViT","repo":"fhshen2022/prunerepaint","repo_kind":"listed","path":"RGB_VST/Models/t2t_vit.py","file_url":"https://github.com/fhshen2022/prunerepaint/blob/HEAD/RGB_VST/Models/t2t_vit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e1a7b5ac434bab26"}},{"code_sha256_prefix":"f9290299b380bb93","entry":"TransformerEncoder","repo":"nnizhang/VST","repo_kind":"named_in_paper","path":"RGBD_VST/Models/Transformer.py","file_url":"https://github.com/nnizhang/VST/blob/HEAD/RGBD_VST/Models/Transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9290299b380bb93"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}