{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/coarse-to-fine-sparse-transformer-for","title":"Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction","arxiv_id":"2203.04845","date":"2022-03-09","proceeding":null,"authors":["Yuanhao Cai","Jing Lin","Xiaowan Hu","Haoqian Wang","Xin Yuan","Yulun Zhang","Radu Timofte","Luc van Gool"],"abstract":"Many algorithms have been developed to solve the inverse problem of coded aperture snapshot spectral imaging (CASSI), i.e., recovering the 3D hyperspectral images (HSIs) from a 2D compressive measurement. In recent years, learning-based methods have demonstrated promising performance and dominated the mainstream research direction. However, existing CNN-based methods show limitations in capturing long-range dependencies and non-local self-similarity. Previous Transformer-based methods densely sample tokens, some of which are uninformative, and calculate the multi-head self-attention (MSA) between some tokens that are unrelated in content. This does not fit the spatially sparse nature of HSI signals and limits the model scalability. In this paper, we propose a novel Transformer-based method, coarse-to-fine sparse Transformer (CST), firstly embedding HSI sparsity into deep learning for HSI reconstruction. In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing. Comprehensive experiments show that our CST significantly outperforms state-of-the-art methods while requiring cheaper computational costs. The code and models will be released at https://github.com/caiyuanhao1998/MST","url_abs":"https://arxiv.org/abs/2203.04845v3","url_pdf":"https://arxiv.org/pdf/2203.04845v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"coarse-to-fine-sparse-transformer-for","repo_url":"https://github.com/caiyuanhao1998/MST","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"compressive-sensing","task_name":"Compressive Sensing"},{"task_slug":"image-reconstruction","task_name":"Image Reconstruction"},{"task_slug":"spectral-reconstruction","task_name":"Spectral Reconstruction"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"sparse-transformer","method_name":"Sparse Transformer"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/spectral-reconstruction-on-cave","task":"Spectral Reconstruction","dataset":"CAVE","model":"CST-L","rank_in_archive_order":6,"of":10,"metrics":{"PSNR":"36.12","SSIM":"0.957"},"uses_additional_data":false},{"leaderboard":"/sota/spectral-reconstruction-on-kaist","task":"Spectral Reconstruction","dataset":"KAIST","model":"CST-L","rank_in_archive_order":6,"of":10,"metrics":{"PSNR":"36.12","SSIM":"0.957"},"uses_additional_data":false},{"leaderboard":"/sota/spectral-reconstruction-on-real-hsi","task":"Spectral Reconstruction","dataset":"Real HSI","model":"CST-L","rank_in_archive_order":4,"of":9,"metrics":{"User Study Score":"14"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2203.04845","atlas_url":"https://app.syntology.ai/?focus=2203.04845","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.04845"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/caiyuanhao1998/MST","reach":null}],"summary":{"ran":9,"ran_fixture":2,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":16,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e6b806ed01455bd5","entry":"ASPP","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6b806ed01455bd5"}},{"code_sha256_prefix":"4bd71cba3602449c","entry":"ASPPConv","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4bd71cba3602449c"}},{"code_sha256_prefix":"dd36c1718253bd99","entry":"ASPPPooling","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd36c1718253bd99"}},{"code_sha256_prefix":"58f6128456d2627d","entry":"AsymmetricTransform","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58f6128456d2627d"}},{"code_sha256_prefix":"af8bb615fc66f945","entry":"LSH","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af8bb615fc66f945"}},{"code_sha256_prefix":"7dea4c5caae45bd4","entry":"SAH_MSA","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7dea4c5caae45bd4"}},{"code_sha256_prefix":"6a998c9227157700","entry":"SALSH","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a998c9227157700"}},{"code_sha256_prefix":"45281a471fb34442","entry":"Sparsity_Estimator","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"45281a471fb34442"}},{"code_sha256_prefix":"6d329196f36c232e","entry":"XBOXPLUS","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d329196f36c232e"}},{"code_sha256_prefix":"169dedccb5536ed4","entry":"batch_gather","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"169dedccb5536ed4"}},{"code_sha256_prefix":"8b3f512fc8acb3da","entry":"batch_scatter","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b3f512fc8acb3da"}},{"code_sha256_prefix":"aa9a1d1268999fa4","entry":"lsh_clustering","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa9a1d1268999fa4"}},{"code_sha256_prefix":"acc89f0bd5a8ecc6","entry":"uniform","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"acc89f0bd5a8ecc6"}},{"code_sha256_prefix":"128f3f681fa4e5a2","entry":"CST","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"128f3f681fa4e5a2"}},{"code_sha256_prefix":"95caa6a06d8dfedc","entry":"SAHAB","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95caa6a06d8dfedc"}},{"code_sha256_prefix":"0885baf6a0f88d0a","entry":"SAHABs","repo":"caiyuanhao1998/MST","repo_kind":"official","path":"real/train_code/architecture/CST.py","file_url":"https://github.com/caiyuanhao1998/MST/blob/HEAD/real/train_code/architecture/CST.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0885baf6a0f88d0a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}