{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fsanet-frequency-self-attention-for-semantic","title":"FsaNet: Frequency Self-attention for Semantic Segmentation","arxiv_id":"2211.15595","date":"2022-11-28","proceeding":null,"authors":["Fengyu Zhang","Ashkan Panahi","Guangjun Gao"],"abstract":"Considering the spectral properties of images, we propose a new self-attention mechanism with highly reduced computational complexity, up to a linear rate. To better preserve edges while promoting similarity within objects, we propose individualized processes over different frequency bands. In particular, we study a case where the process is merely over low-frequency components. By ablation study, we show that low frequency self-attention can achieve very close or better performance relative to full frequency even without retraining the network. Accordingly, we design and embed novel plug-and-play modules to the head of a CNN network that we refer to as FsaNet. The frequency self-attention 1) requires only a few low frequency coefficients as input, 2) can be mathematically equivalent to spatial domain self-attention with linear structures, 3) simplifies token mapping ($1\\times1$ convolution) stage and token mixing stage simultaneously. We show that frequency self-attention requires $87.29\\% \\sim 90.04\\%$ less memory, $96.13\\% \\sim 98.07\\%$ less FLOPs, and $97.56\\% \\sim 98.18\\%$ in run time than the regular self-attention. Compared to other ResNet101-based self-attention networks, \\ourM achieves a new \\sArt result ($83.0\\%$ mIoU) on Cityscape test dataset and competitive results on ADE20k and VOCaug. \\ourM can also enhance MASK R-CNN for instance segmentation on COCO. In addition, utilizing the proposed module, Segformer can be boosted on a series of models with different scales, and Segformer-B5 can be improved even without retraining. Code is accessible at \\url{https://github.com/zfy-csu/FsaNet","url_abs":"https://arxiv.org/abs/2211.15595v3","url_pdf":"https://arxiv.org/pdf/2211.15595v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fsanet-frequency-self-attention-for-semantic","repo_url":"https://github.com/zfy-csu/fsanet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"mask-r-cnn","method_name":"Mask R-CNN"},{"method_slug":"mix-ffn","method_name":"Mix-FFN"},{"method_slug":"rpn","method_name":"RPN"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"roi-align","method_name":"RoIAlign"},{"method_slug":"segformer","method_name":"SegFormer"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2211.15595","atlas_url":"https://app.syntology.ai/?focus=2211.15595","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}