{"url":"/method/spectral-normalized-identity-priors","slug":"spectral-normalized-identity-priors","name":"Spectral-Normalized Identity Priors","full_name":"Spectral-Normalized Identity Priors","full_name_withheld":false,"description_markdown":"**Spectral-Normalized Identity Priors**, or **SNIP**, is a structured pruning approach that penalizes an entire [residual module](https://paperswithcode.com/method/residual-connection) in a [Transformer model](https://paperswithcode.com/method/residual-connection) toward an identity mapping. It is applicable to any structured module, including a single [attention head](https://paperswithcode.com/method/scaled), an [entire attention block](https://paperswithcode.com/method/multi-head-attention), or a [feed-forward subnetwork](https://paperswithcode.com/method/position-wise-feed-forward-layer). The method identifies and discards unimportant non-linear mappings in the [residual connections](https://paperswithcode.com/method/residual-connection) by applying a thresholding operator on the function norm. Furthermore, [spectral normalization](https://paperswithcode.com/method/spectral-normalization) to stabilize the distribution of the post-activation values of the [Transformer](https://paperswithcode.com/method/transformer) layers, further improving the pruning effectiveness of the proposed methodology.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/2010.01791v1","title":"Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Pruning","url":"/methods/category/pruning","pwc_aliases":[]}],"n_papers_tagged":1,"archive_num_papers":null,"papers_newest_first":[{"paper":"/paper/pruning-redundant-mappings-in-transformer","title":"Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior","date":"2020-10-05","arxiv_id":"2010.01791","n_code_links":1,"syntology":null}],"papers_shown":1,"tasks":[],"tasks_shown":0,"n_tasks":0,"usage_by_year":[{"year":"2020","papers":1}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/spectral-normalized-identity-priors"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}