{"url":"/method/cosine-normalization","slug":"cosine-normalization","name":"Cosine Normalization","full_name":"Cosine Normalization","full_name_withheld":false,"description_markdown":"Multi-layer neural networks traditionally use  dot products between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded. To bound dot product and decrease the variance, **Cosine Normalization** uses cosine similarity or centered cosine similarity (Pearson Correlation Coefficient) instead of dot products in neural networks. \r\n\r\nUsing cosine normalization, the output of a hidden unit is computed by:\r\n\r\n$$o = f(net_{norm})= f(\\cos \\theta) = f(\\frac{\\vec{w} \\cdot \\vec{x}} {\\left|\\vec{w}\\right|  \\left|\\vec{x}\\right|})$$\r\n\r\nwhere $net_{norm}$ is the normalized pre-activation,  $\\vec{w}$ is the incoming weight vector and $\\vec{x}$ is the input vector, ($\\cdot$) indicates dot product, $f$ is nonlinear activation function. Cosine normalization bounds the pre-activation between -1 and 1.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks","paper":"/paper/cosine-normalization-using-cosine-similarity","first_author":"Chunjie Luo","n_authors":4,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/cosine-normalization-using-cosine-similarity"},"source":{"url":"http://arxiv.org/abs/1702.05870v5","title":"Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/iwyoo/tf_conv_cosnorm/blob/e76d987110e4af42ad75154bfbf994c5cc935794/conv_cosnorm.py#L13","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Normalization","url":"/methods/category/normalization","pwc_aliases":[]}],"n_papers_tagged":2,"archive_num_papers":2,"papers_newest_first":[{"paper":"/paper/class-incremental-learning-with-rectified","title":"Class-incremental Learning with Rectified Feature-Graph Preservation","date":"2020-12-15","arxiv_id":"2012.08129","n_code_links":1,"syntology":null},{"paper":"/paper/cosine-normalization-using-cosine-similarity","title":"Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks","date":"2017-02-20","arxiv_id":"1702.05870","n_code_links":1,"syntology":{"ran":0,"of":1,"unverified":1,"pointer_only":0}}],"papers_shown":2,"tasks":[{"task":"/task/class-incremental-learning","name":"Class Incremental Learning","papers":1},{"task":"/task/incremental-learning","name":"Incremental Learning","papers":1},{"task":"/task/class-incremental-learning-1","name":"class-incremental learning","papers":1}],"tasks_shown":3,"n_tasks":3,"usage_by_year":[{"year":"2017","papers":1},{"year":"2020","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/cosine-normalization"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}