{"url":"/method/orthogonal-regularization","slug":"orthogonal-regularization","name":"Orthogonal Regularization","full_name":"Orthogonal Regularization","full_name_withheld":false,"description_markdown":"**Orthogonal Regularization** is a regularization technique for convolutional neural networks, introduced with generative modelling as the task in mind. Orthogonality is argued to be a desirable quality in ConvNet filters, partially because multiplication by an orthogonal matrix leaves the norm of the original matrix unchanged. This property is valuable in deep or recurrent networks, where repeated matrix multiplication can result in signals vanishing or exploding. To try to maintain orthogonality throughout training, Orthogonal Regularization encourages weights to be orthogonal by pushing them towards the nearest orthogonal manifold. The objective function is augmented with the cost:\r\n\r\n$$ \\mathcal{L}\\_{ortho} = \\sum\\left(|WW^{T} − I|\\right) $$\r\n\r\nWhere $\\sum$ indicates a sum across all filter banks, $W$ is a filter bank, and $I$ is the identity matrix","description_state":"present","introduced_year":null,"introduced_by":{"title":"Neural Photo Editing with Introspective Adversarial Networks","paper":"/paper/neural-photo-editing-with-introspective","first_author":"Andrew Brock","n_authors":4,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/neural-photo-editing-with-introspective"},"source":{"url":"http://arxiv.org/abs/1609.07093v3","title":"Neural Photo Editing with Introspective Adversarial Networks","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/pytorch/pytorch/blob/b7bda236d18815052378c88081f64935427d7716/torch/nn/init.py#L423","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Regularization","url":"/methods/category/regularization","pwc_aliases":[]}],"n_papers_tagged":27,"archive_num_papers":27,"papers_newest_first":[{"paper":null,"title":"Stochastic Orthogonal Regularization for deep projective priors","date":"2025-05-19","arxiv_id":"2505.13078","n_code_links":0,"syntology":null},{"paper":"/paper/preventing-dimensional-collapse-in-self","title":"Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization","date":"2024-11-01","arxiv_id":"2411.00392","n_code_links":1,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":1}},{"paper":"/paper/principal-orthogonal-latent-components","title":"Principal Orthogonal Latent Components Analysis (POLCA Net)","date":"2024-10-09","arxiv_id":"2410.07289","n_code_links":1,"syntology":null},{"paper":null,"title":"Convolutional Neural Network Compression Based on Low-Rank Decomposition","date":"2024-08-29","arxiv_id":"2408.16289","n_code_links":0,"syntology":null},{"paper":"/paper/cross-view-geolocalization-and-disaster","title":"Cross-View Geolocalization and Disaster Mapping with Street-View and VHR Satellite Imagery: A Case Study of Hurricane IAN","date":"2024-08-13","arxiv_id":"2408.06761","n_code_links":1,"syntology":null},{"paper":null,"title":"Efficient Pareto Manifold Learning with Low-Rank Structure","date":"2024-07-30","arxiv_id":"2407.20734","n_code_links":0,"syntology":null},{"paper":"/paper/learn-to-preserve-and-diversify-parameter","title":"Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalization","date":"2024-07-21","arxiv_id":"2407.15085","n_code_links":1,"syntology":{"ran":11,"of":13,"unverified":2,"pointer_only":0}},{"paper":null,"title":"Energizing Federated Learning via Filter-Aware Attention","date":"2023-11-18","arxiv_id":"2311.12049","n_code_links":0,"syntology":null},{"paper":null,"title":"Free Lunch for Gait Recognition: A Novel Relation Descriptor","date":"2023-08-22","arxiv_id":"2308.11487","n_code_links":0,"syntology":null},{"paper":"/paper/inferior-alveolar-nerve-segmentation-in-cbct","title":"Inferior Alveolar Nerve Segmentation in CBCT images using Connectivity-Based Selective Re-training","date":"2023-08-18","arxiv_id":"2308.09298","n_code_links":1,"syntology":null},{"paper":"/paper/revisiting-orthogonality-regularization-a","title":"Revisiting Orthogonality Regularization: A Study for Convolutional Neural Networks in Image Classification","date":"2022-06-23","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/profairrec-provider-fairness-aware-news","title":"ProFairRec: Provider Fairness-aware News Recommendation","date":"2022-04-10","arxiv_id":"2204.04724","n_code_links":1,"syntology":null},{"paper":null,"title":"Dreaming To Prune Image Deraining Networks","date":"2022-01-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"title":"Ortho-Shot: Low Displacement Rank Regularization with Data Augmentation for Few-Shot Learning","date":"2021-10-18","arxiv_id":"2110.09374","n_code_links":0,"syntology":null},{"paper":"/paper/subspace-learning-for-personalized-federated","title":"Connecting Low-Loss Subspace for Personalized Federated Learning","date":"2021-09-16","arxiv_id":"2109.07628","n_code_links":1,"syntology":{"ran":0,"of":2,"unverified":2,"pointer_only":2}},{"paper":null,"title":"Arabic aspect based sentiment analysis using bidirectional GRU based models","date":"2021-01-23","arxiv_id":"2101.10539","n_code_links":0,"syntology":null},{"paper":"/paper/disentangling-latent-space-for-unsupervised","title":"Towards Disentangling Latent Space for Unsupervised Semantic Face Editing","date":"2020-11-05","arxiv_id":"2011.02638","n_code_links":1,"syntology":null},{"paper":"/paper/a-spectral-energy-distance-for-parallel","title":"A Spectral Energy Distance for Parallel Speech Synthesis","date":"2020-08-03","arxiv_id":"2008.01160","n_code_links":2,"syntology":{"ran":0,"of":3,"unverified":3,"pointer_only":0}},{"paper":"/paper/2003-13549","title":"Rethinking Depthwise Separable Convolutions: How Intra-Kernel Correlations Lead to Improved MobileNets","date":"2020-03-30","arxiv_id":"2003.13549","n_code_links":1,"syntology":null},{"paper":"/paper/transformation-based-adversarial-video","title":"Transformation-based Adversarial Video Prediction on Large-Scale Data","date":"2020-03-09","arxiv_id":"2003.04035","n_code_links":0,"syntology":null},{"paper":"/paper/efficient-model-for-image-classification-with","title":"Efficient Model for Image Classification With Regularization Tricks","date":"2020-02-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/high-fidelity-speech-synthesis-with-1","title":"High Fidelity Speech Synthesis with Adversarial Networks","date":"2019-09-25","arxiv_id":"1909.11646","n_code_links":3,"syntology":{"ran":2,"of":5,"unverified":3,"pointer_only":4}},{"paper":"/paper/efficient-video-generation-on-complex","title":"Adversarial Video Generation on Complex Datasets","date":"2019-07-15","arxiv_id":"1907.06571","n_code_links":1,"syntology":null},{"paper":"/paper/oogan-disentangling-gan-with-one-hot-sampling","title":"OOGAN: Disentangling GAN with One-Hot Sampling and Orthogonal Regularization","date":"2019-05-26","arxiv_id":"1905.10836","n_code_links":1,"syntology":null},{"paper":null,"title":"CAN: Constrained Attention Networks for Multi-Aspect Sentiment Analysis","date":"2018-12-27","arxiv_id":"1812.10735","n_code_links":0,"syntology":null},{"paper":"/paper/large-scale-gan-training-for-high-fidelity","title":"Large Scale GAN Training for High Fidelity Natural Image Synthesis","date":"2018-09-28","arxiv_id":"1809.11096","n_code_links":35,"syntology":{"ran":14,"of":41,"unverified":27,"pointer_only":4}},{"paper":"/paper/neural-photo-editing-with-introspective","title":"Neural Photo Editing with Introspective Adversarial Networks","date":"2016-09-22","arxiv_id":"1609.07093","n_code_links":2,"syntology":{"ran":2,"of":2,"unverified":0,"pointer_only":0}}],"papers_shown":27,"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":3},{"task":"/task/image-generation","name":"Image Generation","papers":3},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":2},{"task":"/task/dimensionality-reduction","name":"Dimensionality Reduction","papers":2},{"task":"/task/federated-learning","name":"Federated Learning","papers":2},{"task":"/task/classification","name":"General Classification","papers":2},{"task":"/task/model-compression","name":"Model Compression","papers":2},{"task":"/task/sentence","name":"Sentence","papers":2},{"task":"/task/sentiment-analysis","name":"Sentiment Analysis","papers":2},{"task":"/task/speech-synthesis","name":"Speech Synthesis","papers":2},{"task":"/task/video-generation","name":"Video Generation","papers":2},{"task":"/task/video-prediction","name":"Video Prediction","papers":2},{"task":"/task/high","name":"Vocal Bursts Intensity Prediction","papers":2},{"task":"/task/image-classification","name":"image-classification","papers":2},{"task":"/task/3d-character-animation-from-a-single-photo","name":"3D Character Animation From A Single Photo","papers":1},{"task":"/task/articles","name":"Articles","papers":1},{"task":"/task/aspect-based-sentiment-analysis-1","name":"Aspect-Based Sentiment Analysis","papers":1},{"task":"/task/aspect-based-sentiment-analysis","name":"Aspect-Based Sentiment Analysis (ABSA)","papers":1},{"task":"/task/attribute","name":"Attribute","papers":1},{"task":"/task/classification-1","name":"Classification","papers":1}],"tasks_shown":20,"n_tasks":52,"usage_by_year":[{"year":"2016","papers":1},{"year":"2018","papers":2},{"year":"2019","papers":3},{"year":"2020","papers":5},{"year":"2021","papers":3},{"year":"2022","papers":3},{"year":"2023","papers":3},{"year":"2024","papers":6},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/orthogonal-regularization"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}