{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/matrix-and-tensor-decompositions-for-training","title":"Matrix and tensor decompositions for training binary neural networks","arxiv_id":"1904.07852","date":"2019-04-16","proceeding":null,"authors":["Adrian Bulat","Jean Kossaifi","Georgios Tzimiropoulos","Maja Pantic"],"abstract":"This paper is on improving the training of binary neural networks in which\nboth activations and weights are binary. While prior methods for neural network\nbinarization binarize each filter independently, we propose to instead\nparametrize the weight tensor of each layer using matrix or tensor\ndecomposition. The binarization process is then performed using this latent\nparametrization, via a quantization function (e.g. sign function) applied to\nthe reconstructed weights. A key feature of our method is that while the\nreconstruction is binarized, the computation in the latent factorized space is\ndone in the real domain. This has several advantages: (i) the latent\nfactorization enforces a coupling of the filters before binarization, which\nsignificantly improves the accuracy of the trained models. (ii) while at\ntraining time, the binary weights of each convolutional layer are parametrized\nusing real-valued matrix or tensor decomposition, during inference we simply\nuse the reconstructed (binary) weights. As a result, our method does not\nsacrifice any advantage of binary networks in terms of model compression and\nspeeding-up inference. As a further contribution, instead of computing the\nbinary weight scaling factors analytically, as in prior work, we propose to\nlearn them discriminatively via back-propagation. Finally, we show that our\napproach significantly outperforms existing methods when tested on the\nchallenging tasks of (a) human pose estimation (more than 4% improvements) and\n(b) ImageNet classification (up to 5% performance gains).","url_abs":"http://arxiv.org/abs/1904.07852v1","url_pdf":"http://arxiv.org/pdf/1904.07852v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"binarization","task_name":"Binarization"},{"task_slug":"classification-with-binary-neural-network","task_name":"Classification with Binary Neural Network"},{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"pose-estimation","task_name":"Pose Estimation"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"tensor-decomposition","task_name":"Tensor Decomposition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/pose-estimation-on-mpii-human-pose","task":"Pose Estimation","dataset":"MPII Human Pose","model":"Matrix","rank_in_archive_order":39,"of":46,"metrics":{"PCKh-0.5":"82.5"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}