{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/understanding-the-disharmony-between-dropout","title":"Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift","arxiv_id":"1801.05134","date":"2018-01-16","proceeding":"CVPR 2019 6","authors":["Xiang Li","Shuo Chen","Xiaolin Hu","Jian Yang"],"abstract":"This paper first answers the question \"why do the two most powerful\ntechniques Dropout and Batch Normalization (BN) often lead to a worse\nperformance when they are combined together?\" in both theoretical and\nstatistical aspects. Theoretically, we find that Dropout would shift the\nvariance of a specific neural unit when we transfer the state of that network\nfrom train to test. However, BN would maintain its statistical variance, which\nis accumulated from the entire learning procedure, in the test phase. The\ninconsistency of that variance (we name this scheme as \"variance shift\") causes\nthe unstable numerical behavior in inference that leads to more erroneous\npredictions finally, when applying Dropout before BN. Thorough experiments on\nDenseNet, ResNet, ResNeXt and Wide ResNet confirm our findings. According to\nthe uncovered mechanism, we next explore several strategies that modifies\nDropout and try to overcome the limitations of their combination by avoiding\nthe variance shift risks.","url_abs":"http://arxiv.org/abs/1801.05134v1","url_pdf":"http://arxiv.org/pdf/1801.05134v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understanding-the-disharmony-between-dropout","repo_url":"https://github.com/m090009/Behavioral_Cloning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"understanding-the-disharmony-between-dropout","repo_url":"https://github.com/simo-bat/Crack_detection","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"understanding-the-disharmony-between-dropout","repo_url":"https://github.com/statsu1990/regression_dropout","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}},{"paper_slug":"understanding-the-disharmony-between-dropout","repo_url":"https://github.com/tcheung99/ResNet_MiniProject","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"understanding-the-disharmony-between-dropout","repo_url":"https://github.com/yang-su2000/ML-practice","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"grouped-convolution","method_name":"Grouped Convolution"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"resnext","method_name":"ResNeXt"},{"method_slug":"resnext-block","method_name":"ResNeXt Block"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1801.05134","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}