{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/non-convex-learning-via-replica-exchange","title":"Non-convex Learning via Replica Exchange Stochastic Gradient MCMC","arxiv_id":"2008.05367","date":"2020-08-12","proceeding":"ICML 2020 1","authors":["Wei Deng","Qi Feng","Liyao Gao","Faming Liang","Guang Lin"],"abstract":"Replica exchange Monte Carlo (reMC), also known as parallel tempering, is an important technique for accelerating the convergence of the conventional Markov Chain Monte Carlo (MCMC) algorithms. However, such a method requires the evaluation of the energy function based on the full dataset and is not scalable to big data. The na\\\"ive implementation of reMC in mini-batch settings introduces large biases, which cannot be directly extended to the stochastic gradient MCMC (SGMCMC), the standard sampling method for simulating from deep neural networks (DNNs). In this paper, we propose an adaptive replica exchange SGMCMC (reSGMCMC) to automatically correct the bias and study the corresponding properties. The analysis implies an acceleration-accuracy trade-off in the numerical discretization of a Markov jump process in a stochastic environment. Empirically, we test the algorithm through extensive experiments on various setups and obtain the state-of-the-art results on CIFAR10, CIFAR100, and SVHN in both supervised learning and semi-supervised learning tasks.","url_abs":"https://arxiv.org/abs/2008.05367v3","url_pdf":"https://arxiv.org/pdf/2008.05367v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"non-convex-learning-via-replica-exchange","repo_url":"https://github.com/gaoliyao/Replica_Exchange_Stochastic_Gradient_MCMC","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"non-convex-learning-via-replica-exchange","repo_url":"https://github.com/WayneDW/Variance_Reduced_Replica_Exchange_SGMCMC","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"}],"methods":[{"method_slug":"resgld","method_name":"reSGLD"}],"datasets_introduced":[],"methods_introduced":[{"slug":"resgld","name":"reSGLD","full_name":"Replica exchange stochastic gradient Langevin Dynamics"}],"results":[{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"WRN-28-10 with reSGHMC","rank_in_archive_order":84,"of":265,"metrics":{"Percentage correct":"97.42"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"WRN-16-8 with reSGHMC","rank_in_archive_order":95,"of":265,"metrics":{"Percentage correct":"96.87"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"ResNet56 with reSGHMC","rank_in_archive_order":119,"of":265,"metrics":{"Percentage correct":"96.12"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"ResNet32 with reSGHMC","rank_in_archive_order":136,"of":265,"metrics":{"Percentage correct":"95.35"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"ResNet20 with reSGHMC","rank_in_archive_order":151,"of":265,"metrics":{"Percentage correct":"94.62"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"WRN-28-10 with reSGHMC","rank_in_archive_order":77,"of":211,"metrics":{"Percentage correct":"84.38"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"WRN-16-8 with reSGHMC","rank_in_archive_order":95,"of":211,"metrics":{"Percentage correct":"82.95"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"ResNet56 with reSGHMC","rank_in_archive_order":132,"of":211,"metrics":{"Percentage correct":"80.14"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"ResNet32 with reSGHMC","rank_in_archive_order":148,"of":211,"metrics":{"Percentage correct":"76.55"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"ResNet20 with reSGHMC","rank_in_archive_order":156,"of":211,"metrics":{"Percentage correct":"74.14"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2008.05367","atlas_url":"https://app.syntology.ai/?focus=2008.05367","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2008.05367"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/WayneDW/Variance_Reduced_Replica_Exchange_SGMCMC","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gaoliyao/Replica_Exchange_Stochastic_Gradient_MCMC","reach":null}],"summary":{"ran":2},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"bd03d320ee349a01","entry":"Sampler","repo":"WayneDW/Variance_Reduced_Replica_Exchange_SGMCMC","repo_kind":"listed","path":"sgmcmc.py","file_url":"https://github.com/WayneDW/Variance_Reduced_Replica_Exchange_SGMCMC/blob/HEAD/sgmcmc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd03d320ee349a01"}},{"code_sha256_prefix":"cf36f84e0e7fbcd1","entry":"Sampler","repo":"gaoliyao/Replica_Exchange_Stochastic_Gradient_MCMC","repo_kind":"official","path":"sgmcmc.py","file_url":"https://github.com/gaoliyao/Replica_Exchange_Stochastic_Gradient_MCMC/blob/HEAD/sgmcmc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf36f84e0e7fbcd1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}