{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deep-reinforcement-learning-with-population","title":"Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control","arxiv_id":"2010.09635","date":"2020-10-19","proceeding":null,"authors":["Guangzhi Tang","Neelesh Kumar","Raymond Yoo","Konstantinos P. Michmizos"],"abstract":"The energy-efficient control of mobile robots is crucial as the complexity of their real-world applications increasingly involves high-dimensional observation and action spaces, which cannot be offset by limited on-board resources. An emerging non-Von Neumann model of intelligence, where spiking neural networks (SNNs) are run on neuromorphic processors, is regarded as an energy-efficient and robust alternative to the state-of-the-art real-time robotic controllers for low dimensional control tasks. The challenge now for this new computing paradigm is to scale so that it can keep up with real-world tasks. To do so, SNNs need to overcome the inherent limitations of their training, namely the limited ability of their spiking neurons to represent information and the lack of effective learning algorithms. Here, we propose a population-coded spiking actor network (PopSAN) trained in conjunction with a deep critic network using deep reinforcement learning (DRL). The population coding scheme dramatically increased the representation capacity of the network and the hybrid learning combined the training advantages of deep networks with the energy-efficient inference of spiking networks. To show the general applicability of our approach, we integrated it with a spectrum of both on-policy and off-policy DRL algorithms. We deployed the trained PopSAN on Intel's Loihi neuromorphic chip and benchmarked our method against the mainstream DRL algorithms for continuous control. To allow for a fair comparison among all methods, we validated them on OpenAI gym tasks. Our Loihi-run PopSAN consumed 140 times less energy per inference when compared against the deep actor network on Jetson TX2, and had the same level of performance. Our results support the efficiency of neuromorphic controllers and suggest our hybrid RL as an alternative to deep learning, when both energy-efficiency and robustness are important.","url_abs":"https://arxiv.org/abs/2010.09635v1","url_pdf":"https://arxiv.org/pdf/2010.09635v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deep-reinforcement-learning-with-population","repo_url":"https://github.com/combra-lab/pop-spiking-deep-rl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"openai-gym","task_name":"OpenAI Gym"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"continuous-control","task_name":"continuous-control"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2010.09635","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.09635"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/combra-lab/pop-spiking-deep-rl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":1,"unverified":6},"by_repo_kind":{"official":{"samples":9,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"57542f87e248db45","entry":"combined_shape","repo":"combra-lab/pop-spiking-deep-rl","repo_kind":"official","path":"popsan_drl/popsan_sac/core_cuda.py","file_url":"https://github.com/combra-lab/pop-spiking-deep-rl/blob/HEAD/popsan_drl/popsan_sac/core_cuda.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57542f87e248db45"}},{"code_sha256_prefix":"4cca5e460067b900","entry":"count_vars","repo":"combra-lab/pop-spiking-deep-rl","repo_kind":"official","path":"popsan_drl/popsan_sac/core_cuda.py","file_url":"https://github.com/combra-lab/pop-spiking-deep-rl/blob/HEAD/popsan_drl/popsan_sac/core_cuda.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cca5e460067b900"}},{"code_sha256_prefix":"7569e2a512ec417c","entry":"mlp","repo":"combra-lab/pop-spiking-deep-rl","repo_kind":"official","path":"popsan_drl/popsan_sac/core_cuda.py","file_url":"https://github.com/combra-lab/pop-spiking-deep-rl/blob/HEAD/popsan_drl/popsan_sac/core_cuda.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7569e2a512ec417c"}},{"code_sha256_prefix":"deb05398b707eadd","entry":"combine_multiple_into_one_int","repo":"combra-lab/pop-spiking-deep-rl","repo_kind":"official","path":"loihi_realization/utility.py","file_url":"https://github.com/combra-lab/pop-spiking-deep-rl/blob/HEAD/loihi_realization/utility.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"deb05398b707eadd"}},{"code_sha256_prefix":"c2985d7efcd097a6","entry":"make_env","repo":"combra-lab/pop-spiking-deep-rl","repo_kind":"official","path":"popsan_drl/popsan_ppo/ppo_cuda_norm.py","file_url":"https://github.com/combra-lab/pop-spiking-deep-rl/blob/HEAD/popsan_drl/popsan_ppo/ppo_cuda_norm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2985d7efcd097a6"}},{"code_sha256_prefix":"5e58beaffd34b5c3","entry":"pytorch_trained_snn_param_2_loihi_snn_param","repo":"combra-lab/pop-spiking-deep-rl","repo_kind":"official","path":"loihi_realization/utility.py","file_url":"https://github.com/combra-lab/pop-spiking-deep-rl/blob/HEAD/loihi_realization/utility.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e58beaffd34b5c3"}},{"code_sha256_prefix":"1e2945be1d8c5361","entry":"read_pytorch_network_parameters_4_loihi","repo":"combra-lab/pop-spiking-deep-rl","repo_kind":"official","path":"loihi_realization/utility.py","file_url":"https://github.com/combra-lab/pop-spiking-deep-rl/blob/HEAD/loihi_realization/utility.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e2945be1d8c5361"}},{"code_sha256_prefix":"760558af7c4999e1","entry":"test_env","repo":"combra-lab/pop-spiking-deep-rl","repo_kind":"official","path":"popsan_drl/popsan_ppo/ppo_cuda_norm.py","file_url":"https://github.com/combra-lab/pop-spiking-deep-rl/blob/HEAD/popsan_drl/popsan_ppo/ppo_cuda_norm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"760558af7c4999e1"}},{"code_sha256_prefix":"9e151424d7e23786","entry":"train_mujoco_ppo","repo":"combra-lab/pop-spiking-deep-rl","repo_kind":"official","path":"popsan_drl/popsan_ppo/ppo_cuda_norm.py","file_url":"https://github.com/combra-lab/pop-spiking-deep-rl/blob/HEAD/popsan_drl/popsan_ppo/ppo_cuda_norm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e151424d7e23786"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}