{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/real-time-reinforcement-learning-for-vision","title":"Real-Time Reinforcement Learning for Vision-Based Robotics Utilizing Local and Remote Computers","arxiv_id":"2210.02317","date":"2022-10-05","proceeding":null,"authors":["Yan Wang","Gautham Vasan","A. Rupam Mahmood"],"abstract":"Real-time learning is crucial for robotic agents adapting to ever-changing, non-stationary environments. A common setup for a robotic agent is to have two different computers simultaneously: a resource-limited local computer tethered to the robot and a powerful remote computer connected wirelessly. Given such a setup, it is unclear to what extent the performance of a learning system can be affected by resource limitations and how to efficiently use the wirelessly connected powerful computer to compensate for any performance loss. In this paper, we implement a real-time learning system called the Remote-Local Distributed (ReLoD) system to distribute computations of two deep reinforcement learning (RL) algorithms, Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO), between a local and a remote computer. The performance of the system is evaluated on two vision-based control tasks developed using a robotic arm and a mobile robot. Our results show that SAC's performance degrades heavily on a resource-limited local computer. Strikingly, when all computations of the learning system are deployed on a remote workstation, SAC fails to compensate for the performance loss, indicating that, without careful consideration, using a powerful remote computer may not result in performance improvement. However, a carefully chosen distribution of computations of SAC consistently and substantially improves its performance on both tasks. On the other hand, the performance of PPO remains largely unaffected by the distribution of computations. In addition, when all computations happen solely on a powerful tethered computer, the performance of our system remains on par with an existing system that is well-tuned for using a single machine. ReLoD is the only publicly available system for real-time RL that applies to multiple robots for vision-based tasks.","url_abs":"https://arxiv.org/abs/2210.02317v2","url_pdf":"https://arxiv.org/pdf/2210.02317v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"real-time-reinforcement-learning-for-vision","repo_url":"https://github.com/rlai-lab/relod","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"real-time-reinforcement-learning-for-vision","repo_url":"https://github.com/rlai-lab/remote-onboard-agent","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dilated-convolution","method_name":"Dilated Convolution"},{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"ppo","method_name":"PPO"},{"method_slug":"sac","method_name":"SAC"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.02317","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.02317"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rlai-lab/relod","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rlai-lab/remote-onboard-agent","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"49d8acf6f75e2e11","entry":"human_format_numbers","repo":"rlai-lab/remote-onboard-agent","repo_kind":"official","path":"plotter.py","file_url":"https://github.com/rlai-lab/remote-onboard-agent/blob/HEAD/plotter.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"49d8acf6f75e2e11"}},{"code_sha256_prefix":"5896913efd950a77","entry":"recv_message","repo":"rlai-lab/relod","repo_kind":"official","path":"relod/algo/comm.py","file_url":"https://github.com/rlai-lab/relod/blob/HEAD/relod/algo/comm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5896913efd950a77"}},{"code_sha256_prefix":"9c6d7b75698b8c70","entry":"get_all_plot_rets","repo":"rlai-lab/remote-onboard-agent","repo_kind":"official","path":"plotter.py","file_url":"https://github.com/rlai-lab/remote-onboard-agent/blob/HEAD/plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c6d7b75698b8c70"}},{"code_sha256_prefix":"9cc852af909e24c1","entry":"tick_function","repo":"rlai-lab/remote-onboard-agent","repo_kind":"official","path":"plotter.py","file_url":"https://github.com/rlai-lab/remote-onboard-agent/blob/HEAD/plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9cc852af909e24c1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}