{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/train-offline-test-online-a-real-robot","title":"Train Offline, Test Online: A Real Robot Learning Benchmark","arxiv_id":"2306.00942","date":"2023-06-01","proceeding":null,"authors":["Gaoyue Zhou","Victoria Dean","Mohan Kumar Srirama","Aravind Rajeswaran","Jyothish Pari","Kyle Hatch","Aryan Jain","Tianhe Yu","Pieter Abbeel","Lerrel Pinto","Chelsea Finn","Abhinav Gupta"],"abstract":"Three challenges limit the progress of robot learning research: robots are expensive (few labs can participate), everyone uses different robots (findings do not generalize across labs), and we lack internet-scale robotics data. We take on these challenges via a new benchmark: Train Offline, Test Online (TOTO). TOTO provides remote users with access to shared robotic hardware for evaluating methods on common tasks and an open-source dataset of these tasks for offline training. Its manipulation task suite requires challenging generalization to unseen objects, positions, and lighting. We present initial results on TOTO comparing five pretrained visual representations and four offline policy learning baselines, remotely contributed by five institutions. The real promise of TOTO, however, lies in the future: we release the benchmark for additional submissions from any user, enabling easy, direct comparison to several methods without the need to obtain hardware or collect data.","url_abs":"https://arxiv.org/abs/2306.00942v2","url_pdf":"https://arxiv.org/pdf/2306.00942v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"train-offline-test-online-a-real-robot","repo_url":"https://github.com/AGI-Labs/toto_benchmark","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"tasks":[],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2306.00942","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.00942"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AGI-Labs/toto_benchmark","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1263c50ebe6172a3","entry":"compute_accuracy","repo":"AGI-Labs/toto_benchmark","repo_kind":"official","path":"toto_benchmark/vision/pvr_model_training.py","file_url":"https://github.com/AGI-Labs/toto_benchmark/blob/HEAD/toto_benchmark/vision/pvr_model_training.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1263c50ebe6172a3"}},{"code_sha256_prefix":"62608541281b5cad","entry":"get_stats","repo":"AGI-Labs/toto_benchmark","repo_kind":"official","path":"toto_benchmark/agents/BCAgent.py","file_url":"https://github.com/AGI-Labs/toto_benchmark/blob/HEAD/toto_benchmark/agents/BCAgent.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"62608541281b5cad"}},{"code_sha256_prefix":"9b85e2ca522ca849","entry":"get_time_chunk","repo":"AGI-Labs/toto_benchmark","repo_kind":"official","path":"toto_benchmark/vision/pvr_model_training.py","file_url":"https://github.com/AGI-Labs/toto_benchmark/blob/HEAD/toto_benchmark/vision/pvr_model_training.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9b85e2ca522ca849"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}