{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/behavior-transformers-cloning-k-modes-with","title":"Behavior Transformers: Cloning $k$ modes with one stone","arxiv_id":"2206.11251","date":"2022-06-22","proceeding":null,"authors":["Nur Muhammad Mahi Shafiullah","Zichen Jeff Cui","Ariuntuya Altanzaya","Lerrel Pinto"],"abstract":"While behavior learning has made impressive progress in recent times, it lags behind computer vision and natural language processing due to its inability to leverage large, human-generated datasets. Human behaviors have wide variance, multiple modes, and human demonstrations typically do not come with reward labels. These properties limit the applicability of current methods in Offline RL and Behavioral Cloning to learn from large, pre-collected datasets. In this work, we present Behavior Transformer (BeT), a new technique to model unlabeled demonstration data with multiple modes. BeT retrofits standard transformer architectures with action discretization coupled with a multi-task action correction inspired by offset prediction in object detection. This allows us to leverage the multi-modal modeling ability of modern transformers to predict multi-modal continuous actions. We experimentally evaluate BeT on a variety of robotic manipulation and self-driving behavior datasets. We show that BeT significantly improves over prior state-of-the-art work on solving demonstrated tasks while capturing the major modes present in the pre-collected datasets. Finally, through an extensive ablation study, we analyze the importance of every crucial component in BeT. Videos of behavior generated by BeT are available at https://notmahi.github.io/bet","url_abs":"https://arxiv.org/abs/2206.11251v2","url_pdf":"https://arxiv.org/pdf/2206.11251v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"behavior-transformers-cloning-k-modes-with","repo_url":"https://github.com/notmahi/bet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"behavior-transformers-cloning-k-modes-with","repo_url":"https://github.com/skandermoalla/bet-reproduction","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"offline-rl","task_name":"Offline RL"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.11251","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.11251"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/skandermoalla/bet-reproduction","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/notmahi/bet","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1},"listed":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f840d01c761c3844","entry":"focal_loss","repo":"notmahi/bet","repo_kind":"official","path":"models/libraries/loss_fn.py","file_url":"https://github.com/notmahi/bet/blob/HEAD/models/libraries/loss_fn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f840d01c761c3844"}},{"code_sha256_prefix":"91252b98bc18bdc0","entry":"get_environment_name","repo":"skandermoalla/bet-reproduction","repo_kind":"listed","path":"compute_metrics.py","file_url":"https://github.com/skandermoalla/bet-reproduction/blob/HEAD/compute_metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91252b98bc18bdc0"}},{"code_sha256_prefix":"0055bdefa26803c4","entry":"get_train_and_eval_log","repo":"skandermoalla/bet-reproduction","repo_kind":"listed","path":"compute_metrics.py","file_url":"https://github.com/skandermoalla/bet-reproduction/blob/HEAD/compute_metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0055bdefa26803c4"}},{"code_sha256_prefix":"aee85acbd82e6f57","entry":"mlp","repo":"notmahi/bet","repo_kind":"official","path":"models/mlp.py","file_url":"https://github.com/notmahi/bet/blob/HEAD/models/mlp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aee85acbd82e6f57"}},{"code_sha256_prefix":"15820c0b4be13799","entry":"mlp","repo":"skandermoalla/bet-reproduction","repo_kind":"listed","path":"models/mlp.py","file_url":"https://github.com/skandermoalla/bet-reproduction/blob/HEAD/models/mlp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15820c0b4be13799"}},{"code_sha256_prefix":"a468c0a9de16ef38","entry":"process_cv_run_idxs","repo":"skandermoalla/bet-reproduction","repo_kind":"listed","path":"run_on_env.py","file_url":"https://github.com/skandermoalla/bet-reproduction/blob/HEAD/run_on_env.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a468c0a9de16ef38"}},{"code_sha256_prefix":"8de152f617bedf07","entry":"process_tags","repo":"skandermoalla/bet-reproduction","repo_kind":"listed","path":"compute_metrics.py","file_url":"https://github.com/skandermoalla/bet-reproduction/blob/HEAD/compute_metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8de152f617bedf07"}},{"code_sha256_prefix":"1fdf23b4c59d9dd5","entry":"random_noise","repo":"notmahi/bet","repo_kind":"official","path":"dataloaders/augmentations.py","file_url":"https://github.com/notmahi/bet/blob/HEAD/dataloaders/augmentations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1fdf23b4c59d9dd5"}},{"code_sha256_prefix":"94dcf41e723717c4","entry":"random_noise_affine","repo":"notmahi/bet","repo_kind":"official","path":"dataloaders/augmentations.py","file_url":"https://github.com/notmahi/bet/blob/HEAD/dataloaders/augmentations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94dcf41e723717c4"}},{"code_sha256_prefix":"5c2032b53ba9edf4","entry":"soft_cross_entropy","repo":"notmahi/bet","repo_kind":"official","path":"models/libraries/loss_fn.py","file_url":"https://github.com/notmahi/bet/blob/HEAD/models/libraries/loss_fn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c2032b53ba9edf4"}},{"code_sha256_prefix":"9e05fa1b88b3794c","entry":"soft_cross_entropy","repo":"skandermoalla/bet-reproduction","repo_kind":"listed","path":"models/libraries/loss_fn.py","file_url":"https://github.com/skandermoalla/bet-reproduction/blob/HEAD/models/libraries/loss_fn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e05fa1b88b3794c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}