{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/transformers-as-decision-makers-provable-in","title":"Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining","arxiv_id":"2310.08566","date":"2023-10-12","proceeding":null,"authors":["Licong Lin","Yu Bai","Song Mei"],"abstract":"Large transformer models pretrained on offline reinforcement learning datasets have demonstrated remarkable in-context reinforcement learning (ICRL) capabilities, where they can make good decisions when prompted with interaction trajectories from unseen environments. However, when and how transformers can be trained to perform ICRL have not been theoretically well-understood. In particular, it is unclear which reinforcement-learning algorithms transformers can perform in context, and how distribution mismatch in offline training data affects the learned algorithms. This paper provides a theoretical framework that analyzes supervised pretraining for ICRL. This includes two recently proposed training methods -- algorithm distillation and decision-pretrained transformers. First, assuming model realizability, we prove the supervised-pretrained transformer will imitate the conditional expectation of the expert algorithm given the observed trajectory. The generalization error will scale with model capacity and a distribution divergence factor between the expert and offline algorithms. Second, we show transformers with ReLU attention can efficiently approximate near-optimal online reinforcement learning algorithms like LinUCB and Thompson sampling for stochastic linear bandits, and UCB-VI for tabular Markov decision processes. This provides the first quantitative analysis of the ICRL capabilities of transformers pretrained from offline trajectories.","url_abs":"https://arxiv.org/abs/2310.08566v2","url_pdf":"https://arxiv.org/pdf/2310.08566v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"transformers-as-decision-makers-provable-in","repo_url":"https://github.com/licong-lin/in-context-rl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"In-Context Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"thompson-sampling","task_name":"Thompson Sampling"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.08566","atlas_url":"https://app.syntology.ai/?focus=2310.08566","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.08566"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/licong-lin/in-context-rl","reach":{"status":"ok"}}],"summary":{"ran":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"860403435e433013","entry":"build_bandit_data_filename","repo":"licong-lin/in-context-rl","repo_kind":"official","path":"utils.py","file_url":"https://github.com/licong-lin/in-context-rl/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"860403435e433013"}},{"code_sha256_prefix":"57e03f5bed079c4d","entry":"build_bandit_model_filename","repo":"licong-lin/in-context-rl","repo_kind":"official","path":"utils.py","file_url":"https://github.com/licong-lin/in-context-rl/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57e03f5bed079c4d"}},{"code_sha256_prefix":"415deb88e0bd17be","entry":"convert_to_tensor","repo":"licong-lin/in-context-rl","repo_kind":"official","path":"utils.py","file_url":"https://github.com/licong-lin/in-context-rl/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"415deb88e0bd17be"}},{"code_sha256_prefix":"9a577e75611ac157","entry":"deploy_online_vec","repo":"licong-lin/in-context-rl","repo_kind":"official","path":"evals/eval_bandit.py","file_url":"https://github.com/licong-lin/in-context-rl/blob/HEAD/evals/eval_bandit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a577e75611ac157"}},{"code_sha256_prefix":"75df7d35faaa9ff0","entry":"get_activation","repo":"licong-lin/in-context-rl","repo_kind":"official","path":"net.py","file_url":"https://github.com/licong-lin/in-context-rl/blob/HEAD/net.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"75df7d35faaa9ff0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}