{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-mamba-learn-how-to-learn-a-comparative","title":"Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks","arxiv_id":"2402.04248","date":"2024-02-06","proceeding":null,"authors":["Jongho Park","Jaeseung Park","Zheyang Xiong","Nayoung Lee","Jaewoong Cho","Samet Oymak","Kangwook Lee","Dimitris Papailiopoulos"],"abstract":"State-space models (SSMs), such as Mamba (Gu & Dao, 2023), have been proposed as alternatives to Transformer networks in language modeling, by incorporating gating, convolutions, and input-dependent token selection to mitigate the quadratic cost of multi-head attention. Although SSMs exhibit competitive performance, their in-context learning (ICL) capabilities, a remarkable emergent property of modern language models that enables task execution without parameter optimization, remain underexplored compared to Transformers. In this study, we evaluate the ICL performance of SSMs, focusing on Mamba, against Transformer models across various tasks. Our results show that SSMs perform comparably to Transformers in standard regression ICL tasks, while outperforming them in tasks like sparse parity learning. However, SSMs fall short in tasks involving non-standard retrieval functionality. To address these limitations, we introduce a hybrid model, MambaFormer, that combines Mamba with attention blocks, surpassing individual models in tasks where they struggle independently. Our findings suggest that hybrid architectures offer promising avenues for enhancing ICL in language models.","url_abs":"https://arxiv.org/abs/2402.04248v2","url_pdf":"https://arxiv.org/pdf/2402.04248v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-mamba-learn-how-to-learn-a-comparative","repo_url":"https://github.com/krafton-ai/mambaformer-icl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"can-mamba-learn-how-to-learn-a-comparative","repo_url":"https://github.com/hazyresearch/based","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"mamba","task_name":"Mamba"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"state-space-models","task_name":"State Space Models"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.04248","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.04248"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/krafton-ai/mambaformer-icl","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hazyresearch/based","reach":null}],"summary":{"ran":8,"ran_violates":1,"unverified":5},"by_repo_kind":{"official":{"samples":14,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5b74687909baf467","entry":"basic_plot","repo":"krafton-ai/mambaformer-icl","repo_kind":"official","path":"src/plot_utils.py","file_url":"https://github.com/krafton-ai/mambaformer-icl/blob/HEAD/src/plot_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5b74687909baf467"}},{"code_sha256_prefix":"b1c1ac424117b8de","entry":"generate_dummy_vector","repo":"krafton-ai/mambaformer-icl","repo_kind":"official","path":"src/filtering_utils.py","file_url":"https://github.com/krafton-ai/mambaformer-icl/blob/HEAD/src/filtering_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b1c1ac424117b8de"}},{"code_sha256_prefix":"fc94f27a0973e093","entry":"mean_squared_error","repo":"krafton-ai/mambaformer-icl","repo_kind":"official","path":"src/tasks.py","file_url":"https://github.com/krafton-ai/mambaformer-icl/blob/HEAD/src/tasks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fc94f27a0973e093"}},{"code_sha256_prefix":"a46d1c2a317b35c5","entry":"orthogonalize","repo":"krafton-ai/mambaformer-icl","repo_kind":"official","path":"src/filtering_utils.py","file_url":"https://github.com/krafton-ai/mambaformer-icl/blob/HEAD/src/filtering_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a46d1c2a317b35c5"}},{"code_sha256_prefix":"348d63a78eb4cb94","entry":"squared_error","repo":"krafton-ai/mambaformer-icl","repo_kind":"official","path":"src/tasks.py","file_url":"https://github.com/krafton-ai/mambaformer-icl/blob/HEAD/src/tasks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"348d63a78eb4cb94"}},{"code_sha256_prefix":"b4326d9d300aa5c0","entry":"squared_norm_error","repo":"krafton-ai/mambaformer-icl","repo_kind":"official","path":"src/tasks.py","file_url":"https://github.com/krafton-ai/mambaformer-icl/blob/HEAD/src/tasks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b4326d9d300aa5c0"}},{"code_sha256_prefix":"8605dc8a088f3db8","entry":"str2bool","repo":"krafton-ai/mambaformer-icl","repo_kind":"official","path":"src/util.py","file_url":"https://github.com/krafton-ai/mambaformer-icl/blob/HEAD/src/util.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8605dc8a088f3db8"}},{"code_sha256_prefix":"47f61861b53edf4d","entry":"train_step","repo":"krafton-ai/mambaformer-icl","repo_kind":"official","path":"src/train_parallel.py","file_url":"https://github.com/krafton-ai/mambaformer-icl/blob/HEAD/src/train_parallel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"47f61861b53edf4d"}},{"code_sha256_prefix":"75e826964a91f44b","entry":"zero_pad_ys","repo":"krafton-ai/mambaformer-icl","repo_kind":"official","path":"src/filtering_utils.py","file_url":"https://github.com/krafton-ai/mambaformer-icl/blob/HEAD/src/filtering_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"75e826964a91f44b"}},{"code_sha256_prefix":"54f3966e15e1afe1","entry":"get_data_sampler","repo":"krafton-ai/mambaformer-icl","repo_kind":"official","path":"src/samplers.py","file_url":"https://github.com/krafton-ai/mambaformer-icl/blob/HEAD/src/samplers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54f3966e15e1afe1"}},{"code_sha256_prefix":"3c24ac252fef0868","entry":"get_final_var","repo":"krafton-ai/mambaformer-icl","repo_kind":"official","path":"src/curriculum.py","file_url":"https://github.com/krafton-ai/mambaformer-icl/blob/HEAD/src/curriculum.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c24ac252fef0868"}},{"code_sha256_prefix":"ddd35c92ba9c2871","entry":"randperm_2d_with_query","repo":"krafton-ai/mambaformer-icl","repo_kind":"official","path":"src/samplers.py","file_url":"https://github.com/krafton-ai/mambaformer-icl/blob/HEAD/src/samplers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ddd35c92ba9c2871"}},{"code_sha256_prefix":"4d49c710f74c99c4","entry":"sample_seeds","repo":"krafton-ai/mambaformer-icl","repo_kind":"official","path":"src/train_parallel.py","file_url":"https://github.com/krafton-ai/mambaformer-icl/blob/HEAD/src/train_parallel.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4d49c710f74c99c4"}},{"code_sha256_prefix":"7f92ad21604fa243","entry":"sample_transformation","repo":"krafton-ai/mambaformer-icl","repo_kind":"official","path":"src/samplers.py","file_url":"https://github.com/krafton-ai/mambaformer-icl/blob/HEAD/src/samplers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f92ad21604fa243"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}