{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/large-scale-transfer-learning-for-tabular","title":"Large Scale Transfer Learning for Tabular Data via Language Modeling","arxiv_id":"2406.12031","date":"2024-06-17","proceeding":null,"authors":["Josh Gardner","Juan C. Perdomo","Ludwig Schmidt"],"abstract":"Tabular data -- structured, heterogeneous, spreadsheet-style data with rows and columns -- is widely used in practice across many domains. However, while recent foundation models have reduced the need for developing task-specific datasets and predictors in domains such as language modeling and computer vision, this transfer learning paradigm has not had similar impact in the tabular domain. In this work, we seek to narrow this gap and present TabuLa-8B, a language model for tabular prediction. We define a process for extracting a large, high-quality training dataset from the TabLib corpus, proposing methods for tabular data filtering and quality control. Using the resulting dataset, which comprises over 2.1B rows from over 4M unique tables, we fine-tune a Llama 3-8B large language model (LLM) for tabular data prediction (classification and binned regression) using a novel packing and attention scheme for tabular prediction. Through evaluation across a test suite of 329 datasets, we find that TabuLa-8B has zero-shot accuracy on unseen tables that is over 15 percentage points (pp) higher than random guessing, a feat that is not possible with existing state-of-the-art tabular prediction models (e.g. XGBoost, TabPFN). In the few-shot setting (1-32 shots), without any fine-tuning on the target datasets, TabuLa-8B is 5-15 pp more accurate than XGBoost and TabPFN models that are explicitly trained on equal, or even up to 16x more data. We release our model, code, and data along with the publication of this paper.","url_abs":"https://arxiv.org/abs/2406.12031v2","url_pdf":"https://arxiv.org/pdf/2406.12031v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"large-scale-transfer-learning-for-tabular","repo_url":"https://github.com/mlfoundations/rtfm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"large-scale-transfer-learning-for-tabular","repo_url":"https://github.com/mlfoundations/tabliblib","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"llama","method_name":"LLaMA"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"tabpfn","method_name":"TABPFN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.12031","atlas_url":"https://app.syntology.ai/?focus=2406.12031","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.12031"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mlfoundations/tabliblib","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mlfoundations/rtfm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":13,"unverified":2},"by_repo_kind":{"official":{"samples":15,"ran":13,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4beae4af6e4059ec","entry":"add_content_hash","repo":"mlfoundations/tabliblib","repo_kind":"official","path":"tabliblib/mappers.py","file_url":"https://github.com/mlfoundations/tabliblib/blob/HEAD/tabliblib/mappers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4beae4af6e4059ec"}},{"code_sha256_prefix":"5af725e76bd12300","entry":"arrow_to_pandas_safe","repo":"mlfoundations/tabliblib","repo_kind":"official","path":"tabliblib/io.py","file_url":"https://github.com/mlfoundations/tabliblib/blob/HEAD/tabliblib/io.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5af725e76bd12300"}},{"code_sha256_prefix":"1af054b95295710e","entry":"batch_to_xpu","repo":"mlfoundations/rtfm","repo_kind":"official","path":"rtfm/torch_utils.py","file_url":"https://github.com/mlfoundations/rtfm/blob/HEAD/rtfm/torch_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1af054b95295710e"}},{"code_sha256_prefix":"0ad24980bda6f6e4","entry":"drop_duplicates","repo":"mlfoundations/tabliblib","repo_kind":"official","path":"tabliblib/mappers.py","file_url":"https://github.com/mlfoundations/tabliblib/blob/HEAD/tabliblib/mappers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ad24980bda6f6e4"}},{"code_sha256_prefix":"74807ffb0cddac16","entry":"example_ids_to_attention_mask","repo":"mlfoundations/rtfm","repo_kind":"official","path":"rtfm/data.py","file_url":"https://github.com/mlfoundations/rtfm/blob/HEAD/rtfm/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74807ffb0cddac16"}},{"code_sha256_prefix":"4767aa5be418958c","entry":"find_covering_set","repo":"mlfoundations/tabliblib","repo_kind":"official","path":"tabliblib/dedup_utils.py","file_url":"https://github.com/mlfoundations/tabliblib/blob/HEAD/tabliblib/dedup_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4767aa5be418958c"}},{"code_sha256_prefix":"8d749d50f25dd3c2","entry":"generate_random_dates","repo":"mlfoundations/tabliblib","repo_kind":"official","path":"tabliblib/summarizers.py","file_url":"https://github.com/mlfoundations/tabliblib/blob/HEAD/tabliblib/summarizers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d749d50f25dd3c2"}},{"code_sha256_prefix":"cd4c51c2326006e9","entry":"get_categorical_idxs","repo":"mlfoundations/rtfm","repo_kind":"official","path":"rtfm/tree_baselines.py","file_url":"https://github.com/mlfoundations/rtfm/blob/HEAD/rtfm/tree_baselines.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd4c51c2326006e9"}},{"code_sha256_prefix":"ba26941c52b7f85a","entry":"get_latest_checkpoint","repo":"mlfoundations/rtfm","repo_kind":"official","path":"rtfm/utils.py","file_url":"https://github.com/mlfoundations/rtfm/blob/HEAD/rtfm/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba26941c52b7f85a"}},{"code_sha256_prefix":"d718f67e89480b5b","entry":"load_model_from_checkpoint","repo":"mlfoundations/rtfm","repo_kind":"official","path":"rtfm/train_utils.py","file_url":"https://github.com/mlfoundations/rtfm/blob/HEAD/rtfm/train_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d718f67e89480b5b"}},{"code_sha256_prefix":"39954605fabc6757","entry":"path_to_str","repo":"mlfoundations/tabliblib","repo_kind":"official","path":"tabliblib/dedup_utils.py","file_url":"https://github.com/mlfoundations/tabliblib/blob/HEAD/tabliblib/dedup_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39954605fabc6757"}},{"code_sha256_prefix":"02efcf6df251ab61","entry":"sample_columns_if_needed","repo":"mlfoundations/tabliblib","repo_kind":"official","path":"tabliblib/io.py","file_url":"https://github.com/mlfoundations/tabliblib/blob/HEAD/tabliblib/io.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"02efcf6df251ab61"}},{"code_sha256_prefix":"61bc6ff6d93f37c7","entry":"write_arrow_bytes","repo":"mlfoundations/tabliblib","repo_kind":"official","path":"tabliblib/io.py","file_url":"https://github.com/mlfoundations/tabliblib/blob/HEAD/tabliblib/io.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61bc6ff6d93f37c7"}},{"code_sha256_prefix":"5326e55483a6717d","entry":"bool_cols_to_categorical","repo":"mlfoundations/rtfm","repo_kind":"official","path":"rtfm/tree_baselines.py","file_url":"https://github.com/mlfoundations/rtfm/blob/HEAD/rtfm/tree_baselines.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5326e55483a6717d"}},{"code_sha256_prefix":"7d1cac2a511468ae","entry":"prepare_4d_attention_mask","repo":"mlfoundations/rtfm","repo_kind":"official","path":"rtfm/data.py","file_url":"https://github.com/mlfoundations/rtfm/blob/HEAD/rtfm/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d1cac2a511468ae"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}