{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tunetables-context-optimization-for-scalable","title":"TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks","arxiv_id":"2402.11137","date":"2024-02-17","proceeding":null,"authors":["Benjamin Feuer","Robin Tibor Schirrmeister","Valeriia Cherepanova","Chinmay Hegde","Frank Hutter","Micah Goldblum","Niv Cohen","Colin White"],"abstract":"While tabular classification has traditionally relied on from-scratch training, a recent breakthrough called prior-data fitted networks (PFNs) challenges this approach. Similar to large language models, PFNs make use of pretraining and in-context learning to achieve strong performance on new tasks in a single forward pass. However, current PFNs have limitations that prohibit their widespread adoption. Notably, TabPFN achieves very strong performance on small tabular datasets but is not designed to make predictions for datasets of size larger than 1000. In this work, we overcome these limitations and substantially improve the performance of PFNs via context optimization. We introduce TuneTables, a parameter-efficient fine-tuning strategy for PFNs that compresses large datasets into a smaller learned context. We conduct extensive experiments on 19 algorithms over 98 datasets and find that TuneTables achieves the best performance on average, outperforming boosted trees such as CatBoost, while optimizing fewer than 5% of TabPFN's parameters. Furthermore, we show that TuneTables can be used as an interpretability tool and can even be used to mitigate biases by optimizing a fairness objective. We open-source our code and raw results at https://github.com/penfever/TuneTables.","url_abs":"https://arxiv.org/abs/2402.11137v3","url_pdf":"https://arxiv.org/pdf/2402.11137v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tunetables-context-optimization-for-scalable","repo_url":"https://github.com/penfever/tabpfn-pt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"tunetables-context-optimization-for-scalable","repo_url":"https://github.com/penfever/tunetables","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"fairness","task_name":"Fairness"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"},{"task_slug":"tabular-classification","task_name":"tabular-classification"}],"methods":[{"method_slug":"tabpfn","method_name":"TABPFN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.11137","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.11137"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/penfever/TuneTables","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/penfever/tabpfn-pt","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/penfever/tunetables","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_violates":1,"ran":6,"ran_draft_wrong":3,"unverified":7},"by_repo_kind":{"official":{"samples":17,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a5c44847d0bc55c2","entry":"is_json_serializable","repo":"penfever/tunetables","repo_kind":"official","path":"tunetables/batch/run_tt_job.py","file_url":"https://github.com/penfever/tunetables/blob/HEAD/tunetables/batch/run_tt_job.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a5c44847d0bc55c2"}},{"code_sha256_prefix":"890fb6ef88c9dbef","entry":"JointBCELossWithLogits","repo":"penfever/TuneTables","repo_kind":"official","path":"tunetables/losses.py","file_url":"https://github.com/penfever/TuneTables/blob/HEAD/tunetables/losses.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"890fb6ef88c9dbef"}},{"code_sha256_prefix":"939b599ed6041748","entry":"get_append_embedding_encoder","repo":"penfever/TuneTables","repo_kind":"official","path":"tunetables/encoders.py","file_url":"https://github.com/penfever/TuneTables/blob/HEAD/tunetables/encoders.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"939b599ed6041748"}},{"code_sha256_prefix":"39f5dc76953cf6db","entry":"get_normalized_encoder","repo":"penfever/TuneTables","repo_kind":"official","path":"tunetables/encoders.py","file_url":"https://github.com/penfever/TuneTables/blob/HEAD/tunetables/encoders.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"39f5dc76953cf6db"}},{"code_sha256_prefix":"b91cdb6ff3f535f8","entry":"get_wandb_api_key","repo":"penfever/tunetables","repo_kind":"official","path":"tunetables/batch/run_tt_job.py","file_url":"https://github.com/penfever/tunetables/blob/HEAD/tunetables/batch/run_tt_job.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b91cdb6ff3f535f8"}},{"code_sha256_prefix":"98d35d058dcc75c7","entry":"kl_divergence","repo":"penfever/TuneTables","repo_kind":"official","path":"tunetables/losses.py","file_url":"https://github.com/penfever/TuneTables/blob/HEAD/tunetables/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"98d35d058dcc75c7"}},{"code_sha256_prefix":"8022d5417a746548","entry":"make_serializable","repo":"penfever/tunetables","repo_kind":"official","path":"tunetables/batch/run_tt_job.py","file_url":"https://github.com/penfever/tunetables/blob/HEAD/tunetables/batch/run_tt_job.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8022d5417a746548"}},{"code_sha256_prefix":"ab1f1da564c5ae70","entry":"set_compatibility_params","repo":"penfever/TuneTables","repo_kind":"official","path":"tunetables/train_loop.py","file_url":"https://github.com/penfever/TuneTables/blob/HEAD/tunetables/train_loop.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ab1f1da564c5ae70"}},{"code_sha256_prefix":"7646a622a583702c","entry":"taylor_linear_attention_forward","repo":"penfever/tunetables","repo_kind":"official","path":"tunetables/transformer.py","file_url":"https://github.com/penfever/tunetables/blob/HEAD/tunetables/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7646a622a583702c"}},{"code_sha256_prefix":"8b70d1a53aec77a8","entry":"taylor_linear_attention_forward","repo":"penfever/TuneTables","repo_kind":"official","path":"tunetables/layer.py","file_url":"https://github.com/penfever/TuneTables/blob/HEAD/tunetables/layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b70d1a53aec77a8"}},{"code_sha256_prefix":"79f5b9e40c967c9b","entry":"TaylorMultiheadAttention","repo":"penfever/tunetables","repo_kind":"official","path":"tunetables/transformer.py","file_url":"https://github.com/penfever/tunetables/blob/HEAD/tunetables/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"79f5b9e40c967c9b"}},{"code_sha256_prefix":"3ca52e651c5bddce","entry":"TransformerEncoderLayer","repo":"penfever/tunetables","repo_kind":"official","path":"tunetables/transformer.py","file_url":"https://github.com/penfever/tunetables/blob/HEAD/tunetables/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ca52e651c5bddce"}},{"code_sha256_prefix":"3b298b0945e919bb","entry":"TransformerModel","repo":"penfever/tunetables","repo_kind":"official","path":"tunetables/transformer.py","file_url":"https://github.com/penfever/tunetables/blob/HEAD/tunetables/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3b298b0945e919bb"}},{"code_sha256_prefix":"9e7eb2a909a1fb2d","entry":"eval_method","repo":"penfever/TuneTables","repo_kind":"official","path":"tunetables/run_baselines.py","file_url":"https://github.com/penfever/TuneTables/blob/HEAD/tunetables/run_baselines.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9e7eb2a909a1fb2d"}},{"code_sha256_prefix":"4c84ce37831d7194","entry":"get_NormalInitializer","repo":"penfever/TuneTables","repo_kind":"official","path":"tunetables/initializers.py","file_url":"https://github.com/penfever/TuneTables/blob/HEAD/tunetables/initializers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4c84ce37831d7194"}},{"code_sha256_prefix":"16eab2bd7416886c","entry":"get_normalized_uniform_encoder","repo":"penfever/TuneTables","repo_kind":"official","path":"tunetables/encoders.py","file_url":"https://github.com/penfever/TuneTables/blob/HEAD/tunetables/encoders.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"16eab2bd7416886c"}},{"code_sha256_prefix":"68a73e2ea3cbee4f","entry":"reload_config","repo":"penfever/TuneTables","repo_kind":"official","path":"tunetables/train_loop.py","file_url":"https://github.com/penfever/TuneTables/blob/HEAD/tunetables/train_loop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68a73e2ea3cbee4f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}