{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/latte-transfering-llms-latent-level-knowledge","title":"Latte: Transfering LLMs` Latent-level Knowledge for Few-shot Tabular Learning","arxiv_id":"2505.05237","date":"2025-05-08","proceeding":null,"authors":["Ruxue Shi","Hengrui Gu","Hangting Ye","YiWei Dai","Xu Shen","Xin Wang"],"abstract":"Few-shot tabular learning, in which machine learning models are trained with a limited amount of labeled data, provides a cost-effective approach to addressing real-world challenges. The advent of Large Language Models (LLMs) has sparked interest in leveraging their pre-trained knowledge for few-shot tabular learning. Despite promising results, existing approaches either rely on test-time knowledge extraction, which introduces undesirable latency, or text-level knowledge, which leads to unreliable feature engineering. To overcome these limitations, we propose Latte, a training-time knowledge extraction framework that transfers the latent prior knowledge within LLMs to optimize a more generalized downstream model. Latte enables general knowledge-guided downstream tabular learning, facilitating the weighted fusion of information across different feature values while reducing the risk of overfitting to limited labeled data. Furthermore, Latte is compatible with existing unsupervised pre-training paradigms and effectively utilizes available unlabeled samples to overcome the performance limitations imposed by an extremely small labeled dataset. Extensive experiments on various few-shot tabular learning benchmarks demonstrate the superior performance of Latte, establishing it as a state-of-the-art approach in this domain","url_abs":"https://arxiv.org/abs/2505.05237v1","url_pdf":"https://arxiv.org/pdf/2505.05237v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"feature-engineering","task_name":"Feature Engineering"},{"task_slug":"general-knowledge","task_name":"General Knowledge"},{"task_slug":"unsupervised-pre-training","task_name":"Unsupervised Pre-training"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.05237","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.05237"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ruxueshi/Latte","reach":null}],"summary":{"ran":5,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"4a40827557b7dc1b","entry":"LatteCLSToken","repo":"ruxueshi/Latte","repo_kind":"found_in_text","path":"model/SaTE/LatteSaTE.py","file_url":"https://github.com/ruxueshi/Latte/blob/HEAD/model/SaTE/LatteSaTE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4a40827557b7dc1b"}},{"code_sha256_prefix":"57ddd5e088a299ef","entry":"LatteInputEncoder","repo":"ruxueshi/Latte","repo_kind":"found_in_text","path":"model/SaTE/LatteSaTE.py","file_url":"https://github.com/ruxueshi/Latte/blob/HEAD/model/SaTE/LatteSaTE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57ddd5e088a299ef"}},{"code_sha256_prefix":"2dbb71a3f0ca2670","entry":"LatteLinearClassifier","repo":"ruxueshi/Latte","repo_kind":"found_in_text","path":"model/SaTE/LatteSaTE.py","file_url":"https://github.com/ruxueshi/Latte/blob/HEAD/model/SaTE/LatteSaTE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2dbb71a3f0ca2670"}},{"code_sha256_prefix":"3bdb2c5c5f697445","entry":"LatteNumEmbedding","repo":"ruxueshi/Latte","repo_kind":"found_in_text","path":"model/SaTE/LatteSaTE.py","file_url":"https://github.com/ruxueshi/Latte/blob/HEAD/model/SaTE/LatteSaTE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3bdb2c5c5f697445"}},{"code_sha256_prefix":"035248bc0c24aaf7","entry":"LatteTransformerLayer","repo":"ruxueshi/Latte","repo_kind":"found_in_text","path":"model/SaTE/LatteSaTE.py","file_url":"https://github.com/ruxueshi/Latte/blob/HEAD/model/SaTE/LatteSaTE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"035248bc0c24aaf7"}},{"code_sha256_prefix":"3f30ed9075fa8d0b","entry":"LatteEncoder","repo":"ruxueshi/Latte","repo_kind":"found_in_text","path":"model/SaTE/LatteSaTE.py","file_url":"https://github.com/ruxueshi/Latte/blob/HEAD/model/SaTE/LatteSaTE.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f30ed9075fa8d0b"}},{"code_sha256_prefix":"20452a179e2267bf","entry":"LatteFeatureExtractor","repo":"ruxueshi/Latte","repo_kind":"found_in_text","path":"model/SaTE/LatteSaTE.py","file_url":"https://github.com/ruxueshi/Latte/blob/HEAD/model/SaTE/LatteSaTE.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20452a179e2267bf"}},{"code_sha256_prefix":"eb65bc8acaad9d41","entry":"LatteFeatureProcessor","repo":"ruxueshi/Latte","repo_kind":"found_in_text","path":"model/SaTE/LatteSaTE.py","file_url":"https://github.com/ruxueshi/Latte/blob/HEAD/model/SaTE/LatteSaTE.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eb65bc8acaad9d41"}},{"code_sha256_prefix":"619752de3440753b","entry":"LatteModel","repo":"ruxueshi/Latte","repo_kind":"found_in_text","path":"model/SaTE/LatteSaTE.py","file_url":"https://github.com/ruxueshi/Latte/blob/HEAD/model/SaTE/LatteSaTE.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"619752de3440753b"}},{"code_sha256_prefix":"531d2e715319c628","entry":"LatteWordEmbedding","repo":"ruxueshi/Latte","repo_kind":"found_in_text","path":"model/SaTE/LatteSaTE.py","file_url":"https://github.com/ruxueshi/Latte/blob/HEAD/model/SaTE/LatteSaTE.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"531d2e715319c628"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}