{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/saint-improved-neural-networks-for-tabular","title":"SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training","arxiv_id":"2106.01342","date":"2021-06-02","proceeding":null,"authors":["Gowthami Somepalli","Micah Goldblum","Avi Schwarzschild","C. Bayan Bruss","Tom Goldstein"],"abstract":"Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-supervised pre-training method for use when labels are scarce. SAINT consistently improves performance over previous deep learning methods, and it even outperforms gradient boosting methods, including XGBoost, CatBoost, and LightGBM, on average over a variety of benchmark tasks.","url_abs":"https://arxiv.org/abs/2106.01342v1","url_pdf":"https://arxiv.org/pdf/2106.01342v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"saint-improved-neural-networks-for-tabular","repo_url":"https://github.com/somepago/saint","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"saint-improved-neural-networks-for-tabular","repo_url":"https://github.com/Actis92/lit-saint","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"saint-improved-neural-networks-for-tabular","repo_url":"https://github.com/Actis92/saint-lightning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"saint-improved-neural-networks-for-tabular","repo_url":"https://github.com/johanattia/tensorflow-saint","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"saint-improved-neural-networks-for-tabular","repo_url":"https://github.com/jrzaurin/pytorch-widedeep","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"saint-improved-neural-networks-for-tabular","repo_url":"https://github.com/ogunlao/saint","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"saint-improved-neural-networks-for-tabular","repo_url":"https://github.com/lucidrains/tab-transformer-pytorch/tree/main/tab_transformer_pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"fraud-detection","task_name":"Fraud Detection"},{"task_slug":null,"task_name":"Insurance Prediction"}],"methods":[{"method_slug":"cutmix","method_name":"CutMix"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"mixup","method_name":"Mixup"},{"method_slug":"saint","method_name":"SAINT"}],"datasets_introduced":[],"methods_introduced":[{"slug":"saint","name":"SAINT","full_name":"SAINT"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.01342","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.01342"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Actis92/saint-lightning","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/tab-transformer-pytorch/tree/main/tab_transformer_pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ogunlao/saint","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Actis92/lit-saint","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jrzaurin/pytorch-widedeep","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/johanattia/tensorflow-saint","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/somepago/saint","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":1,"ran_violates":2,"unverified":23},"by_repo_kind":{"official":{"samples":13,"ran":3,"repositories":1},"listed":{"samples":13,"ran":0,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"f6b944f50d3f15ae","entry":"count_parameters","repo":"somepago/saint","repo_kind":"official","path":"utils.py","file_url":"https://github.com/somepago/saint/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f6b944f50d3f15ae"}},{"code_sha256_prefix":"60fff7c3c400d7ff","entry":"default","repo":"somepago/saint","repo_kind":"official","path":"models/model.py","file_url":"https://github.com/somepago/saint/blob/HEAD/models/model.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"60fff7c3c400d7ff"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"somepago/saint","repo_kind":"official","path":"models/model.py","file_url":"https://github.com/somepago/saint/blob/HEAD/models/model.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"b042dc78ffec4ba8","entry":"add_noise","repo":"somepago/saint","repo_kind":"official","path":"augmentations.py","file_url":"https://github.com/somepago/saint/blob/HEAD/augmentations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b042dc78ffec4ba8"}},{"code_sha256_prefix":"525ff13ecf3ecd51","entry":"attention","repo":"ogunlao/saint","repo_kind":"listed","path":"models/transformer.py","file_url":"https://github.com/ogunlao/saint/blob/HEAD/models/transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"525ff13ecf3ecd51"}},{"code_sha256_prefix":"2d85447d6a2e686d","entry":"concat_data","repo":"somepago/saint","repo_kind":"official","path":"data_openml.py","file_url":"https://github.com/somepago/saint/blob/HEAD/data_openml.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2d85447d6a2e686d"}},{"code_sha256_prefix":"ea091d52c3e000f5","entry":"cutmix","repo":"Actis92/lit-saint","repo_kind":"listed","path":"src/lit_saint/augmentations.py","file_url":"https://github.com/Actis92/lit-saint/blob/HEAD/src/lit_saint/augmentations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea091d52c3e000f5"}},{"code_sha256_prefix":"3c8af69a2bc14906","entry":"data_mask_split","repo":"somepago/saint","repo_kind":"official","path":"old_version/data.py","file_url":"https://github.com/somepago/saint/blob/HEAD/old_version/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3c8af69a2bc14906"}},{"code_sha256_prefix":"afcf8e4fc8cd0b4d","entry":"data_split","repo":"somepago/saint","repo_kind":"official","path":"data_openml.py","file_url":"https://github.com/somepago/saint/blob/HEAD/data_openml.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"afcf8e4fc8cd0b4d"}},{"code_sha256_prefix":"4f7a0a76a82b38c7","entry":"embed_data_mask","repo":"somepago/saint","repo_kind":"official","path":"augmentations.py","file_url":"https://github.com/somepago/saint/blob/HEAD/augmentations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4f7a0a76a82b38c7"}},{"code_sha256_prefix":"6336169744f2eaf8","entry":"feature_space_to_config","repo":"johanattia/tensorflow-saint","repo_kind":"listed","path":"src/rectransformer/feature.py","file_url":"https://github.com/johanattia/tensorflow-saint/blob/HEAD/src/rectransformer/feature.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6336169744f2eaf8"}},{"code_sha256_prefix":"8728271a97b8e7c5","entry":"ff_encodings","repo":"somepago/saint","repo_kind":"official","path":"models/model.py","file_url":"https://github.com/somepago/saint/blob/HEAD/models/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8728271a97b8e7c5"}},{"code_sha256_prefix":"e78cd941e1406260","entry":"generate_dataloader","repo":"ogunlao/saint","repo_kind":"listed","path":"src/dataloader.py","file_url":"https://github.com/ogunlao/saint/blob/HEAD/src/dataloader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e78cd941e1406260"}},{"code_sha256_prefix":"7812364a72fefddf","entry":"generate_dataset","repo":"ogunlao/saint","repo_kind":"listed","path":"src/dataset.py","file_url":"https://github.com/ogunlao/saint/blob/HEAD/src/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7812364a72fefddf"}},{"code_sha256_prefix":"9785dbb6cb3ec3f0","entry":"generate_splits","repo":"ogunlao/saint","repo_kind":"listed","path":"src/dataset.py","file_url":"https://github.com/ogunlao/saint/blob/HEAD/src/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9785dbb6cb3ec3f0"}},{"code_sha256_prefix":"d07d3bae3af0c485","entry":"get_random_index","repo":"Actis92/lit-saint","repo_kind":"listed","path":"src/lit_saint/augmentations.py","file_url":"https://github.com/Actis92/lit-saint/blob/HEAD/src/lit_saint/augmentations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d07d3bae3af0c485"}},{"code_sha256_prefix":"09d16d8b95917125","entry":"intersample","repo":"ogunlao/saint","repo_kind":"listed","path":"models/saint_i.py","file_url":"https://github.com/ogunlao/saint/blob/HEAD/models/saint_i.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09d16d8b95917125"}},{"code_sha256_prefix":"6e97686ea3566edf","entry":"make_default_mask","repo":"somepago/saint","repo_kind":"official","path":"utils.py","file_url":"https://github.com/somepago/saint/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"6e97686ea3566edf"}},{"code_sha256_prefix":"2f37195b67ea2d56","entry":"make_saint","repo":"ogunlao/saint","repo_kind":"listed","path":"models/saint.py","file_url":"https://github.com/ogunlao/saint/blob/HEAD/models/saint.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f37195b67ea2d56"}},{"code_sha256_prefix":"82133b453e0aa786","entry":"make_saint_i","repo":"ogunlao/saint","repo_kind":"listed","path":"models/saint_i.py","file_url":"https://github.com/ogunlao/saint/blob/HEAD/models/saint_i.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"82133b453e0aa786"}},{"code_sha256_prefix":"aa5c35fec761ea99","entry":"make_saint_s","repo":"ogunlao/saint","repo_kind":"listed","path":"models/saint_s.py","file_url":"https://github.com/ogunlao/saint/blob/HEAD/models/saint_s.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5c35fec761ea99"}},{"code_sha256_prefix":"1d107db8a201c1e5","entry":"mixup","repo":"Actis92/lit-saint","repo_kind":"listed","path":"src/lit_saint/augmentations.py","file_url":"https://github.com/Actis92/lit-saint/blob/HEAD/src/lit_saint/augmentations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d107db8a201c1e5"}},{"code_sha256_prefix":"3f8482bac2c6b9db","entry":"mixup_data","repo":"somepago/saint","repo_kind":"official","path":"augmentations.py","file_url":"https://github.com/somepago/saint/blob/HEAD/augmentations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3f8482bac2c6b9db"}},{"code_sha256_prefix":"957d6b14b7e5575a","entry":"preprocess","repo":"ogunlao/saint","repo_kind":"listed","path":"src/dataset.py","file_url":"https://github.com/ogunlao/saint/blob/HEAD/src/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"957d6b14b7e5575a"}},{"code_sha256_prefix":"617821203b46f8a6","entry":"tag_gen","repo":"somepago/saint","repo_kind":"official","path":"utils.py","file_url":"https://github.com/somepago/saint/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"617821203b46f8a6"}},{"code_sha256_prefix":"04ca528d5c7c55fa","entry":"task_dset_ids","repo":"somepago/saint","repo_kind":"official","path":"data_openml.py","file_url":"https://github.com/somepago/saint/blob/HEAD/data_openml.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"04ca528d5c7c55fa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}