{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/t-jepa-augmentation-free-self-supervised","title":"T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Data","arxiv_id":"2410.05016","date":"2024-10-07","proceeding":null,"authors":["Hugo Thimonier","José Lucas De Melo Costa","Fabrice Popineau","Arpad Rimmel","Bich-Liên Doan"],"abstract":"Self-supervision is often used for pre-training to foster performance on a downstream task by constructing meaningful representations of samples. Self-supervised learning (SSL) generally involves generating different views of the same sample and thus requires data augmentations that are challenging to construct for tabular data. This constitutes one of the main challenges of self-supervision for structured data. In the present work, we propose a novel augmentation-free SSL method for tabular data. Our approach, T-JEPA, relies on a Joint Embedding Predictive Architecture (JEPA) and is akin to mask reconstruction in the latent space. It involves predicting the latent representation of one subset of features from the latent representation of a different subset within the same sample, thereby learning rich representations without augmentations. We use our method as a pre-training technique and train several deep classifiers on the obtained representation. Our experimental results demonstrate a substantial improvement in both classification and regression tasks, outperforming models trained directly on samples in their original data space. Moreover, T-JEPA enables some methods to consistently outperform or match the performance of traditional methods likes Gradient Boosted Decision Trees. To understand why, we extensively characterize the obtained representations and show that T-JEPA effectively identifies relevant features for downstream tasks without access to the labels. Additionally, we introduce regularization tokens, a novel regularization method critical for training of JEPA-based models on structured data.","url_abs":"https://arxiv.org/abs/2410.05016v3","url_pdf":"https://arxiv.org/pdf/2410.05016v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.05016","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.05016"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jose-melo/t-jepa","reach":null}],"summary":{"ran":4,"ran_violates":2,"ran_draft_wrong":2,"ran_honours":1,"ran_fixture":1,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":14,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"088ac74f7c0459d3","entry":"PositionalEncoding","repo":"jose-melo/t-jepa","repo_kind":"found_in_text","path":"src/encoder.py","file_url":"https://github.com/jose-melo/t-jepa/blob/HEAD/src/encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"088ac74f7c0459d3"}},{"code_sha256_prefix":"c7213c185d77231d","entry":"Tokenizer","repo":"jose-melo/t-jepa","repo_kind":"found_in_text","path":"src/encoder.py","file_url":"https://github.com/jose-melo/t-jepa/blob/HEAD/src/encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7213c185d77231d"}},{"code_sha256_prefix":"f11d932208f0a4ed","entry":"TransformerEncoder","repo":"jose-melo/t-jepa","repo_kind":"found_in_text","path":"src/encoder.py","file_url":"https://github.com/jose-melo/t-jepa/blob/HEAD/src/encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f11d932208f0a4ed"}},{"code_sha256_prefix":"9a5b310fb0c45b61","entry":"TransformerEncoderLayer","repo":"jose-melo/t-jepa","repo_kind":"found_in_text","path":"src/encoder.py","file_url":"https://github.com/jose-melo/t-jepa/blob/HEAD/src/encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a5b310fb0c45b61"}},{"code_sha256_prefix":"be75e3fe4546c295","entry":"_detect_is_causal_mask","repo":"jose-melo/t-jepa","repo_kind":"found_in_text","path":"src/encoder.py","file_url":"https://github.com/jose-melo/t-jepa/blob/HEAD/src/encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be75e3fe4546c295"}},{"code_sha256_prefix":"65aafd77d580c49d","entry":"_generate_square_subsequent_mask","repo":"jose-melo/t-jepa","repo_kind":"found_in_text","path":"src/encoder.py","file_url":"https://github.com/jose-melo/t-jepa/blob/HEAD/src/encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"65aafd77d580c49d"}},{"code_sha256_prefix":"7372b44b96fac487","entry":"_get_activation_fn","repo":"jose-melo/t-jepa","repo_kind":"found_in_text","path":"src/encoder.py","file_url":"https://github.com/jose-melo/t-jepa/blob/HEAD/src/encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7372b44b96fac487"}},{"code_sha256_prefix":"8487b07e2fbe3aa2","entry":"_get_clones","repo":"jose-melo/t-jepa","repo_kind":"found_in_text","path":"src/encoder.py","file_url":"https://github.com/jose-melo/t-jepa/blob/HEAD/src/encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8487b07e2fbe3aa2"}},{"code_sha256_prefix":"2e73a89e42e7fd36","entry":"_get_seq_len","repo":"jose-melo/t-jepa","repo_kind":"found_in_text","path":"src/encoder.py","file_url":"https://github.com/jose-melo/t-jepa/blob/HEAD/src/encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e73a89e42e7fd36"}},{"code_sha256_prefix":"c3dd55e49fd28138","entry":"torch_cast_to_dtype","repo":"jose-melo/t-jepa","repo_kind":"found_in_text","path":"src/encoder.py","file_url":"https://github.com/jose-melo/t-jepa/blob/HEAD/src/encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3dd55e49fd28138"}},{"code_sha256_prefix":"f36b97662fdb54f7","entry":"Encoder","repo":"jose-melo/t-jepa","repo_kind":"found_in_text","path":"src/encoder.py","file_url":"https://github.com/jose-melo/t-jepa/blob/HEAD/src/encoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f36b97662fdb54f7"}},{"code_sha256_prefix":"ca39498dce297652","entry":"TabularEncoder","repo":"jose-melo/t-jepa","repo_kind":"found_in_text","path":"src/encoder.py","file_url":"https://github.com/jose-melo/t-jepa/blob/HEAD/src/encoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca39498dce297652"}},{"code_sha256_prefix":"3ef6c6b262613f76","entry":"_debug_values","repo":"jose-melo/t-jepa","repo_kind":"found_in_text","path":"src/encoder.py","file_url":"https://github.com/jose-melo/t-jepa/blob/HEAD/src/encoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ef6c6b262613f76"}},{"code_sha256_prefix":"a2d9ad216146c0fd","entry":"apply_masks_from_idx","repo":"jose-melo/t-jepa","repo_kind":"found_in_text","path":"src/encoder.py","file_url":"https://github.com/jose-melo/t-jepa/blob/HEAD/src/encoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a2d9ad216146c0fd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}