{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-attention-between-datapoints-going","title":"Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning","arxiv_id":"2106.02584","date":"2021-06-04","proceeding":"NeurIPS 2021 12","authors":["Jannik Kossen","Neil Band","Clare Lyle","Aidan N. Gomez","Tom Rainforth","Yarin Gal"],"abstract":"We challenge a common assumption underlying most supervised deep learning: that a model makes a prediction depending only on its parameters and the features of a single input. To this end, we introduce a general-purpose deep learning architecture that takes as input the entire dataset instead of processing one datapoint at a time. Our approach uses self-attention to reason about relationships between datapoints explicitly, which can be seen as realizing non-parametric models using parametric attention mechanisms. However, unlike conventional non-parametric models, we let the model learn end-to-end from the data how to make use of other datapoints for prediction. Empirically, our models solve cross-datapoint lookup and complex reasoning tasks unsolvable by traditional deep learning models. We show highly competitive results on tabular data, early results on CIFAR-10, and give insight into how the model makes use of the interactions between points.","url_abs":"https://arxiv.org/abs/2106.02584v2","url_pdf":"https://arxiv.org/pdf/2106.02584v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-attention-between-datapoints-going","repo_url":"https://github.com/OATML/Non-Parametric-Transformers","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"self-attention-between-datapoints-going","repo_url":"https://github.com/oatml-markslab/proteinnpt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"self-attention-between-datapoints-going","repo_url":"https://github.com/richrast/spin","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"3d-part-segmentation","task_name":"3D Part Segmentation"},{"task_slug":"deep-learning","task_name":"Deep Learning"}],"methods":[{"method_slug":"1cycle","method_name":"1cycle"},{"method_slug":"adaptive-masking","method_name":"Adaptive Masking"},{"method_slug":"adaptive-span-transformer","method_name":"Adaptive Span Transformer"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"embedding-dropout","method_name":"Embedding Dropout"},{"method_slug":"l1-regularization","method_name":"L1 Regularization"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.02584","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.02584"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/oatml-markslab/proteinnpt","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/OATML/Non-Parametric-Transformers","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/richrast/spin","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":2,"ran":7,"ran_violates":1,"unverified":20},"by_repo_kind":{"official":{"samples":11,"ran":6,"repositories":1},"listed":{"samples":19,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0e18add484f1a90c","entry":"count_parameters","repo":"oatml/non-parametric-transformers","repo_kind":"official","path":"npt/utils/train_utils.py","file_url":"https://github.com/oatml/non-parametric-transformers/blob/HEAD/npt/utils/train_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0e18add484f1a90c"}},{"code_sha256_prefix":"a94fc028457ce3f8","entry":"Args","repo":"OATML/Non-Parametric-Transformers","repo_kind":"official","path":"npt/model/npt.py","file_url":"https://github.com/OATML/Non-Parametric-Transformers/blob/HEAD/npt/model/npt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a94fc028457ce3f8"}},{"code_sha256_prefix":"7232573389ce2a5a","entry":"MAB","repo":"OATML/Non-Parametric-Transformers","repo_kind":"official","path":"npt/model/npt.py","file_url":"https://github.com/OATML/Non-Parametric-Transformers/blob/HEAD/npt/model/npt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7232573389ce2a5a"}},{"code_sha256_prefix":"d6f128d3e349e966","entry":"Print","repo":"OATML/Non-Parametric-Transformers","repo_kind":"official","path":"npt/model/npt.py","file_url":"https://github.com/OATML/Non-Parametric-Transformers/blob/HEAD/npt/model/npt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d6f128d3e349e966"}},{"code_sha256_prefix":"0ead572d1093cc14","entry":"ReshapeToFlat","repo":"OATML/Non-Parametric-Transformers","repo_kind":"official","path":"npt/model/npt.py","file_url":"https://github.com/OATML/Non-Parametric-Transformers/blob/HEAD/npt/model/npt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ead572d1093cc14"}},{"code_sha256_prefix":"e6ffb4964d98205f","entry":"ReshapeToNested","repo":"OATML/Non-Parametric-Transformers","repo_kind":"official","path":"npt/model/npt.py","file_url":"https://github.com/OATML/Non-Parametric-Transformers/blob/HEAD/npt/model/npt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e6ffb4964d98205f"}},{"code_sha256_prefix":"f6701ae421cbe0d4","entry":"apc","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/esm/modules.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/esm/modules.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6701ae421cbe0d4"}},{"code_sha256_prefix":"2e7bd6c4ccd1ed68","entry":"gelu","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/esm/modules.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/esm/modules.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e7bd6c4ccd1ed68"}},{"code_sha256_prefix":"4305f4fdd8e54bc1","entry":"get_slopes","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/tranception/model_pytorch.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/tranception/model_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4305f4fdd8e54bc1"}},{"code_sha256_prefix":"b385be9826c4a8b3","entry":"symmetrize","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/esm/modules.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/esm/modules.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b385be9826c4a8b3"}},{"code_sha256_prefix":"8d76f18f88c52502","entry":"ImagePatcher","repo":"OATML/Non-Parametric-Transformers","repo_kind":"official","path":"npt/model/npt.py","file_url":"https://github.com/OATML/Non-Parametric-Transformers/blob/HEAD/npt/model/npt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d76f18f88c52502"}},{"code_sha256_prefix":"ce48ecd197be4b48","entry":"LinearImagePatcher","repo":"OATML/Non-Parametric-Transformers","repo_kind":"official","path":"npt/model/npt.py","file_url":"https://github.com/OATML/Non-Parametric-Transformers/blob/HEAD/npt/model/npt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce48ecd197be4b48"}},{"code_sha256_prefix":"f3dd2c06a9c7eb93","entry":"MHSA","repo":"OATML/Non-Parametric-Transformers","repo_kind":"official","path":"npt/model/npt.py","file_url":"https://github.com/OATML/Non-Parametric-Transformers/blob/HEAD/npt/model/npt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f3dd2c06a9c7eb93"}},{"code_sha256_prefix":"611506804344b302","entry":"NPTModel","repo":"OATML/Non-Parametric-Transformers","repo_kind":"official","path":"npt/model/npt.py","file_url":"https://github.com/OATML/Non-Parametric-Transformers/blob/HEAD/npt/model/npt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"611506804344b302"}},{"code_sha256_prefix":"d2d1d1bc5e8b6b1d","entry":"PNPT_sample_training_points_inference","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/proteinnpt/data_processing.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/proteinnpt/data_processing.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2d1d1bc5e8b6b1d"}},{"code_sha256_prefix":"67c82e2cb5a619f5","entry":"SaveAttMaps","repo":"OATML/Non-Parametric-Transformers","repo_kind":"official","path":"npt/model/npt.py","file_url":"https://github.com/OATML/Non-Parametric-Transformers/blob/HEAD/npt/model/npt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"67c82e2cb5a619f5"}},{"code_sha256_prefix":"7fc8e20691370cda","entry":"assign_group_stratified","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/cv_split.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/cv_split.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7fc8e20691370cda"}},{"code_sha256_prefix":"7b4449eb96bfa4eb","entry":"compute_weight","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/msa_utils.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/msa_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b4449eb96bfa4eb"}},{"code_sha256_prefix":"0d397ce1b8012be1","entry":"compute_weight_all","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/msa_utils.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/msa_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d397ce1b8012be1"}},{"code_sha256_prefix":"7dabe8f85e5f4e16","entry":"get_learning_rate","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/model_utils.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7dabe8f85e5f4e16"}},{"code_sha256_prefix":"a8991266f955dd35","entry":"get_parameter_names","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/model_utils.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8991266f955dd35"}},{"code_sha256_prefix":"e20cd5502a6c2414","entry":"get_tranception_tokenizer","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/tranception/model_pytorch.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/tranception/model_pytorch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e20cd5502a6c2414"}},{"code_sha256_prefix":"ea7a41d551b7a7e5","entry":"keep_singles","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/cv_split.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/cv_split.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea7a41d551b7a7e5"}},{"code_sha256_prefix":"355bff3ffb0d66a7","entry":"learning_rate_scheduler","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/model_utils.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"355bff3ffb0d66a7"}},{"code_sha256_prefix":"5574012ba4af04ff","entry":"nanmean","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/tranception/model_pytorch.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/tranception/model_pytorch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5574012ba4af04ff"}},{"code_sha256_prefix":"52a00d097b2f46fd","entry":"one_hot_3D","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/msa_utils.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/msa_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52a00d097b2f46fd"}},{"code_sha256_prefix":"c5e4288eb459c27d","entry":"process_batch","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/baselines/data_processing.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/baselines/data_processing.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5e4288eb459c27d"}},{"code_sha256_prefix":"4074372860cf12f3","entry":"return_position_single","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/cv_split.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/cv_split.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4074372860cf12f3"}},{"code_sha256_prefix":"72ad19ee550806af","entry":"split_data_based_on_test_fold_index","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/data_utils.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72ad19ee550806af"}},{"code_sha256_prefix":"74862b2cc41d3a95","entry":"standardize","repo":"oatml-markslab/proteinnpt","repo_kind":"listed","path":"proteinnpt/utils/data_utils.py","file_url":"https://github.com/oatml-markslab/proteinnpt/blob/HEAD/proteinnpt/utils/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74862b2cc41d3a95"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}