{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/never-train-from-scratch-fair-comparison-of","title":"Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors","arxiv_id":"2310.02980","date":"2023-10-04","proceeding":null,"authors":["Ido Amos","Jonathan Berant","Ankit Gupta"],"abstract":"Modeling long-range dependencies across sequences is a longstanding goal in machine learning and has led to architectures, such as state space models, that dramatically outperform Transformers on long sequences. However, these impressive empirical gains have been by and large demonstrated on benchmarks (e.g. Long Range Arena), where models are randomly initialized and trained to predict a target label from an input sequence. In this work, we show that random initialization leads to gross overestimation of the differences between architectures and that pretraining with standard denoising objectives, using $\\textit{only the downstream task data}$, leads to dramatic gains across multiple architectures and to very small gaps between Transformers and state space models (SSMs). In stark contrast to prior works, we find vanilla Transformers to match the performance of S4 on Long Range Arena when properly pretrained, and we improve the best reported results of SSMs on the PathX-256 task by 20 absolute points. Subsequently, we analyze the utility of previously-proposed structured parameterizations for SSMs and show they become mostly redundant in the presence of data-driven initialization obtained through pretraining. Our work shows that, when evaluating different architectures on supervised tasks, incorporation of data-driven priors via pretraining is essential for reliable performance estimation, and can be done efficiently.","url_abs":"https://arxiv.org/abs/2310.02980v4","url_pdf":"https://arxiv.org/pdf/2310.02980v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"never-train-from-scratch-fair-comparison-of","repo_url":"https://github.com/idoamos/not-from-scratch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"state-space-models","task_name":"State Space Models"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.02980","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.02980"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/idoamos/not-from-scratch","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"ran_violates":2,"ran_draft_wrong":3,"unverified":3},"by_repo_kind":{"official":{"samples":13,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6d3d810524eb100b","entry":"apply_rotary_pos_emb","repo":"idoamos/not-from-scratch","repo_kind":"official","path":"local_attention/rotary.py","file_url":"https://github.com/idoamos/not-from-scratch/blob/HEAD/local_attention/rotary.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d3d810524eb100b"}},{"code_sha256_prefix":"b5307f70349f5c27","entry":"default","repo":"idoamos/not-from-scratch","repo_kind":"official","path":"local_attention/local_attention.py","file_url":"https://github.com/idoamos/not-from-scratch/blob/HEAD/local_attention/local_attention.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5307f70349f5c27"}},{"code_sha256_prefix":"ce310b19a858070f","entry":"dequantize","repo":"idoamos/not-from-scratch","repo_kind":"official","path":"src/dataloaders/audio.py","file_url":"https://github.com/idoamos/not-from-scratch/blob/HEAD/src/dataloaders/audio.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce310b19a858070f"}},{"code_sha256_prefix":"c16aee9490eb8729","entry":"eval_decorator","repo":"idoamos/not-from-scratch","repo_kind":"official","path":"local_attention/transformer.py","file_url":"https://github.com/idoamos/not-from-scratch/blob/HEAD/local_attention/transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c16aee9490eb8729"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"idoamos/not-from-scratch","repo_kind":"official","path":"local_attention/local_attention.py","file_url":"https://github.com/idoamos/not-from-scratch/blob/HEAD/local_attention/local_attention.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"3eb58dae620046d3","entry":"l2norm","repo":"idoamos/not-from-scratch","repo_kind":"official","path":"local_attention/transformer.py","file_url":"https://github.com/idoamos/not-from-scratch/blob/HEAD/local_attention/transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3eb58dae620046d3"}},{"code_sha256_prefix":"1b76e4737067023f","entry":"minmax_scale","repo":"idoamos/not-from-scratch","repo_kind":"official","path":"src/dataloaders/audio.py","file_url":"https://github.com/idoamos/not-from-scratch/blob/HEAD/src/dataloaders/audio.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b76e4737067023f"}},{"code_sha256_prefix":"1631cd936e9b1a90","entry":"quantize","repo":"idoamos/not-from-scratch","repo_kind":"official","path":"src/dataloaders/audio.py","file_url":"https://github.com/idoamos/not-from-scratch/blob/HEAD/src/dataloaders/audio.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1631cd936e9b1a90"}},{"code_sha256_prefix":"1147383164fd3a66","entry":"rotate_half","repo":"idoamos/not-from-scratch","repo_kind":"official","path":"local_attention/rotary.py","file_url":"https://github.com/idoamos/not-from-scratch/blob/HEAD/local_attention/rotary.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1147383164fd3a66"}},{"code_sha256_prefix":"f092dd8a37a00a6e","entry":"to","repo":"idoamos/not-from-scratch","repo_kind":"official","path":"local_attention/local_attention.py","file_url":"https://github.com/idoamos/not-from-scratch/blob/HEAD/local_attention/local_attention.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f092dd8a37a00a6e"}},{"code_sha256_prefix":"25edc240223b76c0","entry":"deprecated","repo":"idoamos/not-from-scratch","repo_kind":"official","path":"src/dataloaders/base.py","file_url":"https://github.com/idoamos/not-from-scratch/blob/HEAD/src/dataloaders/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25edc240223b76c0"}},{"code_sha256_prefix":"9d0777602eda354e","entry":"time_features","repo":"idoamos/not-from-scratch","repo_kind":"official","path":"src/dataloaders/et.py","file_url":"https://github.com/idoamos/not-from-scratch/blob/HEAD/src/dataloaders/et.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d0777602eda354e"}},{"code_sha256_prefix":"f8544563682146e5","entry":"time_features_from_frequency_str","repo":"idoamos/not-from-scratch","repo_kind":"official","path":"src/dataloaders/et.py","file_url":"https://github.com/idoamos/not-from-scratch/blob/HEAD/src/dataloaders/et.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f8544563682146e5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}