{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tract-making-first-layer-pre-activations","title":"TrAct: Making First-layer Pre-Activations Trainable","arxiv_id":"2410.23970","date":"2024-10-31","proceeding":null,"authors":["Felix Petersen","Christian Borgelt","Stefano Ermon"],"abstract":"We consider the training of the first layer of vision models and notice the clear relationship between pixel values and gradient update magnitudes: the gradients arriving at the weights of a first layer are by definition directly proportional to (normalized) input pixel values. Thus, an image with low contrast has a smaller impact on learning than an image with higher contrast, and a very bright or very dark image has a stronger impact on the weights than an image with moderate brightness. In this work, we propose performing gradient descent on the embeddings produced by the first layer of the model. However, switching to discrete inputs with an embedding layer is not a reasonable option for vision models. Thus, we propose the conceptual procedure of (i) a gradient descent step on first layer activations to construct an activation proposal, and (ii) finding the optimal weights of the first layer, i.e., those weights which minimize the squared distance to the activation proposal. We provide a closed form solution of the procedure and adjust it for robust stochastic training while computing everything efficiently. Empirically, we find that TrAct (Training Activations) speeds up training by factors between 1.25x and 4x while requiring only a small computational overhead. We demonstrate the utility of TrAct with different optimizers for a range of different vision models including convolutional and transformer architectures.","url_abs":"https://arxiv.org/abs/2410.23970v1","url_pdf":"https://arxiv.org/pdf/2410.23970v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.23970","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.23970"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Felix-Petersen/tract","reach":null}],"summary":{"ran":1,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":5,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7955512f4cce7a3c","entry":"TrActLinear","repo":"Felix-Petersen/tract","repo_kind":"found_in_text","path":"src/tract.py","file_url":"https://github.com/Felix-Petersen/tract/blob/HEAD/src/tract.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7955512f4cce7a3c"}},{"code_sha256_prefix":"d263982a373ea8a9","entry":"TrAct","repo":"Felix-Petersen/tract","repo_kind":"found_in_text","path":"src/tract.py","file_url":"https://github.com/Felix-Petersen/tract/blob/HEAD/src/tract.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d263982a373ea8a9"}},{"code_sha256_prefix":"5ab4175b9e7b9330","entry":"TrActConv2DFunction","repo":"Felix-Petersen/tract","repo_kind":"found_in_text","path":"src/tract.py","file_url":"https://github.com/Felix-Petersen/tract/blob/HEAD/src/tract.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ab4175b9e7b9330"}},{"code_sha256_prefix":"3f8d5bdef73029eb","entry":"TrActConv2d","repo":"Felix-Petersen/tract","repo_kind":"found_in_text","path":"src/tract.py","file_url":"https://github.com/Felix-Petersen/tract/blob/HEAD/src/tract.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f8d5bdef73029eb"}},{"code_sha256_prefix":"f938fe87ab165c93","entry":"TrActLinearFunction","repo":"Felix-Petersen/tract","repo_kind":"found_in_text","path":"src/tract.py","file_url":"https://github.com/Felix-Petersen/tract/blob/HEAD/src/tract.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f938fe87ab165c93"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}