{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-useful-representations-of-recurrent","title":"Learning Useful Representations of Recurrent Neural Network Weight Matrices","arxiv_id":"2403.11998","date":"2024-03-18","proceeding":null,"authors":["Vincent Herrmann","Francesco Faccio","Jürgen Schmidhuber"],"abstract":"Recurrent Neural Networks (RNNs) are general-purpose parallel-sequential computers. The program of an RNN is its weight matrix. How to learn useful representations of RNN weights that facilitate RNN analysis as well as downstream tasks? While the mechanistic approach directly looks at some RNN's weights to predict its behavior, the functionalist approach analyzes its overall functionality-specifically, its input-output mapping. We consider several mechanistic approaches for RNN weights and adapt the permutation equivariant Deep Weight Space layer for RNNs. Our two novel functionalist approaches extract information from RNN weights by 'interrogating' the RNN through probing inputs. We develop a theoretical framework that demonstrates conditions under which the functionalist approach can generate rich representations that help determine RNN behavior. We release the first two 'model zoo' datasets for RNN weight representation learning. One consists of generative models of a class of formal languages, and the other one of classifiers of sequentially processed MNIST digits.With the help of an emulation-based self-supervised learning technique we compare and evaluate the different RNN weight encoding techniques on multiple downstream applications. On the most challenging one, namely predicting which exact task the RNN was trained on, functionalist approaches show clear superiority.","url_abs":"https://arxiv.org/abs/2403.11998v2","url_pdf":"https://arxiv.org/pdf/2403.11998v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-useful-representations-of-recurrent","repo_url":"https://github.com/vincentherrmann/rnn-weights-representation-learning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.11998","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.11998"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vincentherrmann/rnn-weights-representation-learning","reach":null}],"summary":{"ran_draft_wrong":1,"ran_fixture":2,"ran_honours":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"d6bf9b1d6db6bedd","entry":"conditional_args","repo":"vincentherrmann/rnn-weights-representation-learning","repo_kind":"official","path":"rnn_weights_representation_learning/training_script.py","file_url":"https://github.com/vincentherrmann/rnn-weights-representation-learning/blob/HEAD/rnn_weights_representation_learning/training_script.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6bf9b1d6db6bedd"}},{"code_sha256_prefix":"173e195e1fe036cf","entry":"mlp","repo":"vincentherrmann/rnn-weights-representation-learning","repo_kind":"official","path":"rnn_weights_representation_learning/modules_rnn_encoders.py","file_url":"https://github.com/vincentherrmann/rnn-weights-representation-learning/blob/HEAD/rnn_weights_representation_learning/modules_rnn_encoders.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"173e195e1fe036cf"}},{"code_sha256_prefix":"745af9db259597d8","entry":"task_entropy_criterion","repo":"vincentherrmann/rnn-weights-representation-learning","repo_kind":"official","path":"rnn_weights_representation_learning/dataset_classes.py","file_url":"https://github.com/vincentherrmann/rnn-weights-representation-learning/blob/HEAD/rnn_weights_representation_learning/dataset_classes.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"745af9db259597d8"}},{"code_sha256_prefix":"8db20376ac400c5c","entry":"task_random_criterion","repo":"vincentherrmann/rnn-weights-representation-learning","repo_kind":"official","path":"rnn_weights_representation_learning/dataset_classes.py","file_url":"https://github.com/vincentherrmann/rnn-weights-representation-learning/blob/HEAD/rnn_weights_representation_learning/dataset_classes.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8db20376ac400c5c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}