{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-rainbow-in-deep-network-black-boxes","title":"A Rainbow in Deep Network Black Boxes","arxiv_id":"2305.18512","date":"2023-05-29","proceeding":null,"authors":["Florentin Guth","Brice Ménard","Gaspar Rochette","Stéphane Mallat"],"abstract":"A central question in deep learning is to understand the functions learned by deep networks. What is their approximation class? Do the learned weights and representations depend on initialization? Previous empirical work has evidenced that kernels defined by network activations are similar across initializations. For shallow networks, this has been theoretically studied with random feature models, but an extension to deep networks has remained elusive. Here, we provide a deep extension of such random feature models, which we call the rainbow model. We prove that rainbow networks define deterministic (hierarchical) kernels in the infinite-width limit. The resulting functions thus belong to a data-dependent RKHS which does not depend on the weight randomness. We also verify numerically our modeling assumptions on deep CNNs trained on image classification tasks, and show that the trained networks approximately satisfy the rainbow hypothesis. In particular, rainbow networks sampled from the corresponding random feature model achieve similar performance as the trained networks. Our results highlight the central role played by the covariances of network weights at each layer, which are observed to be low-rank as a result of feature learning.","url_abs":"https://arxiv.org/abs/2305.18512v3","url_pdf":"https://arxiv.org/pdf/2305.18512v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-rainbow-in-deep-network-black-boxes","repo_url":"https://github.com/florentinguth/rainbow","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.18512","atlas_url":"https://app.syntology.ai/?focus=2305.18512","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.18512"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/florentinguth/rainbow","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran_draft_wrong":2,"unverified":12},"by_repo_kind":{"official":{"samples":14,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d9def42110729a85","entry":"conv1x1","repo":"florentinguth/rainbow","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/florentinguth/rainbow/blob/HEAD/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d9def42110729a85"}},{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"florentinguth/rainbow","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/florentinguth/rainbow/blob/HEAD/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"8dafc755add82efd","entry":"DCT_I","repo":"florentinguth/rainbow","repo_kind":"official","path":"models/DCT.py","file_url":"https://github.com/florentinguth/rainbow/blob/HEAD/models/DCT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"8dafc755add82efd"}},{"code_sha256_prefix":"240c75536744ea04","entry":"DCT_II","repo":"florentinguth/rainbow","repo_kind":"official","path":"models/DCT.py","file_url":"https://github.com/florentinguth/rainbow/blob/HEAD/models/DCT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"240c75536744ea04"}},{"code_sha256_prefix":"1dbdb66cd1152c11","entry":"DCT_III","repo":"florentinguth/rainbow","repo_kind":"official","path":"models/DCT.py","file_url":"https://github.com/florentinguth/rainbow/blob/HEAD/models/DCT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1dbdb66cd1152c11"}},{"code_sha256_prefix":"214f33cf948da458","entry":"conv2d","repo":"florentinguth/rainbow","repo_kind":"official","path":"models/LinearProj.py","file_url":"https://github.com/florentinguth/rainbow/blob/HEAD/models/LinearProj.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"214f33cf948da458"}},{"code_sha256_prefix":"de6dcf943e71f6fb","entry":"gaussian_window","repo":"florentinguth/rainbow","repo_kind":"official","path":"models/STFT.py","file_url":"https://github.com/florentinguth/rainbow/blob/HEAD/models/STFT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"de6dcf943e71f6fb"}},{"code_sha256_prefix":"e6fb903521a8cee4","entry":"get_datasets","repo":"florentinguth/rainbow","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/florentinguth/rainbow/blob/HEAD/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e6fb903521a8cee4"}},{"code_sha256_prefix":"7331f2a3d29fc235","entry":"hanning_window","repo":"florentinguth/rainbow","repo_kind":"official","path":"models/STFT.py","file_url":"https://github.com/florentinguth/rainbow/blob/HEAD/models/STFT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"7331f2a3d29fc235"}},{"code_sha256_prefix":"92d96ed40e7d0a99","entry":"modulus","repo":"florentinguth/rainbow","repo_kind":"official","path":"models/Analysis.py","file_url":"https://github.com/florentinguth/rainbow/blob/HEAD/models/Analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"92d96ed40e7d0a99"}},{"code_sha256_prefix":"31e6ebc3a900f3de","entry":"new_logfile","repo":"florentinguth/rainbow","repo_kind":"official","path":"main_block.py","file_url":"https://github.com/florentinguth/rainbow/blob/HEAD/main_block.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"31e6ebc3a900f3de"}},{"code_sha256_prefix":"1a2d7d179b6fcef3","entry":"relu","repo":"florentinguth/rainbow","repo_kind":"official","path":"models/Analysis.py","file_url":"https://github.com/florentinguth/rainbow/blob/HEAD/models/Analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1a2d7d179b6fcef3"}},{"code_sha256_prefix":"ec4c52396ad68e3e","entry":"resnet","repo":"florentinguth/rainbow","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/florentinguth/rainbow/blob/HEAD/models/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ec4c52396ad68e3e"}},{"code_sha256_prefix":"f2a6aad1f81a2872","entry":"softshrink","repo":"florentinguth/rainbow","repo_kind":"official","path":"models/Analysis.py","file_url":"https://github.com/florentinguth/rainbow/blob/HEAD/models/Analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f2a6aad1f81a2872"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}