{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-curves-for-noisy-heterogeneous","title":"Learning Curves for Noisy Heterogeneous Feature-Subsampled Ridge Ensembles","arxiv_id":"2307.03176","date":"2023-07-06","proceeding":"NeurIPS 2023 11","authors":["Benjamin S. Ruben","Cengiz Pehlevan"],"abstract":"Feature bagging is a well-established ensembling method which aims to reduce prediction variance by combining predictions of many estimators trained on subsets or projections of features. Here, we develop a theory of feature-bagging in noisy least-squares ridge ensembles and simplify the resulting learning curves in the special case of equicorrelated data. Using analytical learning curves, we demonstrate that subsampling shifts the double-descent peak of a linear predictor. This leads us to introduce heterogeneous feature ensembling, with estimators built on varying numbers of feature dimensions, as a computationally efficient method to mitigate double-descent. Then, we compare the performance of a feature-subsampling ensemble to a single linear predictor, describing a trade-off between noise amplification due to subsampling and noise reduction due to ensembling. Our qualitative insights carry over to linear classifiers applied to image classification tasks with realistic datasets constructed using a state-of-the-art deep learning feature map.","url_abs":"https://arxiv.org/abs/2307.03176v3","url_pdf":"https://arxiv.org/pdf/2307.03176v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-curves-for-noisy-heterogeneous","repo_url":"https://github.com/benruben87/Learning-Curves-for-Heterogeneous-Feature-Subsampled-Ridge-Ensembles","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.03176","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.03176"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/benruben87/Learning-Curves-for-Heterogeneous-Feature-Subsampled-Ridge-Ensembles","reach":null}],"summary":{"ran_violates":2,"ran_draft_wrong":1,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"caae4fb0bbca3399","entry":"getIsotropicError_diag","repo":"benruben87/learning-curves-for-heterogeneous-feature-subsampled-ridge-ensembles","repo_kind":"official","path":"TheoryCurves.py","file_url":"https://github.com/benruben87/learning-curves-for-heterogeneous-feature-subsampled-ridge-ensembles/blob/HEAD/TheoryCurves.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"caae4fb0bbca3399"}},{"code_sha256_prefix":"1ff5df7a86c54328","entry":"getIsotropicSP","repo":"benruben87/learning-curves-for-heterogeneous-feature-subsampled-ridge-ensembles","repo_kind":"official","path":"TheoryCurves.py","file_url":"https://github.com/benruben87/learning-curves-for-heterogeneous-feature-subsampled-ridge-ensembles/blob/HEAD/TheoryCurves.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1ff5df7a86c54328"}},{"code_sha256_prefix":"e9302d7404571971","entry":"invertRSMatrix","repo":"benruben87/learning-curves-for-heterogeneous-feature-subsampled-ridge-ensembles","repo_kind":"official","path":"TheoryCurves.py","file_url":"https://github.com/benruben87/learning-curves-for-heterogeneous-feature-subsampled-ridge-ensembles/blob/HEAD/TheoryCurves.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e9302d7404571971"}},{"code_sha256_prefix":"8cfff408b7cee6d7","entry":"matrix_sqrt","repo":"benruben87/Learning-Curves-for-Heterogeneous-Feature-Subsampled-Ridge-Ensembles","repo_kind":"official","path":"EnsembleLib.py","file_url":"https://github.com/benruben87/Learning-Curves-for-Heterogeneous-Feature-Subsampled-Ridge-Ensembles/blob/HEAD/EnsembleLib.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8cfff408b7cee6d7"}},{"code_sha256_prefix":"0f233df7d3c42b40","entry":"MultiReadout","repo":"benruben87/Learning-Curves-for-Heterogeneous-Feature-Subsampled-Ridge-Ensembles","repo_kind":"official","path":"EnsembleLib.py","file_url":"https://github.com/benruben87/Learning-Curves-for-Heterogeneous-Feature-Subsampled-Ridge-Ensembles/blob/HEAD/EnsembleLib.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f233df7d3c42b40"}},{"code_sha256_prefix":"133264009d528d4b","entry":"makeExclusiveAList","repo":"benruben87/Learning-Curves-for-Heterogeneous-Feature-Subsampled-Ridge-Ensembles","repo_kind":"official","path":"EnsembleLib.py","file_url":"https://github.com/benruben87/Learning-Curves-for-Heterogeneous-Feature-Subsampled-Ridge-Ensembles/blob/HEAD/EnsembleLib.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"133264009d528d4b"}},{"code_sha256_prefix":"c0973ef950130fd4","entry":"makeProjectionMatrix","repo":"benruben87/Learning-Curves-for-Heterogeneous-Feature-Subsampled-Ridge-Ensembles","repo_kind":"official","path":"EnsembleLib.py","file_url":"https://github.com/benruben87/Learning-Curves-for-Heterogeneous-Feature-Subsampled-Ridge-Ensembles/blob/HEAD/EnsembleLib.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c0973ef950130fd4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}