{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/understanding-deep-representation-learning","title":"Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination","arxiv_id":"2311.02960","date":"2023-11-06","proceeding":null,"authors":["Peng Wang","Xiao Li","Can Yaras","Zhihui Zhu","Laura Balzano","Wei Hu","Qing Qu"],"abstract":"Over the past decade, deep learning has proven to be a highly effective tool for learning meaningful features from raw data. However, it remains an open question how deep networks perform hierarchical feature learning across layers. In this work, we attempt to unveil this mystery by investigating the structures of intermediate features. Motivated by our empirical findings that linear layers mimic the roles of deep layers in nonlinear networks for feature learning, we explore how deep linear networks transform input data into output by investigating the output (i.e., features) of each layer after training in the context of multi-class classification problems. Toward this goal, we first define metrics to measure within-class compression and between-class discrimination of intermediate features, respectively. Through theoretical analysis of these two metrics, we show that the evolution of features follows a simple and quantitative pattern from shallow to deep layers when the input data is nearly orthogonal and the network weights are minimum-norm, balanced, and approximate low-rank: Each layer of the linear network progressively compresses within-class features at a geometric rate and discriminates between-class features at a linear rate with respect to the number of layers that data have passed through. To the best of our knowledge, this is the first quantitative characterization of feature evolution in hierarchical representations of deep linear networks. Empirically, our extensive experiments not only validate our theoretical results numerically but also reveal a similar pattern in deep nonlinear networks which aligns well with recent empirical studies. Moreover, we demonstrate the practical implications of our results in transfer learning. Our code is available at https://github.com/Heimine/PNC_DLN.","url_abs":"https://arxiv.org/abs/2311.02960v3","url_pdf":"https://arxiv.org/pdf/2311.02960v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understanding-deep-representation-learning","repo_url":"https://github.com/heimine/pnc_dln","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"feature-compression","task_name":"Feature Compression"},{"task_slug":"multi-class-classification","task_name":"Multi-class Classification"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.02960","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.02960"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/heimine/pnc_dln","reach":{"status":"ok"}}],"summary":{"ran":11,"unverified":6},"by_repo_kind":{"official":{"samples":17,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"18c52f20911e8eaf","entry":"compute_Sigma_B","repo":"heimine/pnc_dln","repo_kind":"official","path":"assumption/utility.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/assumption/utility.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"18c52f20911e8eaf"}},{"code_sha256_prefix":"08fc1a53e564d2f6","entry":"compute_Sigma_W","repo":"heimine/pnc_dln","repo_kind":"official","path":"assumption/utility.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/assumption/utility.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"08fc1a53e564d2f6"}},{"code_sha256_prefix":"d73469beb39afa9d","entry":"compute_info","repo":"heimine/pnc_dln","repo_kind":"official","path":"utility.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/utility.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d73469beb39afa9d"}},{"code_sha256_prefix":"731852970e97b98b","entry":"dict_to_mat","repo":"heimine/pnc_dln","repo_kind":"official","path":"utility.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/utility.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"731852970e97b98b"}},{"code_sha256_prefix":"61e44a044531d7c1","entry":"evaluate_epoch","repo":"heimine/pnc_dln","repo_kind":"official","path":"train_cifar.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/train_cifar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61e44a044531d7c1"}},{"code_sha256_prefix":"04dc0b82637d2c30","entry":"gaussian_dataset","repo":"heimine/pnc_dln","repo_kind":"official","path":"assumption/dataset.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/assumption/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04dc0b82637d2c30"}},{"code_sha256_prefix":"84b336c1b357cf00","entry":"loss_fn","repo":"heimine/pnc_dln","repo_kind":"official","path":"linear_probe.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/linear_probe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84b336c1b357cf00"}},{"code_sha256_prefix":"158ee19e017d75ab","entry":"loss_fn","repo":"heimine/pnc_dln","repo_kind":"official","path":"train_cifar.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/train_cifar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"158ee19e017d75ab"}},{"code_sha256_prefix":"b12ebb66efa5f408","entry":"orthogonal_dataset","repo":"heimine/pnc_dln","repo_kind":"official","path":"assumption/dataset.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/assumption/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b12ebb66efa5f408"}},{"code_sha256_prefix":"1a72c6410c881f70","entry":"train_epoch","repo":"heimine/pnc_dln","repo_kind":"official","path":"train_cifar.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/train_cifar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a72c6410c881f70"}},{"code_sha256_prefix":"40cfdd79fb01284c","entry":"uniform_dataset","repo":"heimine/pnc_dln","repo_kind":"official","path":"assumption/dataset.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/assumption/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"40cfdd79fb01284c"}},{"code_sha256_prefix":"74f9c675129d4174","entry":"evaluate_epoch","repo":"heimine/pnc_dln","repo_kind":"official","path":"linear_probe.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/linear_probe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74f9c675129d4174"}},{"code_sha256_prefix":"61f1b4ad98c59072","entry":"get_data","repo":"heimine/pnc_dln","repo_kind":"official","path":"partial_data.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/partial_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61f1b4ad98c59072"}},{"code_sha256_prefix":"89c9b510f840472f","entry":"get_data_set","repo":"heimine/pnc_dln","repo_kind":"official","path":"partial_data.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/partial_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89c9b510f840472f"}},{"code_sha256_prefix":"44b3595c5304a3ea","entry":"max_angle_metric","repo":"heimine/pnc_dln","repo_kind":"official","path":"utility.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/utility.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"44b3595c5304a3ea"}},{"code_sha256_prefix":"d758004d188d7ecf","entry":"train_epoch","repo":"heimine/pnc_dln","repo_kind":"official","path":"linear_probe.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/linear_probe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d758004d188d7ecf"}},{"code_sha256_prefix":"15bf27bcb1e61f56","entry":"train_epoch","repo":"heimine/pnc_dln","repo_kind":"official","path":"post_processing.py","file_url":"https://github.com/heimine/pnc_dln/blob/HEAD/post_processing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15bf27bcb1e61f56"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}