{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/measuring-the-intrinsic-dimension-of","title":"Measuring the Intrinsic Dimension of Objective Landscapes","arxiv_id":"1804.08838","date":"2018-04-24","proceeding":"ICLR 2018 1","authors":["Chunyuan Li","Heerad Farkhoor","Rosanne Liu","Jason Yosinski"],"abstract":"Many recently trained neural networks employ large numbers of parameters to\nachieve good performance. One may intuitively use the number of parameters\nrequired as a rough gauge of the difficulty of a problem. But how accurate are\nsuch notions? How many parameters are really needed? In this paper we attempt\nto answer this question by training networks not in their native parameter\nspace, but instead in a smaller, randomly oriented subspace. We slowly increase\nthe dimension of this subspace, note at which dimension solutions first appear,\nand define this to be the intrinsic dimension of the objective landscape. The\napproach is simple to implement, computationally tractable, and produces\nseveral suggestive conclusions. Many problems have smaller intrinsic dimensions\nthan one might suspect, and the intrinsic dimension for a given dataset varies\nlittle across a family of models with vastly different sizes. This latter\nresult has the profound implication that once a parameter space is large enough\nto solve a problem, extra parameters serve directly to increase the\ndimensionality of the solution manifold. Intrinsic dimension allows some\nquantitative comparison of problem difficulty across supervised, reinforcement,\nand other types of learning where we conclude, for example, that solving the\ninverted pendulum problem is 100 times easier than classifying digits from\nMNIST, and playing Atari Pong from pixels is about as hard as classifying\nCIFAR-10. In addition to providing new cartography of the objective landscapes\nwandered by parameterized models, the method is a simple technique for\nconstructively obtaining an upper bound on the minimum description length of a\nsolution. A byproduct of this construction is a simple approach for compressing\nnetworks, in some cases by more than 100 times.","url_abs":"http://arxiv.org/abs/1804.08838v1","url_pdf":"http://arxiv.org/pdf/1804.08838v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"measuring-the-intrinsic-dimension-of","repo_url":"https://github.com/uber-research/intrinsic-dimension","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"measuring-the-intrinsic-dimension-of","repo_url":"https://github.com/AntoAndGar/Intrinsic-Dimension","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"measuring-the-intrinsic-dimension-of","repo_url":"https://github.com/jgamper/intrinsic-dimensionality","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"measuring-the-intrinsic-dimension-of","repo_url":"https://github.com/mdda/deep-learning-workshop/blob/master/notebooks/work-in-progress/IntrinsicDimension/IntrinsicDimension.ipynb","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1804.08838","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1804.08838"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jgamper/intrinsic-dimensionality","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uber-research/intrinsic-dimension","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AntoAndGar/Intrinsic-Dimension","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mdda/deep-learning-workshop/blob/master/notebooks/work-in-progress/IntrinsicDimension/IntrinsicDimension.ipynb","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"unverified":3},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1},"listed":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"fbb1783f0d38e809","entry":"fast_walsh_hadamard_torched","repo":"jgamper/intrinsic-dimensionality","repo_kind":"listed","path":"intrinsic/fastfood.py","file_url":"https://github.com/jgamper/intrinsic-dimensionality/blob/HEAD/intrinsic/fastfood.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbb1783f0d38e809"}},{"code_sha256_prefix":"8ec81869eabc2118","entry":"fastfood_torched","repo":"jgamper/intrinsic-dimensionality","repo_kind":"listed","path":"intrinsic/fastfood.py","file_url":"https://github.com/jgamper/intrinsic-dimensionality/blob/HEAD/intrinsic/fastfood.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ec81869eabc2118"}},{"code_sha256_prefix":"1e8a3fae82b29727","entry":"fastfood_vars","repo":"jgamper/intrinsic-dimensionality","repo_kind":"listed","path":"intrinsic/fastfood.py","file_url":"https://github.com/jgamper/intrinsic-dimensionality/blob/HEAD/intrinsic/fastfood.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e8a3fae82b29727"}},{"code_sha256_prefix":"59341af2589bece0","entry":"make_standard_parser","repo":"uber-research/intrinsic-dimension","repo_kind":"official","path":"intrinsic_dim/standard_parser.py","file_url":"https://github.com/uber-research/intrinsic-dimension/blob/HEAD/intrinsic_dim/standard_parser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"59341af2589bece0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}