{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/estimating-the-intrinsic-dimension-of","title":"Estimating the intrinsic dimension of datasets by a minimal neighborhood information","arxiv_id":"1803.06992","date":"2018-03-19","proceeding":null,"authors":["Elena Facco","Maria d'Errico","Alex Rodriguez","Alessandro Laio"],"abstract":"Analyzing large volumes of high-dimensional data is an issue of fundamental\nimportance in data science, molecular simulations and beyond. Several\napproaches work on the assumption that the important content of a dataset\nbelongs to a manifold whose Intrinsic Dimension (ID) is much lower than the\ncrude large number of coordinates. Such manifold is generally twisted and\ncurved, in addition points on it will be non-uniformly distributed: two factors\nthat make the identification of the ID and its exploitation really hard. Here\nwe propose a new ID estimator using only the distance of the first and the\nsecond nearest neighbor of each point in the sample. This extreme minimality\nenables us to reduce the effects of curvature, of density variation, and the\nresulting computational cost. The ID estimator is theoretically exact in\nuniformly distributed datasets, and provides consistent measures in general.\nWhen used in combination with block analysis, it allows discriminating the\nrelevant dimensions as a function of the block size. This allows estimating the\nID even when the data lie on a manifold perturbed by a high-dimensional noise,\na situation often encountered in real world data sets. We demonstrate the\nusefulness of the approach on molecular simulations and image analysis.","url_abs":"http://arxiv.org/abs/1803.06992v1","url_pdf":"http://arxiv.org/pdf/1803.06992v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"estimating-the-intrinsic-dimension-of","repo_url":"https://github.com/mariaderrico/DPA","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1803.06992","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}