{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/estimating-the-effective-dimension-of-large","title":"Estimating the effective dimension of large biological datasets using Fisher separability analysis","arxiv_id":"1901.06328","date":"2019-01-18","proceeding":null,"authors":["Luca Albergante","Jonathan Bac","Andrei Zinovyev"],"abstract":"Modern large-scale datasets are frequently said to be high-dimensional.\nHowever, their data point clouds frequently possess structures, significantly\ndecreasing their intrinsic dimensionality (ID) due to the presence of clusters,\npoints being located close to low-dimensional varieties or fine-grained\nlumping. We test a recently introduced dimensionality estimator, based on\nanalysing the separability properties of data points, on several benchmarks and\nreal biological datasets. We show that the introduced measure of ID has\nperformance competitive with state-of-the-art measures, being efficient across\na wide range of dimensions and performing better in the case of noisy samples.\nMoreover, it allows estimating the intrinsic dimension in situations where the\nintrinsic manifold assumption is not valid.","url_abs":"http://arxiv.org/abs/1901.06328v1","url_pdf":"http://arxiv.org/pdf/1901.06328v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"estimating-the-effective-dimension-of-large","repo_url":"https://github.com/auranic/FisherSeparabilityAnalysis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"valid"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1901.06328","atlas_url":"https://app.syntology.ai/?focus=1901.06328","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}