{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/textit-greylock-a-python-package-for","title":"$\\textit{greylock}$: A Python Package for Measuring The Composition of Complex Datasets","arxiv_id":"2401.00102","date":"2023-12-29","proceeding":null,"authors":["Phuc Nguyen","Rohit Arora","Elliot D. Hill","Jasper Braun","Alexandra Morgan","Liza M. Quintana","Gabrielle Mazzoni","Ghee Rye Lee","Rima Arnaout","Ramy Arnaout"],"abstract":"Machine-learning datasets are typically characterized by measuring their size and class balance. However, there exists a richer and potentially more useful set of measures, termed diversity measures, that incorporate elements' frequencies and between-element similarities. Although these have been available in the R and Julia programming languages for other applications, they have not been as readily available in Python, which is widely used for machine learning, and are not easily applied to machine-learning-sized datasets without special coding considerations. To address these issues, we developed $\\textit{greylock}$, a Python package that calculates diversity measures and is tailored to large datasets. $\\textit{greylock}$ can calculate any of the frequency-sensitive measures of Hill's D-number framework, and going beyond Hill, their similarity-sensitive counterparts (Greylock is a mountain). $\\textit{greylock}$ also outputs measures that compare datasets (beta diversities). We first briefly review the D-number framework, illustrating how it incorporates elements' frequencies and between-element similarities. We then describe $\\textit{greylock}$'s key features and usage. We end with several examples - immunomics, metagenomics, computational pathology, and medical imaging - illustrating $\\textit{greylock}$'s applicability across a range of dataset types and fields.","url_abs":"https://arxiv.org/abs/2401.00102v1","url_pdf":"https://arxiv.org/pdf/2401.00102v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"textit-greylock-a-python-package-for","repo_url":"https://github.com/arnaoutlab/diversity","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}