{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-10781","title":"Recovering the Zipfian Distribution in Unsupervised Term Discovery","arxiv_id":"2606.10781","date":"2026-06-09","proceeding":null,"authors":["Danel Slabbert","Simon Malan","Herman Kamper"],"abstract":"Unsupervised term discovery involves segmenting unlabelled speech into word- or syllable-like units and clustering these into a lexicon of candidate types. True lexicons follow a Zipfian distribution, yet the dominant centre-based clustering approach -- K-means -- produces a more uniform distribution due to an inductive bias toward spherical clusters. In this paper we revisit graph-based clustering as a bottom-up alternative, where segment embeddings are connected by pairwise similarity and partitioned using the Leiden algorithm. We show that graph clustering substantially outperforms centre-based approaches (K-means, GMM, BIRCH) in both word- and syllable-level lexicon discovery across three languages, producing more Zipf-like distributions. Another bottom-up approach, agglomerative clustering with average linkage, also performs well, although it is computationally less efficient and allows for less control over the resulting distribution. Our work calls into question the dominance of centre-based clustering for term discovery, and promotes graph clustering as an attractive alternative.","url_abs":"https://arxiv.org/abs/2606.10781","url_pdf":"https://arxiv.org/pdf/2606.10781","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2606.10781","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.10781"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/adendorffy/zipf-clus","reach":null}],"summary":{"ran":8},"by_repo_kind":{"found_in_text":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"655716a00beb765c","entry":"compute_edges_batch","repo":"adendorffy/zipf-clus","repo_kind":"found_in_text","path":"cosine_graph.py","file_url":"https://github.com/adendorffy/zipf-clus/blob/HEAD/cosine_graph.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"655716a00beb765c"}},{"code_sha256_prefix":"fd84ac2aae6da5e1","entry":"convert_labels_to_dict","repo":"adendorffy/zipf-clus","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/adendorffy/zipf-clus/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd84ac2aae6da5e1"}},{"code_sha256_prefix":"e7c64745ca79a222","entry":"draw","repo":"adendorffy/zipf-clus","repo_kind":"found_in_text","path":"bayes_gmm/fbgmm.py","file_url":"https://github.com/adendorffy/zipf-clus/blob/HEAD/bayes_gmm/fbgmm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7c64745ca79a222"}},{"code_sha256_prefix":"8472e0dc21084c9e","entry":"frames_to_seconds","repo":"adendorffy/zipf-clus","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/adendorffy/zipf-clus/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8472e0dc21084c9e"}},{"code_sha256_prefix":"084c1abcb3dbae5a","entry":"load_pooled_features","repo":"adendorffy/zipf-clus","repo_kind":"found_in_text","path":"pooling.py","file_url":"https://github.com/adendorffy/zipf-clus/blob/HEAD/pooling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"084c1abcb3dbae5a"}},{"code_sha256_prefix":"80c4fb8f560a5f26","entry":"load_quantised_segments","repo":"adendorffy/zipf-clus","repo_kind":"found_in_text","path":"quantiser.py","file_url":"https://github.com/adendorffy/zipf-clus/blob/HEAD/quantiser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80c4fb8f560a5f26"}},{"code_sha256_prefix":"97b8274e44f71c6e","entry":"log_post_pred_unvectorized","repo":"adendorffy/zipf-clus","repo_kind":"found_in_text","path":"bayes_gmm/gaussian_components.py","file_url":"https://github.com/adendorffy/zipf-clus/blob/HEAD/bayes_gmm/gaussian_components.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"97b8274e44f71c6e"}},{"code_sha256_prefix":"6bf6d3cd9e29fad5","entry":"seconds_to_frames","repo":"adendorffy/zipf-clus","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/adendorffy/zipf-clus/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6bf6d3cd9e29fad5"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}