{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-17299","title":"Exploring Structural Degradation in Dense Representations for Self-supervised Learning","arxiv_id":"2510.17299","date":"2025-10-20","proceeding":"NeurIPS","authors":["Siran Dai","Qianqian Xu","Peisong Wen","Yang Liu","Qingming Huang"],"abstract":"In this work, we observe a counterintuitive phenomenon in self-supervised learning (SSL): longer training may impair the performance of dense prediction tasks (e.g., semantic segmentation). We refer to this phenomenon as Self-supervised Dense Degradation (SDD) and demonstrate its consistent presence across sixteen state-of-the-art SSL methods with various losses, architectures, and datasets. When the model performs suboptimally on dense tasks at the end of training, measuring the performance during training becomes essential. However, evaluating dense performance effectively without annotations remains an open challenge. To tackle this issue, we introduce a Dense representation Structure Estimator (DSE), composed of a class-relevance measure and an effective dimensionality measure. The proposed DSE is both theoretically grounded and empirically validated to be closely correlated with the downstream performance. Based on this metric, we introduce a straightforward yet effective model selection strategy and a DSE-based regularization method. Experiments on sixteen SSL methods across four benchmarks confirm that model selection improves mIoU by $3.0\\%$ on average with negligible computational cost. Additionally, DSE regularization consistently mitigates the effects of dense degradation. Code is available at https://github.com/EldercatSAM/SSL-Degradation.","url_abs":"https://arxiv.org/abs/2510.17299","url_pdf":"https://arxiv.org/pdf/2510.17299","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2510.17299","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.17299"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/EldercatSAM/SSL-Degradation","reach":null}],"summary":{"ran_fixture":2,"ran_draft_wrong":3},"by_repo_kind":{"found_in_text":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"7b1d314303be8de3","entry":"compute_DSE","repo":"EldercatSAM/SSL-Degradation","repo_kind":"found_in_text","path":"estimator.py","file_url":"https://github.com/EldercatSAM/SSL-Degradation/blob/HEAD/estimator.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b1d314303be8de3"}},{"code_sha256_prefix":"9b276701aab17f57","entry":"estimator","repo":"EldercatSAM/SSL-Degradation","repo_kind":"found_in_text","path":"estimator.py","file_url":"https://github.com/EldercatSAM/SSL-Degradation/blob/HEAD/estimator.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b276701aab17f57"}},{"code_sha256_prefix":"0f6d189d67dd587e","entry":"kmeans","repo":"EldercatSAM/SSL-Degradation","repo_kind":"found_in_text","path":"estimator.py","file_url":"https://github.com/EldercatSAM/SSL-Degradation/blob/HEAD/estimator.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f6d189d67dd587e"}},{"code_sha256_prefix":"097f4c464d51be25","entry":"kmeans_plus_plus_init","repo":"EldercatSAM/SSL-Degradation","repo_kind":"found_in_text","path":"estimator.py","file_url":"https://github.com/EldercatSAM/SSL-Degradation/blob/HEAD/estimator.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"097f4c464d51be25"}},{"code_sha256_prefix":"b81c10e5d661ea21","entry":"kmeans_pp_with_early_stop","repo":"EldercatSAM/SSL-Degradation","repo_kind":"found_in_text","path":"estimator.py","file_url":"https://github.com/EldercatSAM/SSL-Degradation/blob/HEAD/estimator.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b81c10e5d661ea21"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CV","source":"arxiv_api"},"syntology_extracted_results":null}