{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/understanding-representation-learnability-of","title":"Understanding Representation Learnability of Nonlinear Self-Supervised Learning","arxiv_id":"2401.03214","date":"2024-01-06","proceeding":null,"authors":["Ruofeng Yang","Xiangyuan Li","Bo Jiang","Shuai Li"],"abstract":"Self-supervised learning (SSL) has empirically shown its data representation learnability in many downstream tasks. There are only a few theoretical works on data representation learnability, and many of those focus on final data representation, treating the nonlinear neural network as a ``black box\". However, the accurate learning results of neural networks are crucial for describing the data distribution features learned by SSL models. Our paper is the first to analyze the learning results of the nonlinear SSL model accurately. We consider a toy data distribution that contains two features: the label-related feature and the hidden feature. Unlike previous linear setting work that depends on closed-form solutions, we use the gradient descent algorithm to train a 1-layer nonlinear SSL model with a certain initialization region and prove that the model converges to a local minimum. Furthermore, different from the complex iterative analysis, we propose a new analysis process which uses the exact version of Inverse Function Theorem to accurately describe the features learned by the local minimum. With this local minimum, we prove that the nonlinear SSL model can capture the label-related feature and hidden feature at the same time. In contrast, the nonlinear supervised learning (SL) model can only learn the label-related feature. We also present the learning processes and results of the nonlinear SSL and SL model via simulation experiments.","url_abs":"https://arxiv.org/abs/2401.03214v1","url_pdf":"https://arxiv.org/pdf/2401.03214v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understanding-representation-learnability-of","repo_url":"https://github.com/wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.03214","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.03214"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL","reach":{"status":"ok"}}],"summary":{"ran_violates":1,"ran":8},"by_repo_kind":{"official":{"samples":9,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"b8e95809ca2c17c9","entry":"sigmoid","repo":"wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL","repo_kind":"official","path":"SSL_update_process.py","file_url":"https://github.com/wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL/blob/HEAD/SSL_update_process.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b8e95809ca2c17c9"}},{"code_sha256_prefix":"294f6cdffc50e80d","entry":"cal_w1w2","repo":"wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL","repo_kind":"official","path":"Figure_learning_results.py","file_url":"https://github.com/wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL/blob/HEAD/Figure_learning_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"294f6cdffc50e80d"}},{"code_sha256_prefix":"31ae9446064e73c7","entry":"dsigmoid","repo":"wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL","repo_kind":"official","path":"SSL_update_process.py","file_url":"https://github.com/wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL/blob/HEAD/SSL_update_process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31ae9446064e73c7"}},{"code_sha256_prefix":"b837be6cc1c7bde3","entry":"get_data","repo":"wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL","repo_kind":"official","path":"Figure_learning_curve.py","file_url":"https://github.com/wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL/blob/HEAD/Figure_learning_curve.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b837be6cc1c7bde3"}},{"code_sha256_prefix":"1e9bd09432bde7ec","entry":"initialization","repo":"wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL","repo_kind":"official","path":"SSL_update_process.py","file_url":"https://github.com/wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL/blob/HEAD/SSL_update_process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e9bd09432bde7ec"}},{"code_sha256_prefix":"52f548e68a2c9f36","entry":"initialization_W_SL","repo":"wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL","repo_kind":"official","path":"SL_update_process.py","file_url":"https://github.com/wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL/blob/HEAD/SL_update_process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"52f548e68a2c9f36"}},{"code_sha256_prefix":"2766ef4e574a9e54","entry":"mean_std_plot","repo":"wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL","repo_kind":"official","path":"Figure_learning_curve.py","file_url":"https://github.com/wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL/blob/HEAD/Figure_learning_curve.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2766ef4e574a9e54"}},{"code_sha256_prefix":"9fef9cb515e19b02","entry":"square_l2_norm","repo":"wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL","repo_kind":"official","path":"Figure_learning_results.py","file_url":"https://github.com/wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL/blob/HEAD/Figure_learning_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9fef9cb515e19b02"}},{"code_sha256_prefix":"4e3fa3873760f767","entry":"w1w2theta","repo":"wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL","repo_kind":"official","path":"Figure_learning_results.py","file_url":"https://github.com/wanshuiyin/AAAI-2023-The-Learnability-of-Nonlinear-SSL/blob/HEAD/Figure_learning_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e3fa3873760f767"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}