{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/md-tree-a-model-diagnostic-tree-grown-on-loss","title":"MD tree: a model-diagnostic tree grown on loss landscape","arxiv_id":"2406.16988","date":"2024-06-24","proceeding":null,"authors":["Yefan Zhou","Jianlong Chen","Qinxue Cao","Konstantin Schürholt","Yaoqing Yang"],"abstract":"This paper considers \"model diagnosis\", which we formulate as a classification problem. Given a pre-trained neural network (NN), the goal is to predict the source of failure from a set of failure modes (such as a wrong hyperparameter, inadequate model size, and insufficient data) without knowing the training configuration of the pre-trained NN. The conventional diagnosis approach uses training and validation errors to determine whether the model is underfitting or overfitting. However, we show that rich information about NN performance is encoded in the optimization loss landscape, which provides more actionable insights than validation-based measurements. Therefore, we propose a diagnosis method called MD tree based on loss landscape metrics and experimentally demonstrate its advantage over classical validation-based approaches. We verify the effectiveness of MD tree in multiple practical scenarios: (1) use several models trained on one dataset to diagnose a model trained on another dataset, essentially a few-shot dataset transfer problem; (2) use small models (or models trained with small data) to diagnose big models (or models trained with big data), essentially a scale transfer problem. In a dataset transfer task, MD tree achieves an accuracy of 87.7%, outperforming validation-based approaches by 14.88%. Our code is available at https://github.com/YefanZhou/ModelDiagnosis.","url_abs":"https://arxiv.org/abs/2406.16988v1","url_pdf":"https://arxiv.org/pdf/2406.16988v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"md-tree-a-model-diagnostic-tree-grown-on-loss","repo_url":"https://github.com/yefanzhou/modeldiagnosis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.16988","atlas_url":"https://app.syntology.ai/?focus=2406.16988","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.16988"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/YefanZhou/ModelDiagnosis","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yefanzhou/modeldiagnosis","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"ran_fixture":9,"unverified":2},"by_repo_kind":{"official":{"samples":17,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"92299c1945663652","entry":"assign_data_batch_label","repo":"YefanZhou/ModelDiagnosis","repo_kind":"official","path":"utils/hyper_failure_utils.py","file_url":"https://github.com/YefanZhou/ModelDiagnosis/blob/HEAD/utils/hyper_failure_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92299c1945663652"}},{"code_sha256_prefix":"6c071983849eb76c","entry":"assign_temp_value","repo":"YefanZhou/ModelDiagnosis","repo_kind":"official","path":"utils/hyper_failure_utils.py","file_url":"https://github.com/YefanZhou/ModelDiagnosis/blob/HEAD/utils/hyper_failure_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c071983849eb76c"}},{"code_sha256_prefix":"8400f8f798b63074","entry":"assign_width_batch_label","repo":"YefanZhou/ModelDiagnosis","repo_kind":"official","path":"utils/hyper_failure_utils.py","file_url":"https://github.com/YefanZhou/ModelDiagnosis/blob/HEAD/utils/hyper_failure_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8400f8f798b63074"}},{"code_sha256_prefix":"0cf3649fb980c8ee","entry":"baseline_diagnoise","repo":"YefanZhou/ModelDiagnosis","repo_kind":"official","path":"one_step/one_step_hpo_utils.py","file_url":"https://github.com/YefanZhou/ModelDiagnosis/blob/HEAD/one_step/one_step_hpo_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0cf3649fb980c8ee"}},{"code_sha256_prefix":"372c749783ab6674","entry":"get_metric_file","repo":"YefanZhou/ModelDiagnosis","repo_kind":"official","path":"utils/data.py","file_url":"https://github.com/YefanZhou/ModelDiagnosis/blob/HEAD/utils/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"372c749783ab6674"}},{"code_sha256_prefix":"f923e268a40605e4","entry":"get_metric_val","repo":"YefanZhou/ModelDiagnosis","repo_kind":"official","path":"utils/data.py","file_url":"https://github.com/YefanZhou/ModelDiagnosis/blob/HEAD/utils/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f923e268a40605e4"}},{"code_sha256_prefix":"e8741f72ae5726ea","entry":"objective_data","repo":"yefanzhou/modeldiagnosis","repo_kind":"official","path":"method/md_tree.py","file_url":"https://github.com/yefanzhou/modeldiagnosis/blob/HEAD/method/md_tree.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8741f72ae5726ea"}},{"code_sha256_prefix":"18e31b2fb6dd638a","entry":"objective_temp","repo":"yefanzhou/modeldiagnosis","repo_kind":"official","path":"method/md_tree.py","file_url":"https://github.com/yefanzhou/modeldiagnosis/blob/HEAD/method/md_tree.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18e31b2fb6dd638a"}},{"code_sha256_prefix":"821904534ab3b21b","entry":"objective_temp_cka","repo":"yefanzhou/modeldiagnosis","repo_kind":"official","path":"method/md_tree.py","file_url":"https://github.com/yefanzhou/modeldiagnosis/blob/HEAD/method/md_tree.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"821904534ab3b21b"}},{"code_sha256_prefix":"516ef9a96aecdc93","entry":"objective_width","repo":"yefanzhou/modeldiagnosis","repo_kind":"official","path":"method/md_tree.py","file_url":"https://github.com/yefanzhou/modeldiagnosis/blob/HEAD/method/md_tree.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"516ef9a96aecdc93"}},{"code_sha256_prefix":"c87846c634884e4c","entry":"predictor_data_temp","repo":"yefanzhou/modeldiagnosis","repo_kind":"official","path":"method/md_tree.py","file_url":"https://github.com/yefanzhou/modeldiagnosis/blob/HEAD/method/md_tree.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c87846c634884e4c"}},{"code_sha256_prefix":"32e14b47fa143ba0","entry":"predictor_temp","repo":"yefanzhou/modeldiagnosis","repo_kind":"official","path":"method/md_tree.py","file_url":"https://github.com/yefanzhou/modeldiagnosis/blob/HEAD/method/md_tree.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32e14b47fa143ba0"}},{"code_sha256_prefix":"94e52e9951bfd64f","entry":"predictor_temp_w_cka","repo":"yefanzhou/modeldiagnosis","repo_kind":"official","path":"method/md_tree.py","file_url":"https://github.com/yefanzhou/modeldiagnosis/blob/HEAD/method/md_tree.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94e52e9951bfd64f"}},{"code_sha256_prefix":"c494fe11b676fc77","entry":"predictor_width_temp","repo":"yefanzhou/modeldiagnosis","repo_kind":"official","path":"method/md_tree.py","file_url":"https://github.com/yefanzhou/modeldiagnosis/blob/HEAD/method/md_tree.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c494fe11b676fc77"}},{"code_sha256_prefix":"64408f31933c8ad0","entry":"sample_train_set","repo":"yefanzhou/modeldiagnosis","repo_kind":"official","path":"method/md_tree.py","file_url":"https://github.com/yefanzhou/modeldiagnosis/blob/HEAD/method/md_tree.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"64408f31933c8ad0"}},{"code_sha256_prefix":"f78fc3e8ac9740fa","entry":"evaluate_mdtree","repo":"yefanzhou/modeldiagnosis","repo_kind":"official","path":"method/md_tree.py","file_url":"https://github.com/yefanzhou/modeldiagnosis/blob/HEAD/method/md_tree.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f78fc3e8ac9740fa"}},{"code_sha256_prefix":"977e8ff87e42d094","entry":"preprocess","repo":"YefanZhou/ModelDiagnosis","repo_kind":"official","path":"utils/data.py","file_url":"https://github.com/YefanZhou/ModelDiagnosis/blob/HEAD/utils/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"977e8ff87e42d094"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}