{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/label-free-model-evaluation-with-semi","title":"Label-Free Model Evaluation with Semi-Structured Dataset Representations","arxiv_id":"2112.00694","date":"2021-12-01","proceeding":null,"authors":["Xiaoxiao Sun","Yunzhong Hou","Hongdong Li","Liang Zheng"],"abstract":"Label-free model evaluation, or AutoEval, estimates model accuracy on unlabeled test sets, and is critical for understanding model behaviors in various unseen environments. In the absence of image labels, based on dataset representations, we estimate model performance for AutoEval with regression. On the one hand, image feature is a straightforward choice for such representations, but it hampers regression learning due to being unstructured (\\ie no specific meanings for component at certain location) and of large-scale. On the other hand, previous methods adopt simple structured representations (like average confidence or average feature), but insufficient to capture the data characteristics given their limited dimensions. In this work, we take the best of both worlds and propose a new semi-structured dataset representation that is manageable for regression learning while containing rich information for AutoEval. Based on image features, we integrate distribution shapes, clusters, and representative samples for a semi-structured dataset representation. Besides the structured overall description with distribution shapes, the unstructured description with clusters and representative samples include additional fine-grained information facilitating the AutoEval task. On three existing datasets and 25 newly introduced ones, we experimentally show that the proposed representation achieves competitive results. Code and dataset are available at https://github.com/sxzrt/Semi-Structured-Dataset-Representations.","url_abs":"https://arxiv.org/abs/2112.00694v1","url_pdf":"https://arxiv.org/pdf/2112.00694v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"label-free-model-evaluation-with-semi","repo_url":"https://github.com/sxzrt/semi-structured-dataset-representations","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"regression-1","task_name":"regression"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2112.00694","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2112.00694"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sxzrt/semi-structured-dataset-representations","reach":null}],"summary":{"ran_honours":2,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"c9aa42afcc694d93","entry":"cal_pairwise_dist","repo":"sxzrt/semi-structured-dataset-representations","repo_kind":"official","path":"Set_rep/get_set_representation.py","file_url":"https://github.com/sxzrt/semi-structured-dataset-representations/blob/HEAD/Set_rep/get_set_representation.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c9aa42afcc694d93"}},{"code_sha256_prefix":"59678b193fa7196c","entry":"getGreedyPerm","repo":"sxzrt/semi-structured-dataset-representations","repo_kind":"official","path":"Set_rep/get_set_representation.py","file_url":"https://github.com/sxzrt/semi-structured-dataset-representations/blob/HEAD/Set_rep/get_set_representation.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59678b193fa7196c"}},{"code_sha256_prefix":"0def50a351111624","entry":"calculate_frechet_distance","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"0def50a351111624"}},{"code_sha256_prefix":"ea2853d78b6caf15","entry":"calculate_frechet_distance","repo":"sxzrt/semi-structured-dataset-representations","repo_kind":"official","path":"Set_rep/test_regnet_new.py","file_url":"https://github.com/sxzrt/semi-structured-dataset-representations/blob/HEAD/Set_rep/test_regnet_new.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea2853d78b6caf15"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}