{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lca-on-the-line-benchmarking-out-of","title":"LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies","arxiv_id":"2407.16067","date":"2024-07-22","proceeding":null,"authors":["Jia Shi","Gautam Gare","Jinjin Tian","Siqi Chai","Zhiqiu Lin","Arun Vasudevan","Di Feng","Francesco Ferroni","Shu Kong"],"abstract":"We tackle the challenge of predicting models' Out-of-Distribution (OOD) performance using in-distribution (ID) measurements without requiring OOD data. Existing evaluations with \"Effective Robustness\", which use ID accuracy as an indicator of OOD accuracy, encounter limitations when models are trained with diverse supervision and distributions, such as class labels (Vision Models, VMs, on ImageNet) and textual descriptions (Visual-Language Models, VLMs, on LAION). VLMs often generalize better to OOD data than VMs despite having similar or lower ID performance. To improve the prediction of models' OOD performance from ID measurements, we introduce the Lowest Common Ancestor (LCA)-on-the-Line framework. This approach revisits the established concept of LCA distance, which measures the hierarchical distance between labels and predictions within a predefined class hierarchy, such as WordNet. We assess 75 models using ImageNet as the ID dataset and five significantly shifted OOD variants, uncovering a strong linear correlation between ID LCA distance and OOD top-1 accuracy. Our method provides a compelling alternative for understanding why VLMs tend to generalize better. Additionally, we propose a technique to construct a taxonomic hierarchy on any dataset using K-means clustering, demonstrating that LCA distance is robust to the constructed taxonomic hierarchy. Moreover, we demonstrate that aligning model predictions with class taxonomies, through soft labels or prompt engineering, can enhance model generalization. Open source code in our Project Page: https://elvishelvis.github.io/papers/lca/.","url_abs":"https://arxiv.org/abs/2407.16067v1","url_pdf":"https://arxiv.org/pdf/2407.16067v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lca-on-the-line-benchmarking-out-of","repo_url":"https://github.com/elvishelvis/lca-on-the-line","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"out-of-distribution-generalization","task_name":"Out-of-Distribution Generalization"},{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2407.16067","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.16067"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/elvishelvis/lca-on-the-line","reach":null}],"summary":{"ran_honours":2,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"a553a0729c18f519","entry":"create_lca_matrix","repo":"elvishelvis/lca-on-the-line","repo_kind":"official","path":"create_hierarchy.py","file_url":"https://github.com/elvishelvis/lca-on-the-line/blob/HEAD/create_hierarchy.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a553a0729c18f519"}},{"code_sha256_prefix":"ca658df713e1859a","entry":"hierarchical_clustering","repo":"elvishelvis/lca-on-the-line","repo_kind":"official","path":"create_hierarchy.py","file_url":"https://github.com/elvishelvis/lca-on-the-line/blob/HEAD/create_hierarchy.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca658df713e1859a"}},{"code_sha256_prefix":"8460a056798aeee7","entry":"lca_alignment_loss","repo":"elvishelvis/lca-on-the-line","repo_kind":"official","path":"linear_probe_loss.py","file_url":"https://github.com/elvishelvis/lca-on-the-line/blob/HEAD/linear_probe_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8460a056798aeee7"}},{"code_sha256_prefix":"7c04913ae4face36","entry":"construct_latent_hierachy","repo":"elvishelvis/lca-on-the-line","repo_kind":"official","path":"create_hierarchy.py","file_url":"https://github.com/elvishelvis/lca-on-the-line/blob/HEAD/create_hierarchy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c04913ae4face36"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}