{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vlg-cbm-training-concept-bottleneck-models","title":"VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance","arxiv_id":"2408.01432","date":"2024-07-18","proceeding":null,"authors":["Divyansh Srivastava","Ge Yan","Tsui-Wei Weng"],"abstract":"Concept Bottleneck Models (CBMs) provide interpretable prediction by introducing an intermediate Concept Bottleneck Layer (CBL), which encodes human-understandable concepts to explain models' decision. Recent works proposed to utilize Large Language Models and pre-trained Vision-Language Models to automate the training of CBMs, making it more scalable and automated. However, existing approaches still fall short in two aspects: First, the concepts predicted by CBL often mismatch the input image, raising doubts about the faithfulness of interpretation. Second, it has been shown that concept values encode unintended information: even a set of random concepts could achieve comparable test accuracy to state-of-the-art CBMs. To address these critical limitations, in this work, we propose a novel framework called Vision-Language-Guided Concept Bottleneck Model (VLG-CBM) to enable faithful interpretability with the benefits of boosted performance. Our method leverages off-the-shelf open-domain grounded object detectors to provide visually grounded concept annotation, which largely enhances the faithfulness of concept prediction while further improving the model performance. In addition, we propose a new metric called Number of Effective Concepts (NEC) to control the information leakage and provide better interpretability. Extensive evaluations across five standard benchmarks show that our method, VLG-CBM, outperforms existing methods by at least 4.27% and up to 51.09% on Accuracy at NEC=5 (denoted as ANEC-5), and by at least 0.45% and up to 29.78% on average accuracy (denoted as ANEC-avg), while preserving both faithfulness and interpretability of the learned concepts as demonstrated in extensive experiments.","url_abs":"https://arxiv.org/abs/2408.01432v3","url_pdf":"https://arxiv.org/pdf/2408.01432v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vlg-cbm-training-concept-bottleneck-models","repo_url":"https://github.com/trustworthy-ml-lab/vlg-cbm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"Avg"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.01432","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.01432"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/trustworthy-ml-lab/vlg-cbm","reach":null}],"summary":{"ran_draft_wrong":5,"ran":6,"ran_honours":1,"unverified":7},"by_repo_kind":{"found_in_text":{"samples":19,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":19,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"clip/simple_tokenizer.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"cc642412d4347cce","entry":"cos_similarity","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"similarity.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/similarity.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc642412d4347cce"}},{"code_sha256_prefix":"977ef28dbc4a682a","entry":"format_value","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"plots.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/plots.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"977ef28dbc4a682a"}},{"code_sha256_prefix":"680ec5c6103dc854","entry":"get_activation","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"680ec5c6103dc854"}},{"code_sha256_prefix":"406014b884a2aaa7","entry":"get_device","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"glm_saga/elasticnet.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/glm_saga/elasticnet.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"406014b884a2aaa7"}},{"code_sha256_prefix":"48f671cc09cab33f","entry":"get_init_conceptnet","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"conceptset_utils.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/conceptset_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48f671cc09cab33f"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"clip/simple_tokenizer.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"c6a6eb6a488f7d1a","entry":"get_save_names","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6a6eb6a488f7d1a"}},{"code_sha256_prefix":"192f187efaa61625","entry":"get_similarity_from_activations","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"192f187efaa61625"}},{"code_sha256_prefix":"00efcf0359457b86","entry":"remove_too_long","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"conceptset_utils.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/conceptset_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00efcf0359457b86"}},{"code_sha256_prefix":"bfe4b187a1c5aeaf","entry":"soft_threshold","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"glm_saga/elasticnet.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/glm_saga/elasticnet.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bfe4b187a1c5aeaf"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"clip/simple_tokenizer.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"7798dffe63b4a970","entry":"add_index_to_dataloader","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"glm_saga/elasticnet.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/glm_saga/elasticnet.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7798dffe63b4a970"}},{"code_sha256_prefix":"3662e54c46137680","entry":"build_model","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"clip/model.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3662e54c46137680"}},{"code_sha256_prefix":"892d44d97047a10d","entry":"cos_similarity_cubed","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"similarity.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/similarity.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"892d44d97047a10d"}},{"code_sha256_prefix":"cf659682a960fdc0","entry":"cos_similarity_cubed_single","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"similarity.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/similarity.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cf659682a960fdc0"}},{"code_sha256_prefix":"50160a0b071c120c","entry":"get_data","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"data_utils.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50160a0b071c120c"}},{"code_sha256_prefix":"111e91ef3f6b1e38","entry":"get_targets_only","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"data_utils.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"111e91ef3f6b1e38"}},{"code_sha256_prefix":"b91b981015bbf498","entry":"load","repo":"Trustworthy-ML-Lab/Label-free-CBM","repo_kind":"found_in_text","path":"clip/clip.py","file_url":"https://github.com/Trustworthy-ML-Lab/Label-free-CBM/blob/HEAD/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b91b981015bbf498"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}