{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/localized-symbolic-knowledge-distillation-for-1","title":"Localized Symbolic Knowledge Distillation for Visual Commonsense Models","arxiv_id":"2312.04837","date":"2023-12-08","proceeding":"NeurIPS 2023 11","authors":["Jae Sung Park","Jack Hessel","Khyathi Raghavi Chandu","Paul Pu Liang","Ximing Lu","Peter West","Youngjae Yu","Qiuyuan Huang","Jianfeng Gao","Ali Farhadi","Yejin Choi"],"abstract":"Instruction following vision-language (VL) models offer a flexible interface that supports a broad range of multimodal tasks in a zero-shot fashion. However, interfaces that operate on full images do not directly enable the user to \"point to\" and access specific regions within images. This capability is important not only to support reference-grounded VL benchmarks, but also, for practical applications that require precise within-image reasoning. We build Localized Visual Commonsense models, which allow users to specify (multiple) regions as input. We train our model by sampling localized commonsense knowledge from a large language model (LLM): specifically, we prompt an LLM to collect commonsense knowledge given a global literal image description and a local literal region description automatically generated by a set of VL models. With a separately trained critic model that selects high-quality examples, we find that training on the localized commonsense corpus can successfully distill existing VL models to support a reference-as-input interface. Empirical results and human evaluations in a zero-shot setup demonstrate that our distillation method results in more precise VL models of reasoning compared to a baseline of passing a generated referring expression to an LLM.","url_abs":"https://arxiv.org/abs/2312.04837v2","url_pdf":"https://arxiv.org/pdf/2312.04837v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"localized-symbolic-knowledge-distillation-for-1","repo_url":"https://github.com/jamespark3922/localized-skd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"localized-symbolic-knowledge-distillation-for-1","repo_url":"https://github.com/jamespark3922/lskd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":null,"task_name":"Image Description"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"referring-expression","task_name":"Referring Expression"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.04837","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.04837"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jamespark3922/lskd","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jamespark3922/localized-skd","reach":null}],"summary":{"ran":7,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"56f02812d66a0d11","entry":"generate_caption","repo":"jamespark3922/lskd","repo_kind":"official","path":"app/caption.py","file_url":"https://github.com/jamespark3922/lskd/blob/HEAD/app/caption.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"56f02812d66a0d11"}},{"code_sha256_prefix":"7d214655954e2bc6","entry":"getAttMap","repo":"jamespark3922/localized-skd","repo_kind":"official","path":"lavis/common/gradcam.py","file_url":"https://github.com/jamespark3922/localized-skd/blob/HEAD/lavis/common/gradcam.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"7d214655954e2bc6"}},{"code_sha256_prefix":"92f2e16ee0d3a24d","entry":"get_concat_v","repo":"jamespark3922/lskd","repo_kind":"official","path":"app/dataset_browser.py","file_url":"https://github.com/jamespark3922/lskd/blob/HEAD/app/dataset_browser.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"92f2e16ee0d3a24d"}},{"code_sha256_prefix":"13a43a7d815937e6","entry":"read_img","repo":"jamespark3922/lskd","repo_kind":"official","path":"app/calculate_coco_features.py","file_url":"https://github.com/jamespark3922/lskd/blob/HEAD/app/calculate_coco_features.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"13a43a7d815937e6"}},{"code_sha256_prefix":"926bb38f979a61ef","entry":"resize_img","repo":"jamespark3922/lskd","repo_kind":"official","path":"app/utils.py","file_url":"https://github.com/jamespark3922/lskd/blob/HEAD/app/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"926bb38f979a61ef"}},{"code_sha256_prefix":"f809f2ecf34c1c99","entry":"resize_img_w","repo":"jamespark3922/lskd","repo_kind":"official","path":"app/dataset_browser.py","file_url":"https://github.com/jamespark3922/lskd/blob/HEAD/app/dataset_browser.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f809f2ecf34c1c99"}},{"code_sha256_prefix":"3d4bbb5ca53fd24a","entry":"sample_dataset","repo":"jamespark3922/lskd","repo_kind":"official","path":"app/dataset_browser.py","file_url":"https://github.com/jamespark3922/lskd/blob/HEAD/app/dataset_browser.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"3d4bbb5ca53fd24a"}},{"code_sha256_prefix":"cb33571427334815","entry":"tile","repo":"jamespark3922/lskd","repo_kind":"official","path":"lavis/models/base_model.py","file_url":"https://github.com/jamespark3922/lskd/blob/HEAD/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"cb33571427334815"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"jamespark3922/lskd","repo_kind":"official","path":"lavis/models/base_model.py","file_url":"https://github.com/jamespark3922/lskd/blob/HEAD/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"3e12ea8c09674aa8","entry":"compute_gradcam_batch","repo":"jamespark3922/lskd","repo_kind":"official","path":"app/multimodal_search.py","file_url":"https://github.com/jamespark3922/lskd/blob/HEAD/app/multimodal_search.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"3e12ea8c09674aa8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}