{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/language-informed-visual-concept-learning","title":"Language-Informed Visual Concept Learning","arxiv_id":"2312.03587","date":"2023-12-06","proceeding":null,"authors":["Sharon Lee","Yunzhi Zhang","Shangzhe Wu","Jiajun Wu"],"abstract":"Our understanding of the visual world is centered around various concept axes, characterizing different aspects of visual entities. While different concept axes can be easily specified by language, e.g. color, the exact visual nuances along each axis often exceed the limitations of linguistic articulations, e.g. a particular style of painting. In this work, our goal is to learn a language-informed visual concept representation, by simply distilling large pre-trained vision-language models. Specifically, we train a set of concept encoders to encode the information pertinent to a set of language-informed concept axes, with an objective of reproducing the input image through a pre-trained Text-to-Image (T2I) model. To encourage better disentanglement of different concept encoders, we anchor the concept embeddings to a set of text embeddings obtained from a pre-trained Visual Question Answering (VQA) model. At inference time, the model extracts concept embeddings along various axes from new test images, which can be remixed to generate images with novel compositions of visual concepts. With a lightweight test-time finetuning procedure, it can also generalize to novel concepts unseen at training.","url_abs":"https://arxiv.org/abs/2312.03587v2","url_pdf":"https://arxiv.org/pdf/2312.03587v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"language-informed-visual-concept-learning","repo_url":"https://github.com/sharonal10/langint","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"disentanglement","task_name":"Disentanglement"},{"task_slug":"novel-concepts","task_name":"Novel Concepts"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.03587","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.03587"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sharonal10/langint","reach":{"status":"ok"}}],"summary":{"ran":10,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":13,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"ade563988ef3b47d","entry":"build_variable_level_templates","repo":"sharonal10/langint","repo_kind":"official","path":"langint/utils/dataset.py","file_url":"https://github.com/sharonal10/langint/blob/HEAD/langint/utils/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ade563988ef3b47d"}},{"code_sha256_prefix":"93d28ca67303e04b","entry":"clip_magnitude","repo":"sharonal10/langint","repo_kind":"official","path":"tu/tu/configs.py","file_url":"https://github.com/sharonal10/langint/blob/HEAD/tu/tu/configs.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"93d28ca67303e04b"}},{"code_sha256_prefix":"90a904ea4b63f324","entry":"get_attrdict","repo":"sharonal10/langint","repo_kind":"official","path":"tu/tu/configs.py","file_url":"https://github.com/sharonal10/langint/blob/HEAD/tu/tu/configs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90a904ea4b63f324"}},{"code_sha256_prefix":"c2e55ad99d31aa2b","entry":"get_distributed_dataloader","repo":"sharonal10/langint","repo_kind":"official","path":"tu/tu/ddp.py","file_url":"https://github.com/sharonal10/langint/blob/HEAD/tu/tu/ddp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c2e55ad99d31aa2b"}},{"code_sha256_prefix":"5bead0c66171eac5","entry":"get_distributed_model","repo":"sharonal10/langint","repo_kind":"official","path":"tu/tu/ddp.py","file_url":"https://github.com/sharonal10/langint/blob/HEAD/tu/tu/ddp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5bead0c66171eac5"}},{"code_sha256_prefix":"4fb10880f0fad299","entry":"glide_sample_prompt","repo":"sharonal10/langint","repo_kind":"official","path":"langint/datasets/glide.py","file_url":"https://github.com/sharonal10/langint/blob/HEAD/langint/datasets/glide.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4fb10880f0fad299"}},{"code_sha256_prefix":"f73c48357fb0fb9a","entry":"glide_sample_prompts","repo":"sharonal10/langint","repo_kind":"official","path":"langint/datasets/glide.py","file_url":"https://github.com/sharonal10/langint/blob/HEAD/langint/datasets/glide.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f73c48357fb0fb9a"}},{"code_sha256_prefix":"9910e2fc297f8665","entry":"identity","repo":"sharonal10/langint","repo_kind":"official","path":"tu/tu/configs.py","file_url":"https://github.com/sharonal10/langint/blob/HEAD/tu/tu/configs.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9910e2fc297f8665"}},{"code_sha256_prefix":"198460b1eb7db45d","entry":"master_only","repo":"sharonal10/langint","repo_kind":"official","path":"tu/tu/ddp.py","file_url":"https://github.com/sharonal10/langint/blob/HEAD/tu/tu/ddp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"198460b1eb7db45d"}},{"code_sha256_prefix":"e96992b0551c7e3b","entry":"perform_kmeans","repo":"sharonal10/langint","repo_kind":"official","path":"langint/utils/clip_clusters.py","file_url":"https://github.com/sharonal10/langint/blob/HEAD/langint/utils/clip_clusters.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e96992b0551c7e3b"}},{"code_sha256_prefix":"57976cb3bd9fd03c","entry":"separate_clusters","repo":"sharonal10/langint","repo_kind":"official","path":"langint/utils/clip_clusters.py","file_url":"https://github.com/sharonal10/langint/blob/HEAD/langint/utils/clip_clusters.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57976cb3bd9fd03c"}},{"code_sha256_prefix":"7be381c916d43757","entry":"deepfloyd_sample_prompts","repo":"sharonal10/langint","repo_kind":"official","path":"langint/datasets/glide.py","file_url":"https://github.com/sharonal10/langint/blob/HEAD/langint/datasets/glide.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7be381c916d43757"}},{"code_sha256_prefix":"ddb37c6af0e280d4","entry":"get_clip_features","repo":"sharonal10/langint","repo_kind":"official","path":"langint/utils/clip_clusters.py","file_url":"https://github.com/sharonal10/langint/blob/HEAD/langint/utils/clip_clusters.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ddb37c6af0e280d4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}