{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-neuro-symbolic-concept-learner-1","title":"The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision","arxiv_id":"1904.12584","date":"2019-04-26","proceeding":"ICLR 2019 5","authors":["Jiayuan Mao","Chuang Gan","Pushmeet Kohli","Joshua B. Tenenbaum","Jiajun Wu"],"abstract":"We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns\nvisual concepts, words, and semantic parsing of sentences without explicit\nsupervision on any of them; instead, our model learns by simply looking at\nimages and reading paired questions and answers. Our model builds an\nobject-based scene representation and translates sentences into executable,\nsymbolic programs. To bridge the learning of two modules, we use a\nneuro-symbolic reasoning module that executes these programs on the latent\nscene representation. Analogical to human concept learning, the perception\nmodule learns visual concepts based on the language description of the object\nbeing referred to. Meanwhile, the learned visual concepts facilitate learning\nnew words and parsing new sentences. We use curriculum learning to guide the\nsearching over the large compositional space of images and language. Extensive\nexperiments demonstrate the accuracy and efficiency of our model on learning\nvisual concepts, word representations, and semantic parsing of sentences.\nFurther, our method allows easy generalization to new object attributes,\ncompositions, language concepts, scenes and questions, and even new program\ndomains. It also empowers applications including visual question answering and\nbidirectional image-text retrieval.","url_abs":"http://arxiv.org/abs/1904.12584v1","url_pdf":"http://arxiv.org/pdf/1904.12584v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-neuro-symbolic-concept-learner-1","repo_url":"https://github.com/vacancy/NSCL-PyTorch-Release","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-neuro-symbolic-concept-learner-1","repo_url":"https://github.com/nerdimite/neuro-symbolic-ai-soc","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-text-retrieval","task_name":"Image-text Retrieval"},{"task_slug":"object","task_name":"Object"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"semantic-parsing","task_name":"Semantic Parsing"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-clevr","task":"Visual Question Answering (VQA)","dataset":"CLEVR","model":"NS-CL","rank_in_archive_order":6,"of":15,"metrics":{"Accuracy":"98.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1904.12584","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1904.12584"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vacancy/NSCL-PyTorch-Release","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nerdimite/neuro-symbolic-ai-soc","reach":{"status":"ok"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0e1e8fb4bf2a96d6","entry":"build_clevr_dataset","repo":"vacancy/NSCL-PyTorch-Release","repo_kind":"official","path":"nscl/datasets/clevr/definition.py","file_url":"https://github.com/vacancy/NSCL-PyTorch-Release/blob/HEAD/nscl/datasets/clevr/definition.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e1e8fb4bf2a96d6"}},{"code_sha256_prefix":"9829bd711c3852e1","entry":"build_concept_quantization_clevr_dataset","repo":"vacancy/NSCL-PyTorch-Release","repo_kind":"official","path":"nscl/datasets/clevr/definition.py","file_url":"https://github.com/vacancy/NSCL-PyTorch-Release/blob/HEAD/nscl/datasets/clevr/definition.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9829bd711c3852e1"}},{"code_sha256_prefix":"b7ded7965050e458","entry":"build_concept_retrieval_clevr_dataset","repo":"vacancy/NSCL-PyTorch-Release","repo_kind":"official","path":"nscl/datasets/clevr/definition.py","file_url":"https://github.com/vacancy/NSCL-PyTorch-Release/blob/HEAD/nscl/datasets/clevr/definition.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b7ded7965050e458"}},{"code_sha256_prefix":"4d7c9bd38decfc39","entry":"clevr_to_nsclseq","repo":"vacancy/NSCL-PyTorch-Release","repo_kind":"official","path":"nscl/datasets/clevr/program_translator.py","file_url":"https://github.com/vacancy/NSCL-PyTorch-Release/blob/HEAD/nscl/datasets/clevr/program_translator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d7c9bd38decfc39"}},{"code_sha256_prefix":"0c3d9ae18dee8195","entry":"get_clevr_op_attribute","repo":"vacancy/NSCL-PyTorch-Release","repo_kind":"official","path":"nscl/datasets/clevr/program_translator.py","file_url":"https://github.com/vacancy/NSCL-PyTorch-Release/blob/HEAD/nscl/datasets/clevr/program_translator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c3d9ae18dee8195"}},{"code_sha256_prefix":"52682558c8a4e93d","entry":"get_clevr_pblock_op","repo":"vacancy/NSCL-PyTorch-Release","repo_kind":"official","path":"nscl/datasets/clevr/program_translator.py","file_url":"https://github.com/vacancy/NSCL-PyTorch-Release/blob/HEAD/nscl/datasets/clevr/program_translator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52682558c8a4e93d"}},{"code_sha256_prefix":"30208acaa5238ae7","entry":"make_positions","repo":"vacancy/NSCL-PyTorch-Release","repo_kind":"official","path":"nscl/nn/embedding.py","file_url":"https://github.com/vacancy/NSCL-PyTorch-Release/blob/HEAD/nscl/nn/embedding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30208acaa5238ae7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}