{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/curi-a-benchmark-for-productive-concept-1","title":"CURI: A Benchmark for Productive Concept Learning Under Uncertainty","arxiv_id":"2010.02855","date":"2020-10-06","proceeding":null,"authors":["Ramakrishna Vedantam","Arthur Szlam","Maximilian Nickel","Ari Morcos","Brenden Lake"],"abstract":"Humans can learn and reason under substantial uncertainty in a space of infinitely many concepts, including structured relational concepts (\"a scene with objects that have the same color\") and ad-hoc categories defined through goals (\"objects that could fall on one's head\"). In contrast, standard classification benchmarks: 1) consider only a fixed set of category labels, 2) do not evaluate compositional concept learning and 3) do not explicitly capture a notion of reasoning under uncertainty. We introduce a new few-shot, meta-learning benchmark, Compositional Reasoning Under Uncertainty (CURI) to bridge this gap. CURI evaluates different aspects of productive and systematic generalization, including abstract understandings of disentangling, productive generalization, learning boolean operations, variable binding, etc. Importantly, it also defines a model-independent \"compositionality gap\" to evaluate the difficulty of generalizing out-of-distribution along each of these axes. Extensive evaluations across a range of modeling choices spanning different modalities (image, schemas, and sounds), splits, privileged auxiliary concept information, and choices of negatives reveal substantial scope for modeling advances on the proposed task. All code and datasets will be available online.","url_abs":"https://arxiv.org/abs/2010.02855v1","url_pdf":"https://arxiv.org/pdf/2010.02855v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"curi-a-benchmark-for-productive-concept-1","repo_url":"https://github.com/facebookresearch/productive_concept_learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"meta-learning","task_name":"Meta-Learning"},{"task_slug":"systematic-generalization","task_name":"Systematic Generalization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2010.02855","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.02855"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/productive_concept_learning","reach":null}],"summary":{"unverified":1},"by_repo_kind":{"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"3d4bf01aa4c60523","entry":"load_model","repo":"facebookresearch/productive_concept_learning","repo_kind":"listed","path":"hydra_train.py","file_url":"https://github.com/facebookresearch/productive_concept_learning/blob/HEAD/hydra_train.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3d4bf01aa4c60523"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}