{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pug-photorealistic-and-semantically-1","title":"PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning","arxiv_id":"2308.03977","date":"2023-08-08","proceeding":"NeurIPS 2023 11","authors":["Florian Bordes","Shashank Shekhar","Mark Ibrahim","Diane Bouchacourt","Pascal Vincent","Ari S. Morcos"],"abstract":"Synthetic image datasets offer unmatched advantages for designing and evaluating deep neural networks: they make it possible to (i) render as many data samples as needed, (ii) precisely control each scene and yield granular ground truth labels (and captions), (iii) precisely control distribution shifts between training and testing to isolate variables of interest for sound experimentation. Despite such promise, the use of synthetic image data is still limited -- and often played down -- mainly due to their lack of realism. Most works therefore rely on datasets of real images, which have often been scraped from public images on the internet, and may have issues with regards to privacy, bias, and copyright, while offering little control over how objects precisely appear. In this work, we present a path to democratize the use of photorealistic synthetic data: we develop a new generation of interactive environments for representation learning research, that offer both controllability and realism. We use the Unreal Engine, a powerful game engine well known in the entertainment industry, to produce PUG (Photorealistic Unreal Graphics) environments and datasets for representation learning. In this paper, we demonstrate the potential of PUG to enable more rigorous evaluations of vision models.","url_abs":"https://arxiv.org/abs/2308.03977v2","url_pdf":"https://arxiv.org/pdf/2308.03977v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pug-photorealistic-and-semantically-1","repo_url":"https://github.com/facebookresearch/pug","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2308.03977","atlas_url":"https://app.syntology.ai/?focus=2308.03977","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.03977"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/pug","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"79300bdd65defd71","entry":"convertByteMessageToInt","repo":"facebookresearch/pug","repo_kind":"official","path":"torchmultiverse/client.py","file_url":"https://github.com/facebookresearch/pug/blob/HEAD/torchmultiverse/client.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"79300bdd65defd71"}},{"code_sha256_prefix":"d5f6ee7052bf3d48","entry":"convertIntMessageToByte","repo":"facebookresearch/pug","repo_kind":"official","path":"torchmultiverse/client.py","file_url":"https://github.com/facebookresearch/pug/blob/HEAD/torchmultiverse/client.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d5f6ee7052bf3d48"}},{"code_sha256_prefix":"5a6e55178416f43d","entry":"emitDescriptor","repo":"facebookresearch/pug","repo_kind":"official","path":"torchmultiverse/client.py","file_url":"https://github.com/facebookresearch/pug/blob/HEAD/torchmultiverse/client.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5a6e55178416f43d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}