{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/laion-sg-an-enhanced-large-scale-dataset-for","title":"LAION-SG: An Enhanced Large-Scale Dataset for Training Complex Image-Text Models with Structural Annotations","arxiv_id":"2412.08580","date":"2024-12-11","proceeding":null,"authors":["Zejian Li","Chenye Meng","Yize Li","Ling Yang","Shengyuan Zhang","Jiarui Ma","Jiayi Li","Guang Yang","Changyuan Yang","Zhiyuan Yang","Jinxiong Chang","Lingyun Sun"],"abstract":"Recent advances in text-to-image (T2I) generation have shown remarkable success in producing high-quality images from text. However, existing T2I models show decayed performance in compositional image generation involving multiple objects and intricate relationships. We attribute this problem to limitations in existing datasets of image-text pairs, which lack precise inter-object relationship annotations with prompts only. To address this problem, we construct LAION-SG, a large-scale dataset with high-quality structural annotations of scene graphs (SG), which precisely describe attributes and relationships of multiple objects, effectively representing the semantic structure in complex scenes. Based on LAION-SG, we train a new foundation model SDXL-SG to incorporate structural annotation information into the generation process. Extensive experiments show advanced models trained on our LAION-SG boast significant performance improvements in complex scene generation over models on existing datasets. We also introduce CompSG-Bench, a benchmark that evaluates models on compositional image generation, establishing a new standard for this domain. Our annotations with the associated processing code, the foundation model and the benchmark protocol are publicly available at https://github.com/mengcye/LAION-SG.","url_abs":"https://arxiv.org/abs/2412.08580v2","url_pdf":"https://arxiv.org/pdf/2412.08580v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"laion-sg-an-enhanced-large-scale-dataset-for","repo_url":"https://github.com/mengcye/LAION-SG","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"laion-sg-an-enhanced-large-scale-dataset-for","repo_url":"https://github.com/yangling0818/sgdiff","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"scene-generation","task_name":"Scene Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.08580","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.08580"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mengcye/LAION-SG","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangling0818/sgdiff","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"ran_violates":2,"ran":2,"unverified":17},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1},"listed":{"samples":12,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"fe5dd5258046898c","entry":"always","repo":"yangling0818/sgdiff","repo_kind":"listed","path":"ldm/modules/x_transformer.py","file_url":"https://github.com/yangling0818/sgdiff/blob/HEAD/ldm/modules/x_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe5dd5258046898c"}},{"code_sha256_prefix":"424012cb37b31172","entry":"default","repo":"yangling0818/sgdiff","repo_kind":"listed","path":"ldm/modules/attention.py","file_url":"https://github.com/yangling0818/sgdiff/blob/HEAD/ldm/modules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"424012cb37b31172"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"yangling0818/sgdiff","repo_kind":"listed","path":"ldm/modules/attention.py","file_url":"https://github.com/yangling0818/sgdiff/blob/HEAD/ldm/modules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"b1368330cf0f5642","entry":"isimage","repo":"yangling0818/sgdiff","repo_kind":"listed","path":"ldm/util.py","file_url":"https://github.com/yangling0818/sgdiff/blob/HEAD/ldm/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1368330cf0f5642"}},{"code_sha256_prefix":"d72762b700feee6f","entry":"ismap","repo":"yangling0818/sgdiff","repo_kind":"listed","path":"ldm/util.py","file_url":"https://github.com/yangling0818/sgdiff/blob/HEAD/ldm/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d72762b700feee6f"}},{"code_sha256_prefix":"9a299fe5ae09e407","entry":"uniq","repo":"yangling0818/sgdiff","repo_kind":"listed","path":"ldm/modules/attention.py","file_url":"https://github.com/yangling0818/sgdiff/blob/HEAD/ldm/modules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a299fe5ae09e407"}},{"code_sha256_prefix":"13f977de43dee032","entry":"boxes_to_mask","repo":"mengcye/LAION-SG","repo_kind":"official","path":"sgEncoderTraining/utils.py","file_url":"https://github.com/mengcye/LAION-SG/blob/HEAD/sgEncoderTraining/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"13f977de43dee032"}},{"code_sha256_prefix":"1117cb855a81abc3","entry":"build_laion_dsets","repo":"mengcye/LAION-SG","repo_kind":"official","path":"sgEncoderTraining/datasets/laion_dataset.py","file_url":"https://github.com/mengcye/LAION-SG/blob/HEAD/sgEncoderTraining/datasets/laion_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1117cb855a81abc3"}},{"code_sha256_prefix":"def453b3dce37100","entry":"build_laion_loaders","repo":"mengcye/LAION-SG","repo_kind":"official","path":"sgEncoderTraining/datasets/laion_dataset.py","file_url":"https://github.com/mengcye/LAION-SG/blob/HEAD/sgEncoderTraining/datasets/laion_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"def453b3dce37100"}},{"code_sha256_prefix":"8692859d555c8768","entry":"build_mlp","repo":"mengcye/LAION-SG","repo_kind":"official","path":"sgEncoderTraining/sgEncoder/module.py","file_url":"https://github.com/mengcye/LAION-SG/blob/HEAD/sgEncoderTraining/sgEncoder/module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8692859d555c8768"}},{"code_sha256_prefix":"39e6b23b55f376ea","entry":"build_model","repo":"yangling0818/sgdiff","repo_kind":"listed","path":"sg_image_pretraining/clip/model.py","file_url":"https://github.com/yangling0818/sgdiff/blob/HEAD/sg_image_pretraining/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39e6b23b55f376ea"}},{"code_sha256_prefix":"932d44bbf8bd0e3f","entry":"cosine_lr","repo":"mengcye/LAION-SG","repo_kind":"official","path":"sgEncoderTraining/training/scheduler.py","file_url":"https://github.com/mengcye/LAION-SG/blob/HEAD/sgEncoderTraining/training/scheduler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"932d44bbf8bd0e3f"}},{"code_sha256_prefix":"245ff5228caf9482","entry":"draw_scene_graph","repo":"yangling0818/sgdiff","repo_kind":"listed","path":"testset_ddim_sampler.py","file_url":"https://github.com/yangling0818/sgdiff/blob/HEAD/testset_ddim_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"245ff5228caf9482"}},{"code_sha256_prefix":"08c9ee87b8be3253","entry":"draw_scene_graph","repo":"yangling0818/sgdiff","repo_kind":"listed","path":"sg2im/vis.py","file_url":"https://github.com/yangling0818/sgdiff/blob/HEAD/sg2im/vis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"08c9ee87b8be3253"}},{"code_sha256_prefix":"15167a5cc461251c","entry":"generate_box_mask","repo":"mengcye/LAION-SG","repo_kind":"official","path":"sgEncoderTraining/utils.py","file_url":"https://github.com/mengcye/LAION-SG/blob/HEAD/sgEncoderTraining/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15167a5cc461251c"}},{"code_sha256_prefix":"f988e73e85f4ea4a","entry":"get_autocast","repo":"mengcye/LAION-SG","repo_kind":"official","path":"sgEncoderTraining/training/precision.py","file_url":"https://github.com/mengcye/LAION-SG/blob/HEAD/sgEncoderTraining/training/precision.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f988e73e85f4ea4a"}},{"code_sha256_prefix":"df129d4646ebbcc0","entry":"get_linear_feas_by_hook","repo":"mengcye/LAION-SG","repo_kind":"official","path":"sgEncoderTraining/utils.py","file_url":"https://github.com/mengcye/LAION-SG/blob/HEAD/sgEncoderTraining/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df129d4646ebbcc0"}},{"code_sha256_prefix":"5fc36bbf7ad81a90","entry":"import_model_class_from_model_name_or_path","repo":"mengcye/LAION-SG","repo_kind":"official","path":"trainer_laion.py","file_url":"https://github.com/mengcye/LAION-SG/blob/HEAD/trainer_laion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5fc36bbf7ad81a90"}},{"code_sha256_prefix":"f9bd2e83191afad1","entry":"log_txt_as_img","repo":"yangling0818/sgdiff","repo_kind":"listed","path":"ldm/util.py","file_url":"https://github.com/yangling0818/sgdiff/blob/HEAD/ldm/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9bd2e83191afad1"}},{"code_sha256_prefix":"6124caed65f1c444","entry":"my_laion_collate_fn","repo":"mengcye/LAION-SG","repo_kind":"official","path":"sgEncoderTraining/datasets/laion_dataset.py","file_url":"https://github.com/mengcye/LAION-SG/blob/HEAD/sgEncoderTraining/datasets/laion_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6124caed65f1c444"}},{"code_sha256_prefix":"87fbedd0c27cf378","entry":"nondefault_trainer_args","repo":"yangling0818/sgdiff","repo_kind":"listed","path":"trainer.py","file_url":"https://github.com/yangling0818/sgdiff/blob/HEAD/trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87fbedd0c27cf378"}},{"code_sha256_prefix":"2bf62c0287050bdd","entry":"unwrap_model","repo":"mengcye/LAION-SG","repo_kind":"official","path":"sgEncoderTraining/training/train_and_val_one_iter.py","file_url":"https://github.com/mengcye/LAION-SG/blob/HEAD/sgEncoderTraining/training/train_and_val_one_iter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2bf62c0287050bdd"}},{"code_sha256_prefix":"c31f6b9b1b521862","entry":"vg_collate_fn","repo":"yangling0818/sgdiff","repo_kind":"listed","path":"testset_ddim_sampler.py","file_url":"https://github.com/yangling0818/sgdiff/blob/HEAD/testset_ddim_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c31f6b9b1b521862"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}