{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/image-content-generation-with-causal","title":"Image Content Generation with Causal Reasoning","arxiv_id":"2312.07132","date":"2023-12-12","proceeding":null,"authors":["Xiaochuan Li","Baoyu Fan","Runze Zhang","Liang Jin","Di Wang","Zhenhua Guo","YaQian Zhao","RenGang Li"],"abstract":"The emergence of ChatGPT has once again sparked research in generative artificial intelligence (GAI). While people have been amazed by the generated results, they have also noticed the reasoning potential reflected in the generated textual content. However, this current ability for causal reasoning is primarily limited to the domain of language generation, such as in models like GPT-3. In visual modality, there is currently no equivalent research. Considering causal reasoning in visual content generation is significant. This is because visual information contains infinite granularity. Particularly, images can provide more intuitive and specific demonstrations for certain reasoning tasks, especially when compared to coarse-grained text. Hence, we propose a new image generation task called visual question answering with image (VQAI) and establish a dataset of the same name based on the classic \\textit{Tom and Jerry} animated series. Additionally, we develop a new paradigm for image generation to tackle the challenges of this task. Finally, we perform extensive experiments and analyses, including visualizations of the generated content and discussions on the potentials and limitations. The code and data are publicly available under the license of CC BY-NC-SA 4.0 for academic and non-commercial usage. The code and dataset are publicly available at: https://github.com/IEIT-AGI/MIX-Shannon/blob/main/projects/VQAI/lgd_vqai.md.","url_abs":"https://arxiv.org/abs/2312.07132v1","url_pdf":"https://arxiv.org/pdf/2312.07132v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"image-content-generation-with-causal","repo_url":"https://github.com/ieit-agi/mix-shannon","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.07132","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.07132"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ieit-agi/mix-shannon","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"65b76f6fcd86e5be","entry":"create_positive_map","repo":"ieit-agi/mix-shannon","repo_kind":"official","path":"src/datamodules/data_bases.py","file_url":"https://github.com/ieit-agi/mix-shannon/blob/HEAD/src/datamodules/data_bases.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"65b76f6fcd86e5be"}},{"code_sha256_prefix":"a1af7feaf9cc90fb","entry":"field_align_in_FREC","repo":"ieit-agi/mix-shannon","repo_kind":"official","path":"src/datamodules/frec_datamodule.py","file_url":"https://github.com/ieit-agi/mix-shannon/blob/HEAD/src/datamodules/frec_datamodule.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1af7feaf9cc90fb"}},{"code_sha256_prefix":"c6ec173f19f5c34d","entry":"interpolate_pos_embed","repo":"ieit-agi/mix-shannon","repo_kind":"official","path":"lavis/models/vit.py","file_url":"https://github.com/ieit-agi/mix-shannon/blob/HEAD/lavis/models/vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c6ec173f19f5c34d"}},{"code_sha256_prefix":"cb33571427334815","entry":"tile","repo":"ieit-agi/mix-shannon","repo_kind":"official","path":"lavis/models/base_model.py","file_url":"https://github.com/ieit-agi/mix-shannon/blob/HEAD/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb33571427334815"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"ieit-agi/mix-shannon","repo_kind":"official","path":"lavis/models/base_model.py","file_url":"https://github.com/ieit-agi/mix-shannon/blob/HEAD/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"5435e3d77040cfbb","entry":"create_eva_vit_g","repo":"ieit-agi/mix-shannon","repo_kind":"official","path":"lavis/models/eva_vit.py","file_url":"https://github.com/ieit-agi/mix-shannon/blob/HEAD/lavis/models/eva_vit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5435e3d77040cfbb"}},{"code_sha256_prefix":"525aaa24d3342b71","entry":"main","repo":"ieit-agi/mix-shannon","repo_kind":"official","path":"validate.py","file_url":"https://github.com/ieit-agi/mix-shannon/blob/HEAD/validate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"525aaa24d3342b71"}},{"code_sha256_prefix":"481508014360cd2a","entry":"make_coco_transforms","repo":"ieit-agi/mix-shannon","repo_kind":"official","path":"src/datamodules/data_bases.py","file_url":"https://github.com/ieit-agi/mix-shannon/blob/HEAD/src/datamodules/data_bases.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"481508014360cd2a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}