{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/harmonizing-visual-text-comprehension-and","title":"Harmonizing Visual Text Comprehension and Generation","arxiv_id":"2407.16364","date":"2024-07-23","proceeding":null,"authors":["Zhen Zhao","Jingqun Tang","Binghong Wu","Chunhui Lin","Shu Wei","Hao liu","Xin Tan","Zhizhong Zhang","Can Huang","Yuan Xie"],"abstract":"In this work, we present TextHarmony, a unified and versatile multimodal generative model proficient in comprehending and generating visual text. Simultaneously generating images and texts typically results in performance degradation due to the inherent inconsistency between vision and language modalities. To overcome this challenge, existing approaches resort to modality-specific data for supervised fine-tuning, necessitating distinct model instances. We propose Slide-LoRA, which dynamically aggregates modality-specific and modality-agnostic LoRA experts, partially decoupling the multimodal generation space. Slide-LoRA harmonizes the generation of vision and language within a singular model instance, thereby facilitating a more unified generative process. Additionally, we develop a high-quality image caption dataset, DetailedTextCaps-100K, synthesized with a sophisticated closed-source MLLM to enhance visual text generation capabilities further. Comprehensive experiments across various benchmarks demonstrate the effectiveness of the proposed approach. Empowered by Slide-LoRA, TextHarmony achieves comparable performance to modality-specific fine-tuning results with only a 2% increase in parameters and shows an average improvement of 2.5% in visual text comprehension tasks and 4.0% in visual text generation tasks. Our work delineates the viability of an integrated approach to multimodal generation within the visual text domain, setting a foundation for subsequent inquiries. Code is available at https://github.com/bytedance/TextHarmony.","url_abs":"https://arxiv.org/abs/2407.16364v2","url_pdf":"https://arxiv.org/pdf/2407.16364v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"harmonizing-visual-text-comprehension-and","repo_url":"https://github.com/bytedance/textharmony","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"multimodal-generation","task_name":"multimodal generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.16364","atlas_url":"https://app.syntology.ai/?focus=2407.16364","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.16364"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/bytedance/TextHarmony","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bytedance/textharmony","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":2,"ran_honours":1,"ran":6,"ran_fixture":2,"unverified":7},"by_repo_kind":{"official":{"samples":18,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"94e70fb96cf2444b","entry":"load_jsonl","repo":"bytedance/TextHarmony","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/bytedance/TextHarmony/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"94e70fb96cf2444b"}},{"code_sha256_prefix":"ce749f837424dd8c","entry":"box_iou","repo":"bytedance/textharmony","repo_kind":"official","path":"TextHarmony/utils/grounding_score.py","file_url":"https://github.com/bytedance/textharmony/blob/HEAD/TextHarmony/utils/grounding_score.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce749f837424dd8c"}},{"code_sha256_prefix":"f3030aefe59749fb","entry":"colorEncode","repo":"bytedance/textharmony","repo_kind":"official","path":"TextHarmony/custom_datasets/ade20k_preparation.py","file_url":"https://github.com/bytedance/textharmony/blob/HEAD/TextHarmony/custom_datasets/ade20k_preparation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f3030aefe59749fb"}},{"code_sha256_prefix":"c1b06375bdb265f4","entry":"extract_answer","repo":"bytedance/textharmony","repo_kind":"official","path":"TextHarmony/utils/vqa_score.py","file_url":"https://github.com/bytedance/textharmony/blob/HEAD/TextHarmony/utils/vqa_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c1b06375bdb265f4"}},{"code_sha256_prefix":"218b45d457ba5f5b","entry":"grounding_eval","repo":"bytedance/textharmony","repo_kind":"official","path":"TextHarmony/utils/grounding_score.py","file_url":"https://github.com/bytedance/textharmony/blob/HEAD/TextHarmony/utils/grounding_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"218b45d457ba5f5b"}},{"code_sha256_prefix":"6ae320f7c8e2bf52","entry":"load_annt_data","repo":"bytedance/TextHarmony","repo_kind":"official","path":"inference.py","file_url":"https://github.com/bytedance/TextHarmony/blob/HEAD/inference.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6ae320f7c8e2bf52"}},{"code_sha256_prefix":"e6cb89e8ea67aeef","entry":"model_gen","repo":"bytedance/TextHarmony","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/bytedance/TextHarmony/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e6cb89e8ea67aeef"}},{"code_sha256_prefix":"13e859c753057e62","entry":"parse_box","repo":"bytedance/textharmony","repo_kind":"official","path":"TextHarmony/utils/grounding_score.py","file_url":"https://github.com/bytedance/textharmony/blob/HEAD/TextHarmony/utils/grounding_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"13e859c753057e62"}},{"code_sha256_prefix":"1af97eee0d5e9a78","entry":"tensor_to_pil","repo":"bytedance/textharmony","repo_kind":"official","path":"TextHarmony/utils/clip_sim_score.py","file_url":"https://github.com/bytedance/textharmony/blob/HEAD/TextHarmony/utils/clip_sim_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1af97eee0d5e9a78"}},{"code_sha256_prefix":"e813a6aeca895c35","entry":"unique","repo":"bytedance/textharmony","repo_kind":"official","path":"TextHarmony/custom_datasets/ade20k_preparation.py","file_url":"https://github.com/bytedance/textharmony/blob/HEAD/TextHarmony/custom_datasets/ade20k_preparation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e813a6aeca895c35"}},{"code_sha256_prefix":"6aae1c40e557fe64","entry":"update_texts","repo":"bytedance/TextHarmony","repo_kind":"official","path":"inference.py","file_url":"https://github.com/bytedance/TextHarmony/blob/HEAD/inference.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6aae1c40e557fe64"}},{"code_sha256_prefix":"58f3cb144de3c6f9","entry":"calculate_activation_statistics","repo":"bytedance/textharmony","repo_kind":"official","path":"TextHarmony/utils/fid_score.py","file_url":"https://github.com/bytedance/textharmony/blob/HEAD/TextHarmony/utils/fid_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"58f3cb144de3c6f9"}},{"code_sha256_prefix":"a2cc9c29fefca146","entry":"calculate_clip_sim_i2i","repo":"bytedance/textharmony","repo_kind":"official","path":"TextHarmony/utils/clip_sim_score.py","file_url":"https://github.com/bytedance/textharmony/blob/HEAD/TextHarmony/utils/clip_sim_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a2cc9c29fefca146"}},{"code_sha256_prefix":"0def50a351111624","entry":"calculate_frechet_distance","repo":"bytedance/textharmony","repo_kind":"official","path":"TextHarmony/utils/fid_score.py","file_url":"https://github.com/bytedance/textharmony/blob/HEAD/TextHarmony/utils/fid_score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0def50a351111624"}},{"code_sha256_prefix":"b6d159b5ffc47aa4","entry":"clip_rerank_generated_images","repo":"bytedance/textharmony","repo_kind":"official","path":"TextHarmony/utils/clip_sim_score.py","file_url":"https://github.com/bytedance/textharmony/blob/HEAD/TextHarmony/utils/clip_sim_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b6d159b5ffc47aa4"}},{"code_sha256_prefix":"696501a9b5a48e4b","entry":"get_activations","repo":"bytedance/textharmony","repo_kind":"official","path":"TextHarmony/utils/fid_score.py","file_url":"https://github.com/bytedance/textharmony/blob/HEAD/TextHarmony/utils/fid_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"696501a9b5a48e4b"}},{"code_sha256_prefix":"521448a8a9392444","entry":"vizwiz_vqa_eval","repo":"bytedance/textharmony","repo_kind":"official","path":"TextHarmony/utils/vqa_score.py","file_url":"https://github.com/bytedance/textharmony/blob/HEAD/TextHarmony/utils/vqa_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"521448a8a9392444"}},{"code_sha256_prefix":"a80245c5d6d89489","entry":"vqa_eval","repo":"bytedance/textharmony","repo_kind":"official","path":"TextHarmony/utils/vqa_score.py","file_url":"https://github.com/bytedance/textharmony/blob/HEAD/TextHarmony/utils/vqa_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a80245c5d6d89489"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}