{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/question-answer-cross-language-image-matching","title":"Question-Answer Cross Language Image Matching for Weakly Supervised Semantic Segmentation","arxiv_id":"2401.09883","date":"2024-01-18","proceeding":null,"authors":["Songhe Deng","Wei Zhuo","Jinheng Xie","Linlin Shen"],"abstract":"Class Activation Map (CAM) has emerged as a popular tool for weakly supervised semantic segmentation (WSSS), allowing the localization of object regions in an image using only image-level labels. However, existing CAM methods suffer from under-activation of target object regions and false-activation of background regions due to the fact that a lack of detailed supervision can hinder the model's ability to understand the image as a whole. In this paper, we propose a novel Question-Answer Cross-Language-Image Matching framework for WSSS (QA-CLIMS), leveraging the vision-language foundation model to maximize the text-based understanding of images and guide the generation of activation maps. First, a series of carefully designed questions are posed to the VQA (Visual Question Answering) model with Question-Answer Prompt Engineering (QAPE) to generate a corpus of both foreground target objects and backgrounds that are adaptive to query images. We then employ contrastive learning in a Region Image Text Contrastive (RITC) network to compare the obtained foreground and background regions with the generated corpus. Our approach exploits the rich textual information from the open vocabulary as additional supervision, enabling the model to generate high-quality CAMs with a more complete object region and reduce false-activation of background regions. We conduct extensive analysis to validate the proposed method and show that our approach performs state-of-the-art on both PASCAL VOC 2012 and MS COCO datasets. Code is available at: https://github.com/CVI-SZU/QA-CLIMS","url_abs":"https://arxiv.org/abs/2401.09883v1","url_pdf":"https://arxiv.org/pdf/2401.09883v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"question-answer-cross-language-image-matching","repo_url":"https://github.com/cvi-szu/qa-clims","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"weakly-supervised-semantic-segmentation-1","task_name":"Weakly supervised Semantic Segmentation"},{"task_slug":"weakly-supervised-semantic-segmentation","task_name":"Weakly-Supervised Semantic Segmentation"}],"methods":[{"method_slug":"cam","method_name":"CAM"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/weakly-supervised-semantic-segmentation-on-1","task":"Weakly-Supervised Semantic Segmentation","dataset":"PASCAL VOC 2012 test","model":"QA-CLIMS","rank_in_archive_order":7,"of":60,"metrics":{"Mean IoU":"75.5"},"uses_additional_data":true},{"leaderboard":"/sota/weakly-supervised-semantic-segmentation-on","task":"Weakly-Supervised Semantic Segmentation","dataset":"PASCAL VOC 2012 val","model":"QA-CLIMS","rank_in_archive_order":9,"of":73,"metrics":{"Mean IoU":"75.6"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.09883","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.09883"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cvi-szu/qa-clims","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"24f97933bc49cab7","entry":"make_cam","repo":"cvi-szu/qa-clims","repo_kind":"official","path":"step/make_clims.py","file_url":"https://github.com/cvi-szu/qa-clims/blob/HEAD/step/make_clims.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24f97933bc49cab7"}},{"code_sha256_prefix":"e1c280d662e89083","entry":"preprocess","repo":"cvi-szu/qa-clims","repo_kind":"official","path":"step/train_qa_clims.py","file_url":"https://github.com/cvi-szu/qa-clims/blob/HEAD/step/train_qa_clims.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1c280d662e89083"}},{"code_sha256_prefix":"39c2c070ce06a6f8","entry":"resnet50","repo":"cvi-szu/qa-clims","repo_kind":"official","path":"net/resnet50.py","file_url":"https://github.com/cvi-szu/qa-clims/blob/HEAD/net/resnet50.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39c2c070ce06a6f8"}},{"code_sha256_prefix":"c4a9cc2719c2ba98","entry":"resnet50","repo":"cvi-szu/qa-clims","repo_kind":"official","path":"net/resnet50_v2.py","file_url":"https://github.com/cvi-szu/qa-clims/blob/HEAD/net/resnet50_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4a9cc2719c2ba98"}},{"code_sha256_prefix":"85a712d3b5d2e0b7","entry":"transpose","repo":"cvi-szu/qa-clims","repo_kind":"official","path":"step/make_clims.py","file_url":"https://github.com/cvi-szu/qa-clims/blob/HEAD/step/make_clims.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85a712d3b5d2e0b7"}},{"code_sha256_prefix":"d688d50b535e08bc","entry":"reduce_mean","repo":"cvi-szu/qa-clims","repo_kind":"official","path":"step/train_qa_clims.py","file_url":"https://github.com/cvi-szu/qa-clims/blob/HEAD/step/train_qa_clims.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d688d50b535e08bc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}