{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/leveraging-hallucinations-to-reduce-manual","title":"Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation","arxiv_id":"2408.15205","date":"2024-08-27","proceeding":null,"authors":["Jian Hu","Jiayi Lin","Junchi Yan","Shaogang Gong"],"abstract":"Promptable segmentation typically requires instance-specific manual prompts to guide the segmentation of each desired object. To minimize such a need, task-generic promptable segmentation has been introduced, which employs a single task-generic prompt to segment various images of different objects in the same task. Current methods use Multimodal Large Language Models (MLLMs) to reason detailed instance-specific prompts from a task-generic prompt for improving segmentation accuracy. The effectiveness of this segmentation heavily depends on the precision of these derived prompts. However, MLLMs often suffer hallucinations during reasoning, resulting in inaccurate prompting. While existing methods focus on eliminating hallucinations to improve a model, we argue that MLLM hallucinations can reveal valuable contextual insights when leveraged correctly, as they represent pre-trained large-scale knowledge beyond individual images. In this paper, we utilize hallucinations to mine task-related information from images and verify its accuracy for enhancing precision of the generated prompts. Specifically, we introduce an iterative Prompt-Mask Cycle generation framework (ProMaC) with a prompt generator and a mask generator.The prompt generator uses a multi-scale chain of thought prompting, initially exploring hallucinations for extracting extended contextual knowledge on a test image.These hallucinations are then reduced to formulate precise instance-specific prompts, directing the mask generator to produce masks that are consistent with task semantics by mask semantic alignment. The generated masks iteratively induce the prompt generator to focus more on task-relevant image areas and reduce irrelevant hallucinations, resulting jointly in better prompts and masks. Experiments on 5 benchmarks demonstrate the effectiveness of ProMaC. Code given in https://lwpyh.github.io/ProMaC/.","url_abs":"https://arxiv.org/abs/2408.15205v3","url_pdf":"https://arxiv.org/pdf/2408.15205v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"leveraging-hallucinations-to-reduce-manual","repo_url":"https://github.com/lwpyh/ProMaC_code","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"camouflaged-object-segmentation","task_name":"Camouflaged Object Segmentation"},{"task_slug":"camouflaged-object-segmentation-with-a-single","task_name":"Camouflaged Object Segmentation with a Single Task-generic Prompt"},{"task_slug":"medical-image-segmentation","task_name":"Medical Image Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"}],"methods":[{"method_slug":"gensam","method_name":"GenSAM"},{"method_slug":"sam","method_name":"SAM"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/camouflaged-object-segmentation-with-a-single-1","task":"Camouflaged Object Segmentation with a Single Task-generic Prompt","dataset":"CAMO","model":"ProMaC","rank_in_archive_order":2,"of":3,"metrics":{"E_{\\phi}":"0.846","F_{\\beta}":"0.725","MAE":"0.09","S_{\\alpha}":"0.767"},"uses_additional_data":false},{"leaderboard":"/sota/camouflaged-object-segmentation-with-a-single-2","task":"Camouflaged Object Segmentation with a Single Task-generic Prompt","dataset":"COD10K","model":"ProMaC","rank_in_archive_order":2,"of":3,"metrics":{"E_{\\phi}":"0.876","F_{\\beta}":"0.716","MAE":"0.042","S_{\\alpha}":"0.805"},"uses_additional_data":false},{"leaderboard":"/sota/camouflaged-object-segmentation-with-a-single","task":"Camouflaged Object Segmentation with a Single Task-generic Prompt","dataset":"Chameleon","model":"ProMaC","rank_in_archive_order":1,"of":2,"metrics":{"E_{\\phi}":"0.899","F_{\\beta}":"0.79","MAE":"0.044","S_{\\alpha}":"0.833"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2408.15205","atlas_url":"https://app.syntology.ai/?focus=2408.15205","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.15205"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lwpyh/ProMaC_code","reach":null}],"summary":{"ran_violates":1,"ran_fixture":1,"unverified":7},"by_repo_kind":{"official":{"samples":9,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"923b02f12c4e8943","entry":"convert_patch_bbox_to_original","repo":"lwpyh/ProMaC_code","repo_kind":"official","path":"utils_mllm.py","file_url":"https://github.com/lwpyh/ProMaC_code/blob/HEAD/utils_mllm.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"923b02f12c4e8943"}},{"code_sha256_prefix":"93aa0fead442ddef","entry":"expand_bbox","repo":"lwpyh/ProMaC_code","repo_kind":"official","path":"utils_mllm.py","file_url":"https://github.com/lwpyh/ProMaC_code/blob/HEAD/utils_mllm.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93aa0fead442ddef"}},{"code_sha256_prefix":"4379261942974adb","entry":"generate_image","repo":"lwpyh/ProMaC_code","repo_kind":"official","path":"utils_mllm.py","file_url":"https://github.com/lwpyh/ProMaC_code/blob/HEAD/utils_mllm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4379261942974adb"}},{"code_sha256_prefix":"fc5b045f2c663b01","entry":"get_reflected_text_from_full_img_llava","repo":"lwpyh/ProMaC_code","repo_kind":"official","path":"utils_mllm.py","file_url":"https://github.com/lwpyh/ProMaC_code/blob/HEAD/utils_mllm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fc5b045f2c663b01"}},{"code_sha256_prefix":"47abbc2ac21defe8","entry":"get_reflected_text_from_img","repo":"lwpyh/ProMaC_code","repo_kind":"official","path":"utils_mllm.py","file_url":"https://github.com/lwpyh/ProMaC_code/blob/HEAD/utils_mllm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47abbc2ac21defe8"}},{"code_sha256_prefix":"4f885a4ecc54c188","entry":"get_reflected_text_from_img_llava_collected","repo":"lwpyh/ProMaC_code","repo_kind":"official","path":"utils_mllm.py","file_url":"https://github.com/lwpyh/ProMaC_code/blob/HEAD/utils_mllm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f885a4ecc54c188"}},{"code_sha256_prefix":"141d2009ede3317e","entry":"get_text_from_img_llava_with_bbox_iter0","repo":"lwpyh/ProMaC_code","repo_kind":"official","path":"utils_mllm.py","file_url":"https://github.com/lwpyh/ProMaC_code/blob/HEAD/utils_mllm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"141d2009ede3317e"}},{"code_sha256_prefix":"b470f2ff84a1db05","entry":"get_text_from_img_llava_with_bbox_patch","repo":"lwpyh/ProMaC_code","repo_kind":"official","path":"utils_mllm.py","file_url":"https://github.com/lwpyh/ProMaC_code/blob/HEAD/utils_mllm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b470f2ff84a1db05"}},{"code_sha256_prefix":"03c001cd5ce7f974","entry":"show","repo":"lwpyh/ProMaC_code","repo_kind":"official","path":"utils_mllm.py","file_url":"https://github.com/lwpyh/ProMaC_code/blob/HEAD/utils_mllm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03c001cd5ce7f974"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}