{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llm-wrapper-black-box-semantic-aware","title":"LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension","arxiv_id":"2409.11919","date":"2024-09-18","proceeding":null,"authors":["Amaia Cardiel","Eloi Zablocki","Elias Ramzi","Oriane Siméoni","Matthieu Cord"],"abstract":"Vision Language Models (VLMs) have demonstrated remarkable capabilities in various open-vocabulary tasks, yet their zero-shot performance lags behind task-specific finetuned models, particularly in complex tasks like Referring Expression Comprehension (REC). Fine-tuning usually requires 'white-box' access to the model's architecture and weights, which is not always feasible due to proprietary or privacy concerns. In this work, we propose LLM-wrapper, a method for 'black-box' adaptation of VLMs for the REC task using Large Language Models (LLMs). LLM-wrapper capitalizes on the reasoning abilities of LLMs, improved with a light fine-tuning, to select the most relevant bounding box matching the referring expression, from candidates generated by a zero-shot black-box VLM. Our approach offers several advantages: it enables the adaptation of closed-source models without needing access to their internal workings, it is versatile as it works with any VLM, it transfers to new VLMs, and it allows for the adaptation of an ensemble of VLMs. We evaluate LLM-wrapper on multiple datasets using different VLMs and LLMs, demonstrating significant performance improvements and highlighting the versatility of our method. While LLM-wrapper is not meant to directly compete with standard white-box fine-tuning, it offers a practical and effective alternative for black-box VLM adaptation. The code will be open-sourced.","url_abs":"https://arxiv.org/abs/2409.11919v2","url_pdf":"https://arxiv.org/pdf/2409.11919v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llm-wrapper-black-box-semantic-aware","repo_url":"https://github.com/valeoai/LLM_wrapper","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"referring-expression","task_name":"Referring Expression"},{"task_slug":"referring-expression-comprehension","task_name":"Referring Expression Comprehension"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.11919","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.11919"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/valeoai/LLM_wrapper","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":4,"ran":3,"ran_honours":1,"unverified":9},"by_repo_kind":{"official":{"samples":18,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"496f36883982907e","entry":"convert_box_coord_list","repo":"valeoai/LLM_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/LLM_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"496f36883982907e"}},{"code_sha256_prefix":"b62369eaf6443bf1","entry":"create_my_prompt","repo":"valeoai/llm_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_finetuning_script.py","file_url":"https://github.com/valeoai/llm_wrapper/blob/HEAD/llm_wrapper/LLM_finetuning_script.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b62369eaf6443bf1"}},{"code_sha256_prefix":"f96ceeee851477a5","entry":"define_modules_to_FT","repo":"valeoai/llm_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_finetuning_script.py","file_url":"https://github.com/valeoai/llm_wrapper/blob/HEAD/llm_wrapper/LLM_finetuning_script.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f96ceeee851477a5"}},{"code_sha256_prefix":"d1d34bdea6597b80","entry":"define_short_LLM_folder_name","repo":"valeoai/LLM_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/LLM_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1d34bdea6597b80"}},{"code_sha256_prefix":"6921e7c0cacc7dc8","entry":"eval_post_step_3","repo":"valeoai/LLM_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/LLM_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6921e7c0cacc7dc8"}},{"code_sha256_prefix":"ff499b564cc9cdab","entry":"extract_int_result_from_LLM_gen","repo":"valeoai/LLM_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/LLM_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff499b564cc9cdab"}},{"code_sha256_prefix":"0262028d2a06ec2b","entry":"get_HF_model_id","repo":"valeoai/LLM_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/LLM_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0262028d2a06ec2b"}},{"code_sha256_prefix":"8e0fab2c4512a48c","entry":"get_all_rec_and_thres_boxes_concat","repo":"valeoai/llm_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/llm_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e0fab2c4512a48c"}},{"code_sha256_prefix":"d4ea0c52a00729fd","entry":"get_box_key_list","repo":"valeoai/llm_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/llm_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d4ea0c52a00729fd"}},{"code_sha256_prefix":"9b28e2f8ef578805","entry":"REC_eval_step","repo":"valeoai/LLM_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/LLM_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b28e2f8ef578805"}},{"code_sha256_prefix":"d8cd5bf64f19e875","entry":"compute_and_save_infer_info","repo":"valeoai/LLM_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/LLM_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8cd5bf64f19e875"}},{"code_sha256_prefix":"9c81b8cf59bd8a8f","entry":"define_model_result_name","repo":"valeoai/LLM_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/LLM_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c81b8cf59bd8a8f"}},{"code_sha256_prefix":"b2acae0f31bbf97f","entry":"design_prompt_for_step_3","repo":"valeoai/LLM_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/LLM_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2acae0f31bbf97f"}},{"code_sha256_prefix":"1ee6940816fb4c9c","entry":"gen_step_3_with_HF","repo":"valeoai/LLM_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/LLM_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ee6940816fb4c9c"}},{"code_sha256_prefix":"14a0b26b9f4d5f93","entry":"get_all_rec_and_thres_dict_info_concat","repo":"valeoai/LLM_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/LLM_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14a0b26b9f4d5f93"}},{"code_sha256_prefix":"60c7a23e50408bb4","entry":"load_FT_model_via_base_model","repo":"valeoai/LLM_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/LLM_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60c7a23e50408bb4"}},{"code_sha256_prefix":"69336c48b701b32e","entry":"load_HF_model_tok","repo":"valeoai/LLM_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/LLM_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"69336c48b701b32e"}},{"code_sha256_prefix":"10a0bb17e5386ba9","entry":"unify_output_step_3","repo":"valeoai/LLM_wrapper","repo_kind":"official","path":"llm_wrapper/LLM_inference_script.py","file_url":"https://github.com/valeoai/LLM_wrapper/blob/HEAD/llm_wrapper/LLM_inference_script.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10a0bb17e5386ba9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}