{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/instruct-me-more-random-prompting-for-visual","title":"Instruct Me More! Random Prompting for Visual In-Context Learning","arxiv_id":"2311.03648","date":"2023-11-07","proceeding":null,"authors":["Jiahao Zhang","Bowen Wang","Liangzhi Li","Yuta Nakashima","Hajime Nagahara"],"abstract":"Large-scale models trained on extensive datasets, have emerged as the preferred approach due to their high generalizability across various tasks. In-context learning (ICL), a popular strategy in natural language processing, uses such models for different tasks by providing instructive prompts but without updating model parameters. This idea is now being explored in computer vision, where an input-output image pair (called an in-context pair) is supplied to the model with a query image as a prompt to exemplify the desired output. The efficacy of visual ICL often depends on the quality of the prompts. We thus introduce a method coined Instruct Me More (InMeMo), which augments in-context pairs with a learnable perturbation (prompt), to explore its potential. Our experiments on mainstream tasks reveal that InMeMo surpasses the current state-of-the-art performance. Specifically, compared to the baseline without learnable prompt, InMeMo boosts mIoU scores by 7.35 and 15.13 for foreground segmentation and single object detection tasks, respectively. Our findings suggest that InMeMo offers a versatile and efficient way to enhance the performance of visual ICL with lightweight training. Code is available at https://github.com/Jackieam/InMeMo.","url_abs":"https://arxiv.org/abs/2311.03648v1","url_pdf":"https://arxiv.org/pdf/2311.03648v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"instruct-me-more-random-prompting-for-visual","repo_url":"https://github.com/jackieam/inmemo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"foreground-segmentation","task_name":"Foreground Segmentation"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.03648","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.03648"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jackieam/inmemo","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran_fixture":1,"unverified":7},"by_repo_kind":{"official":{"samples":9,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"221b2d116fdf1032","entry":"get_obj_from_str","repo":"jackieam/inmemo","repo_kind":"official","path":"vqgan.py","file_url":"https://github.com/jackieam/inmemo/blob/HEAD/vqgan.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"221b2d116fdf1032"}},{"code_sha256_prefix":"cb49209c125de1b4","entry":"get_timestep_embedding","repo":"jackieam/inmemo","repo_kind":"official","path":"vqgan.py","file_url":"https://github.com/jackieam/inmemo/blob/HEAD/vqgan.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb49209c125de1b4"}},{"code_sha256_prefix":"764136a5eff74434","entry":"calculate_metric","repo":"jackieam/inmemo","repo_kind":"official","path":"evaluate/evaluate_colorization.py","file_url":"https://github.com/jackieam/inmemo/blob/HEAD/evaluate/evaluate_colorization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"764136a5eff74434"}},{"code_sha256_prefix":"69532fa60013e9f4","entry":"calculate_metric","repo":"jackieam/inmemo","repo_kind":"official","path":"evaluate/evaluate_reasoning.py","file_url":"https://github.com/jackieam/inmemo/blob/HEAD/evaluate/evaluate_reasoning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69532fa60013e9f4"}},{"code_sha256_prefix":"c68f71948cfa9d7c","entry":"get_demo_predictions","repo":"jackieam/inmemo","repo_kind":"official","path":"viz_utils.py","file_url":"https://github.com/jackieam/inmemo/blob/HEAD/viz_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c68f71948cfa9d7c"}},{"code_sha256_prefix":"8c8d20b40760059c","entry":"get_demo_predictions_with_mask","repo":"jackieam/inmemo","repo_kind":"official","path":"viz_utils.py","file_url":"https://github.com/jackieam/inmemo/blob/HEAD/viz_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c8d20b40760059c"}},{"code_sha256_prefix":"f0ac78b4e05be194","entry":"instantiate_from_config","repo":"jackieam/inmemo","repo_kind":"official","path":"vqgan.py","file_url":"https://github.com/jackieam/inmemo/blob/HEAD/vqgan.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f0ac78b4e05be194"}},{"code_sha256_prefix":"ce2513917f2cc4b5","entry":"is_square","repo":"jackieam/inmemo","repo_kind":"official","path":"evaluate/evaluate_reasoning.py","file_url":"https://github.com/jackieam/inmemo/blob/HEAD/evaluate/evaluate_reasoning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce2513917f2cc4b5"}},{"code_sha256_prefix":"ebd3af8bde9488f1","entry":"reverse_trans","repo":"jackieam/inmemo","repo_kind":"official","path":"tta.py","file_url":"https://github.com/jackieam/inmemo/blob/HEAD/tta.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ebd3af8bde9488f1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}