{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/voxelprompt-a-vision-language-agent-for","title":"VoxelPrompt: A Vision-Language Agent for Grounded Medical Image Analysis","arxiv_id":"2410.08397","date":"2024-10-10","proceeding":null,"authors":["Andrew Hoopes","Victor Ion Butoi","John V. Guttag","Adrian V. Dalca"],"abstract":"We present VoxelPrompt, an agent-driven vision-language framework that tackles diverse radiological tasks through joint modeling of natural language, image volumes, and analytical metrics. VoxelPrompt is multi-modal and versatile, leveraging the flexibility of language interaction while providing quantitatively grounded image analysis. Given a variable number of 3D medical volumes, such as MRI and CT scans, VoxelPrompt employs a language agent that iteratively predicts executable instructions to solve a task specified by an input prompt. These instructions communicate with a vision network to encode image features and generate volumetric outputs (e.g., segmentations). VoxelPrompt interprets the results of intermediate instructions and plans further actions to compute discrete measures (e.g., tumor growth across a series of scans) and present relevant outputs to the user. We evaluate this framework in a sandbox of diverse neuroimaging tasks, and we show that the single VoxelPrompt model can delineate hundreds of anatomical and pathological features, measure many complex morphological properties, and perform open-language analysis of lesion characteristics. VoxelPrompt carries out these objectives with accuracy similar to that of fine-tuned, single-task models for segmentation and visual question-answering, while facilitating a much larger range of tasks. Therefore, by supporting accurate image processing with language interaction, VoxelPrompt provides comprehensive utility for numerous imaging tasks that traditionally require specialized models to address.","url_abs":"https://arxiv.org/abs/2410.08397v1","url_pdf":"https://arxiv.org/pdf/2410.08397v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"voxelprompt-a-vision-language-agent-for","repo_url":"https://github.com/dalcalab/voxel","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"medical-image-analysis","task_name":"Medical Image Analysis"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.08397","atlas_url":"https://app.syntology.ai/?focus=2410.08397","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.08397"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dalcalab/voxel","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bcc1c38f67fb9d91","entry":"cached_transferable","repo":"dalcalab/voxel","repo_kind":"official","path":"voxel/caching.py","file_url":"https://github.com/dalcalab/voxel/blob/HEAD/voxel/caching.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bcc1c38f67fb9d91"}},{"code_sha256_prefix":"848288d538f87c21","entry":"expand_slicing","repo":"dalcalab/voxel","repo_kind":"official","path":"voxel/slicing.py","file_url":"https://github.com/dalcalab/voxel/blob/HEAD/voxel/slicing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"848288d538f87c21"}},{"code_sha256_prefix":"6c74ad7a5798845e","entry":"slicing_to_coordinates","repo":"dalcalab/voxel","repo_kind":"official","path":"voxel/slicing.py","file_url":"https://github.com/dalcalab/voxel/blob/HEAD/voxel/slicing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c74ad7a5798845e"}},{"code_sha256_prefix":"09b012888a989fc7","entry":"cached","repo":"dalcalab/voxel","repo_kind":"official","path":"voxel/caching.py","file_url":"https://github.com/dalcalab/voxel/blob/HEAD/voxel/caching.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09b012888a989fc7"}},{"code_sha256_prefix":"c8fba5cb8d7c05a8","entry":"coordinates_to_slicing","repo":"dalcalab/voxel","repo_kind":"official","path":"voxel/slicing.py","file_url":"https://github.com/dalcalab/voxel/blob/HEAD/voxel/slicing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8fba5cb8d7c05a8"}},{"code_sha256_prefix":"c88f6db2a98446ed","entry":"gaussian_kernel_1d","repo":"dalcalab/voxel","repo_kind":"official","path":"voxel/filters.py","file_url":"https://github.com/dalcalab/voxel/blob/HEAD/voxel/filters.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c88f6db2a98446ed"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}