Papers › Visual Program Distillation: Distilling Tools and Programmatic Reasoning into...
Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models
Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, Ariel Fuxman
Solving complex visual tasks such as "Who invented the musical instrument on the right?" involves a composition of skills: understanding space, recognizing instruments, and also retrieving prior knowledge. Recent work shows promise by decomposing such tasks using a large language model (LLM) into an executable program that invokes specialized vision models. However, generated programs are error-prone: they omit necessary steps, include spurious ones, and are unable to recover when the specialized models give incorrect outputs. Moreover, they require loading multiple models, incurring high latency and computation costs. We propose Visual Program Distillation (VPD), an instruction tuning framework that produces a vision-language model (VLM) capable of solving complex visual tasks with a single forward pass. VPD distills the reasoning ability of LLMs by using them to sample multiple candidate programs, which are then executed and verified to identify a correct one. It translates each correct program into a language description of the reasoning steps, which are then distilled into a VLM. Extensive experiments show that VPD improves the VLM's ability to count, understand spatial relations, and reason compositionally. Our VPD-trained PaLI-X outperforms all prior VLMs, achieving state-of-the-art performance across complex vision tasks, including MMBench, OK-VQA, A-OKVQA, TallyQA, POPE, and Hateful Memes. An evaluation with human annotators also confirms that VPD improves model response factuality and consistency. Finally, experiments on content moderation demonstrate that VPD is also helpful for adaptation to real-world applications with limited data.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Meme Classification | Hateful Memes | PaLI-X-VPD | ROC-AUC | 0.892 | #3 of 17 | Archive leaderboard | report |
| Object Counting | TallyQA-Complex | PaLI-X-VPD | Accuracy | 76.6 | #2 of 6 | Archive leaderboard | report |
| Object Counting | TallyQA-Simple | PaLI-X-VPD | Accuracy | 86.2 | #2 of 6 | Archive leaderboard | report |
| Visual Question Answering (VQA) | A-OKVQA | PaLI-X-VPD | DA VQA Score | 68.2 | #2 of 15 | Archive leaderboard | report |
| Visual Question Answering (VQA) | A-OKVQA | PaLI-X-VPD | MC Accuracy | 80.4 | #2 of 15 | Archive leaderboard | report |
| Visual Question Answering (VQA) | GQA test-dev | PaLI-X-VPD | Accuracy | 67.3 | #2 of 17 | Archive leaderboard | report |
| Visual Question Answering (VQA) | OK-VQA | PaLI-X-VPD | Accuracy | 66.8 | #1 of 37 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections