Papers › FETA: Towards Specializing Foundation Models for Expert Task Applications
FETA: Towards Specializing Foundation Models for Expert Task Applications
Amit Alfassy, Assaf Arbelle, Oshri Halimi, Sivan Harary, Roei Herzig, Eli Schwartz, Rameswar Panda, Michele Dolfi, Christoph Auer, Kate Saenko, PeterW. J. Staar, Rogerio Feris, Leonid Karlinsky
Foundation Models (FMs) have demonstrated unprecedented capabilities including zero-shot learning, high fidelity data synthesis, and out of domain generalization. However, as we show in this paper, FMs still have poor out-of-the-box performance on expert tasks (e.g. retrieval of car manuals technical illustrations from language queries), data for which is either unseen or belonging to a long-tail part of the data distribution of the huge datasets used for FM pre-training. This underlines the necessity to explicitly evaluate and finetune FMs on such expert tasks, arguably ones that appear the most in practical real-world applications. In this paper, we propose a first of its kind FETA benchmark built around the task of teaching FMs to understand technical documentation, via learning to match their graphical illustrations to corresponding language descriptions. Our FETA benchmark focuses on text-to-image and image-to-text retrieval in public car manuals and sales catalogue brochures. FETA is equipped with a procedure for completely automatic annotation extraction (code would be released upon acceptance), allowing easy extension of FETA to more documentation types and application domains in the future. Our automatic annotation leads to an automated performance metric shown to be consistent with metrics computed on human-curated annotations (also released). We provide multiple baselines and analysis of popular FMs on FETA leading to several interesting findings that we believe would be very valuable to the FM community, paving the way towards real-world application of FMs for practical expert tasks currently 'overlooked' by standard benchmarks focusing on common objects.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Datasets
Introduced by this paper, per the archive.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Image Retrieval | FETA Car-Manuals | FETA's CLIP-MIL (Many-Shot Image-to-text) | R@1 | 29 | #1 of 1 | Archive leaderboard | report |
| Image Retrieval | FETA Car-Manuals | FETA's CLIP-MIL (Many-Shot Image-to-text) | R@10 | 72.6 | #1 of 1 | Archive leaderboard | report |
| Image Retrieval | FETA Car-Manuals | FETA's CLIP-MIL (Many-Shot Image-to-text) | R@5 | 59.9 | #1 of 1 | Archive leaderboard | report |
| Image-to-Text Retrieval | FETA Car-Manuals | FETA's CLIP-MIL (Many-Shot Image-to-text) | R@1 | 35.5 | #1 of 1 | Archive leaderboard | report |
| Image-to-Text Retrieval | FETA Car-Manuals | FETA's CLIP-MIL (Many-Shot Image-to-text) | R@10 | 67 | #1 of 1 | Archive leaderboard | report |
| Image-to-Text Retrieval | FETA Car-Manuals | FETA's CLIP-MIL (Many-Shot Image-to-text) | R@5 | 58.3 | #1 of 1 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections