Browse › Computer Vision › Visual Question Answering › MM-Vet

Visual Question Answering archive 2025-07-28

MM-Vet Benchmark (Visual Question Answering)

231 rows 167 with code listed 2 metrics Dataset page

Syntology Since the archive (Syntology): none placed; 3 newer papers judged to report on this board so far.

MLLM Leaderboard

The archive carries no text for this table; the description above is the archive's text for the task Visual Question Answering. archive 2025-07-28

Over time archive 2025-07-28

Direction inferred from the metric name, not from the archive: Params (lower is better). Not inferred (points only, no best-so-far line): GPT-4 score. Points are placed at the row's paper date, not an evaluation date; 223 of 231 rows carry one.

The chart needs JavaScript; the table below carries every value.

Results archive 2025-07-28

Archive rows end at the archive snapshot, 2025-07-28: no result published after that date is in this table. Rank is the archive's row order at that snapshot; not re-ranked here. Metric values are the archive's strings. Column headers sort the table in your browser; each row keeps its archive rank.

Paper Code Ran Syntology Report
1 gemini-2.0-flash-exp 81.2±0.4 – – – not matched report
2 gemini-exp-1206 78.1±0.2 – – – not matched report
3 Gemini 1.5 Pro (gemini-1.5-pro-002) 76.9±0.1 – Paper Code 2024 linked, not harvested report
4 MMCTAgent (GPT-4 + GPT-4V) 74.24 – Paper – 2024 no code linked report
5 Claude 3.5 Sonnet (claude-3-5-sonnet-20240620) 74.2±0.2 – Paper – 2024 no code linked report
6 Qwen2-VL-72B 74.0 – Paper Code 2024 official (archive's flag): 4 ran · 12 ran (of which 1 constructed an object rather than computing a result; 10 with no instrument failure: 4 honoured, 1 violated, 5 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
7 InternVL2.5-78B 72.378B – Paper Code 2024 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 8 unverified report
8 GPT-4o +text rationale +IoT 72.2 – Paper – 2024 no code linked report
9 Lyra-Pro 71.474B – Paper Code 2024 official (archive's flag): 14 ran · 14 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 1 violated, 7 with no contract checked; 6 where Syntology's instrument failed) · 5 unverified report
10 GLM-4V-Plus 71.1 – Paper Code 2024 official (archive's flag): 12 ran · 12 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 2 honoured, 0 violated, 6 with no contract checked; 4 where Syntology's instrument failed) · 5 unverified report
11 Phantom-7B 70.8 – Paper Code 2024 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified report
12 GPT-4o (gpt-4o-2024-05-13) 69.3±0.1 – Paper Code 2023 community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified report
13 InternVL2.5-38B 68.838B – Paper Code 2024 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 8 unverified report
14 gpt-4o-mini-2024-07-18 68.6±0.1 – Paper Code 2023 community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified report
15 GPT-4V 67.7±0.3 – Paper Code 2023 community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified report
16 GPT-4V-Turbo-detail:high 67.6±0.1 – Paper Code 2023 community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified report
17 Qwen-VL-Max 66.6±0.5 – Paper Code 2023 official: harvested for another paper · 0 ran · 2 unverified report
18 Gemini 1.5 Pro (gemini-1.5-pro) 65.8±0.1 – Paper Code 2024 linked, not harvested report
19 InternVL2-26B (SGP, token ratio 64%) 65.6026B – Paper Code 2024 official: harvested, nothing ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 2 honoured, 2 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
20 Baichuan-Omni (7B) 65.4 – Paper Code 2024 linked, not harvested report
21 InternVL2.5-26B 65.026B – Paper Code 2024 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 8 unverified report
22 Qwen2-VL-7B (finetuned on GAP-VQA train) 64.954 – Paper – 2024 no code linked report
23 InternVL2-Llama3-76B 64.4 – – – not matched report
24 Gemini 1.0 Pro Vision (gemini-pro-vision) 64.3±0.4 – Paper Code 2023 linked, not harvested report
25 GLM4 Vision 63.9 – Paper Code 2023 linked, not harvested report
26 LLaVA-OneVision-72B 63.7 – Paper Code 2024 linked, not harvested report
27 Lyra-Base 63.59B – Paper Code 2024 official (archive's flag): 14 ran · 14 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 1 violated, 7 with no contract checked; 6 where Syntology's instrument failed) · 5 unverified report
28 InternVL2-26B (SGP, token ratio 35%) 63.2026B – Paper Code 2024 official: harvested, nothing ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 2 honoured, 2 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
29 InternVL 1.5 62.826B – Paper Code 2024 linked, not harvested report
30 InternVL2.5-8B 62.88B – Paper Code 2024 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 8 unverified report
31 MAmmoTH-VL-8B 62.3 – Paper Code 2024 linked, not harvested report
32 Qwen2-VL-7B 62.0 – Paper Code 2024 official (archive's flag): 4 ran · 12 ran (of which 1 constructed an object rather than computing a result; 10 with no instrument failure: 4 honoured, 1 violated, 5 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
33 InternVL2-40B 61.8 – – – not matched report
34 Qwen-VL-Plus 61.1±0.2 – Paper Code 2023 official: harvested for another paper · 0 ran · 2 unverified report
35 Mini-Gemini-HD-BS 60.8 – Paper Code 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 1 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified report
36 InternVL2.5-2B 60.82B – Paper Code 2024 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 8 unverified report
37 MAmmoTH-VL-8B (SI) 60.6 – Paper Code 2024 linked, not harvested report
38 InternVL2.5-4B 60.64B – Paper Code 2024 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 8 unverified report
39 GPT-4V-Turbo-detail:low 60.2±0.3 – Paper Code 2023 community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified report
40 Mini-Gemini-HD 59.3 – Paper Code 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 1 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified report
41 Claude 3 Opus (claude-3-opus-20240229) 58.1±0.1 – – – not matched report
42 GLM-4V-9B 58.0 – Paper Code 2024 official (archive's flag): 12 ran · 12 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 2 honoured, 0 violated, 6 with no contract checked; 4 where Syntology's instrument failed) · 5 unverified report
43 LLaVA-OneVision-7B 57.5 – Paper Code 2024 linked, not harvested report
44 LLaVA-NeXT-34B 57.434B – – – not matched report
45 Meteor 57.37B – Paper Code 2024 official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified report
46 CROME (Vicuna-13B) 55.1 – Paper – 2024 no code linked report
47 IXC2-4KHD 54.9 – Paper Code 2024 linked, not harvested report
48 Weitu-VL-1.0 54.715B – – – not matched report
49 TroL-7B 54.77B – Paper Code 2024 official (archive's flag): 23 ran · 23 ran (of which 11 constructed an object rather than computing a result; 12 with no instrument failure: 0 honoured, 0 violated, 12 with no contract checked; 11 where Syntology's instrument failed) · 7 unverified report
50 Mini-Gemini 53.0 – Paper Code 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 1 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified report
51 CogVLM(Vicuna-7B) 52.817B – Paper Code 2023 linked, not harvested report
52 CogAgent 52.818B – Paper Code 2023 official (archive's flag): 12 ran · 12 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 8 where Syntology's instrument failed) · 6 unverified report
53 Qwen2-VL-2B (finetuned on GAP-VQA train) 52.43 – Paper – 2024 no code linked report
54 InternVL2-26B (SGP, token ratio 9%) 52.1026B – Paper Code 2024 official: harvested, nothing ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 2 honoured, 2 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
55 MM1.5-30B 52.0 – Paper – 2024 no code linked report
56 MiniCPM-Llama3-V-2.5-8B (finetuned on GAP-VQA train) 51.789 – Paper – 2024 no code linked report
57 IXC-2.5-7B 51.7 – Paper Code 2024 official: no sample here; runs from other or unrecorded repositories · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
58 InternLM-XComposer2 51.2 – Paper Code 2024 linked, not harvested report
59 Lyra-Mini 51.23B – Paper Code 2024 official (archive's flag): 14 ran · 14 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 1 violated, 7 with no contract checked; 6 where Syntology's instrument failed) · 5 unverified report
60 CuMo-7B 51.07B – Paper Code 2024 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 9 where Syntology's instrument failed) · 1 unverified report
61 TACO (Qwen2-7B / SigLIP) 50.9 – Paper Code 2024 linked, not harvested report
62 Qwen-VL-Chat (+ SFT (GPT-4V in VLFeedback)) 50.7 – Paper – 2024 5 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 2 unverified report
63 POINTS-9B 50.0 – Paper – 2024 no code linked report
64 VILA^2-8B 50.0 – Paper – 2024 no code linked report
65 Janus-Pro-7B 50.0 – Paper Code 2025 linked, not harvested report
66 Silkie 49.97B – Paper – 2023 no code linked report
67 Silkie (Qwen-VL-Chat + DPO w/ VLFeedback) 49.9 – Paper – 2024 5 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 2 unverified report
68 Qwen2-VL-2B 49.5 – Paper Code 2024 official (archive's flag): 4 ran · 12 ran (of which 1 constructed an object rather than computing a result; 10 with no instrument failure: 4 honoured, 1 violated, 5 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
69 FlashSloth-HD 49.03.2B – Paper Code 2024 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified report
70 InternVL 1.2 48.940B – Paper Code 2024 linked, not harvested report
71 SEA-PRIME (Vicuna-13B) 48.8 – Paper – 2024 no code linked report
72 InternVL2.5-1B 48.81B – Paper Code 2024 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 8 unverified report
73 MM1-30B-Chat 48.7 – Paper – 2024 no code linked report
74 SETOKIM (13B) 48.713B – Paper – 2024 no code linked report
75 Emu2-Chat 48.537B – Paper Code 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified report
76 MG-LLaVA(34B) 48.5 – Paper Code 2024 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified report
77 SPHINX-Plus 47.9 – Paper Code 2024 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
78 ConvLLaVA 45.97B – Paper Code 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 6 where Syntology's instrument failed) · 2 unverified report
79 VILA-13B 45.7 – Paper Code 2023 linked, not harvested report
80 TACO (LLaMA3-8B / SigLIP) 45.7 – Paper Code 2024 linked, not harvested report
81 HPT 1.5 Edge 45.3 – – – not matched report
82 TACO (LLaMA3-8B / CLIP) 45.2 – Paper Code 2024 linked, not harvested report
83 LLaVA-v1.6 (7B, w/ STIC) 45.07B – Paper Code 2024 official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified report
84 H2OVL-Mississippi-2B 44.7 – Paper – 2024 no code linked report
85 PIIP-LLaVA (Vicuna-7B, ConvNeXt-L, CLIP-L ) 44.77B – Paper Code 2025 linked, not harvested report
86 MM-ReAct-GPT-4 44.6±0.2 – Paper Code 2023 official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified report
87 Imp-4B 44.64B – Paper Code 2024 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 3 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified report
88 LLaVA-Next-Mistral-7b (+ DPO w/ VLFeedback) 44.2 – Paper – 2024 5 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 2 unverified report
89 MGM-7B+RP 44.1 – Paper – 2024 no code linked report
90 LLaVA-Next-Vicuna-7b (+ DPO w/ VLFeedback) 44.1 – Paper – 2024 5 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 2 unverified report
91 VW-LMM 44.0 – Paper Code 2024 linked, not harvested report
92 MoAI 43.77B – Paper Code 2024 official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 1 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified report
93 MM1-3B-Chat 43.7 – Paper – 2024 no code linked report
94 MM1.5-3B-MoE 43.7 – Paper – 2024 no code linked report
95 Imp-3B 43.33B – Paper Code 2024 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 3 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified report
96 ShareGPT4V-13B 43.113B – Paper Code 2023 linked, not harvested report
97 Mini-Gemini (+MoCa) 42.9 – Paper Code 2024 official (archive's flag): 12 ran · 12 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 10 where Syntology's instrument failed) · 0 unverified report
98 MM1.5-7B 42.2 – Paper – 2024 no code linked report
99 MM1-7B-Chat 42.1 – Paper – 2024 no code linked report
100 FlashSloth 41.93.2B – Paper Code 2024 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified report
101 DeepSeek-VL 41.5 – Paper Code 2024 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 7 where Syntology's instrument failed) · 0 unverified report
102 LaVA1.5-13B-BPO 41.413B – Paper – 2024 no code linked report
103 ASMv2 41.3 – Paper Code 2024 official (archive's flag): 7 ran · 7 ran (of which 2 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified report
104 FocusLLaVA 41.3 – Paper – 2024 no code linked report
105 SeVa-13B 41.0 – Paper Code 2024 official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 3 honoured, 1 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified report
106 MM1.5-3B 41.0 – Paper – 2024 no code linked report
107 LLaVA-1.5-7B (VG-S) 40.4 – Paper Code 2024 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 8 unverified report
108 CoLLaVO 40.37B – Paper Code 2024 official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified report
109 SPHINX-2k 40.2 – Paper Code 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
110 LLaVA-1.5 (LVIS-Instrcut4V) 40.213B – Paper Code 2023 linked, not harvested report
111 mPLUG-Owl3 40.1 – Paper Code 2024 linked, not harvested report
112 Mono-InternVL-2B 40.12B – Paper – 2024 no code linked report
113 LLaVA1.5-13B-MDA 39.9013B – Paper – 2024 no code linked report
114 LLaVA-VT (Vicuna-13B) 39.8 – Paper Code 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 2 violated, 1 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified report
115 MM1.5-1B-MoE 39.8 – Paper – 2024 no code linked report
116 Janus-Pro-1B 39.8 – Paper Code 2025 linked, not harvested report
117 SQ-LLaVA∗ 39.7 – Paper Code 2024 official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified report
118 OmniFusion (grid split + ruDocVQA) 39.40 – Paper – 2024 no code linked report
119 DeepStack-L-HD (Vicuna-13B) 39.3 – Paper – 2024 no code linked report
120 LAF-13B 38.9 – Paper – 2024 no code linked report
121 InfiMM-HD 38.9 – Paper – 2024 no code linked report
122 InternLM-XC2 + MMDU-45k 38.8 – Paper Code 2024 official (archive's flag): 5 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 6 unverified report
123 LLaVA-1.5-7B (DC-S) 38.5 – Paper Code 2024 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 8 unverified report
124 LayoutLMv3+ConvNeXt+CLIP 38.47.9B – Paper Code 2024 linked, not harvested report
125 VOLCANO 13B 38.013B – Paper Code 2023 official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 1 violated, 1 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified report
126 LLaVA-1.5+MMInstruct (Vicuna-13B) 37.9 – Paper Code 2024 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
127 LLaVA-1.5-13B (+CSR) 37.813B – Paper Code 2024 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified report
128 LLaVA-1.5-LLaMA3-8B 37.8 – Paper – 2024 no code linked report
129 LLaVA-1.5 + DenseFusion-1M (Vicuna-7B) 37.8 – Paper Code 2024 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified report
130 ShareGPT4V-7B 37.67B – Paper Code 2023 linked, not harvested report
131 LLaVA-1.5+CoS 37.6 – Paper Code 2024 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
132 LLaVA-COCO-13B 37.5 – Paper – 2024 no code linked report
133 LLaVA-S^2 + DenseFusion-1M (Vicuna-7B) 37.5 – Paper Code 2024 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified report
134 MM1.5-1B 37.4 – Paper – 2024 no code linked report
135 Dynamic-LLaVA-13B 37.3 – Paper Code 2024 official (archive's flag): 13 ran · 13 ran (of which 4 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 1 violated, 4 with no contract checked; 8 where Syntology's instrument failed) · 5 unverified report
136 SeVa-7B 37.2 – Paper Code 2024 official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 3 honoured, 1 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified report
137 SoM-LLaVA-1.5-T 37.2 – Paper Code 2024 linked, not harvested report
138 Emu3 37.2 – Paper Code 2024 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified report
139 LLaVA-Instruct (Vicuna-1.5-13B) 37.1 – Paper Code 2024 linked, not harvested report
140 ILLUME 37.0 – Paper – 2024 no code linked report
141 LLaVA1.5-7B-BPO 36.87B – Paper – 2024 no code linked report
142 LLaVA-1.5-13B (+ MMFuser) 36.6 – Paper Code 2024 linked, not harvested report
143 CaMML-13B 36.4 – Paper Code 2024 linked, not harvested report
144 LLaVA-65B (Data Mixing) 36.4 – Paper Code 2023 official (archive's flag): 1 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified report
145 LLaVA-1.5-13B 36.3±0.213B – Paper Code 2023 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 3 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified report
146 Emu-14B 36.3±0.314B – Paper Code 2023 official: harvested for another paper · 0 ran · 2 unverified report
147 mPLUG-Owl2 36.3±0.17B – Paper Code 2023 official: no sample here; runs from other or unrecorded repositories · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified report
148 Vary-base 36.2 – Paper Code 2023 linked, not harvested report
149 StableLLaVA 36.1 – Paper Code 2023 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 2 honoured, 1 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified report
150 DreamLLM-7B 35.97B – Paper Code 2023 official (archive's flag): 1 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified report
151 MoE-LLaVA-2.7B×4-Top2 35.9 – Paper Code 2024 official: harvested, nothing ran · 0 ran · 2 unverified report
152 SoM-LLaVA-1.5 35.9 – Paper Code 2024 linked, not harvested report
153 Dragonfly (Llama3-8B) 35.9 – Paper Code 2024 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified report
154 Ferret-v2-13B 35.7 – Paper Code 2024 4 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified report
155 AlignGPT (Vicuna-13B) 35.6 – Paper – 2024 no code linked report
156 LLaVA-HR-X 35.5 – Paper Code 2024 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 2 violated, 1 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified report
157 SQ-LLaVA 35.5 – Paper Code 2024 official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified report
158 LOVA³ 35.27B – Paper Code 2024 official (archive's flag): 7 ran · 8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 3 honoured, 2 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified report
159 LLaVA-InternLM2-7B-ViT + MoSLoRA 35.2 – Paper Code 2024 official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified report
160 InternLM2+ViT (QMoSLoRA) 35.2 – Paper Code 2024 official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified report
161 LLaVA1.5-7B-MDA 35.207B – Paper – 2024 no code linked report
162 Mipha-3B+ 35.1 – Paper – 2024 no code linked report
163 LLaVA-Plus-13B (All Tools, V1.3, 336px) 35.0±0.013B – Paper Code 2023 official (archive's flag): 1 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
164 Merlin 34.9 – Paper – 2023 no code linked report
165 Arcana 34.8 – Paper – 2024 no code linked report
166 INF-LLaVA 34.5 – Paper Code 2024 official (archive's flag): 3 ran · 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified report
167 LLaVA-1.5+MMInstruct (Vicuna-7B) 34.4 – Paper Code 2024 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
168 Janus 34.3 – Paper Code 2024 linked, not harvested report
169 LLaVA-TokenPacker (Vicuna-13B) 34.1 – Paper Code 2024 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified report
170 γ-MoD-LLaVA-HR 34.0 – Paper – 2024 no code linked report
171 LLaVA-1.5-7B (CSR) 33.97B – Paper Code 2024 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified report
172 DynMOE-LLaVA 33.6 – Paper Code 2024 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified report
173 Imp-2B 33.52B – Paper Code 2024 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 3 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified report
174 InfMLLM-7B-Chat 33.4 – Paper Code 2023 linked, not harvested report
175 Video-LaVIT 33.27B – Paper Code 2024 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified report
176 LLaVA-Instruct (Vicuna-1.5-7B) 32.9 – Paper Code 2024 linked, not harvested report
177 VisionZip (Retain 128 Tokens, fine-tuning) 32.9 – Paper Code 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 3 unverified report
178 Uni-MoE 32.8 – Paper Code 2024 official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 2 unverified report
179 VL-Mamba (Mamba LLM-2.8B) 32.6 – Paper – 2024 no code linked report
180 LLaVA-v1.5 (7B, w/ STIC) 32.67B – Paper Code 2024 official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified report
181 VisionZip (Retain 192 Tokens, fine-tuning) 32.6 – Paper Code 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 3 unverified report
182 VisionZip (Retain 128 Tokens) 32.6 – Paper Code 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 3 unverified report
183 LLaVA-v1.5 (+MoCa) 32.2 – Paper Code 2024 official (archive's flag): 12 ran · 12 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 10 where Syntology's instrument failed) · 0 unverified report
184 Dynamic-LLaVA-7B 32.2 – Paper Code 2024 official (archive's flag): 13 ran · 13 ran (of which 4 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 1 violated, 4 with no contract checked; 8 where Syntology's instrument failed) · 5 unverified report
185 Mipha-3B 32.13B – Paper Code 2024 linked, not harvested report
186 VOLCANO 7B 32.07B – Paper Code 2023 official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 1 violated, 1 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified report
187 Video-LLaVA 32.0 – Paper Code 2023 community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified report
188 TinyLLaVA-share-Sig-Ph 32.03.1B – Paper Code 2024 official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 5 where Syntology's instrument failed) · 4 unverified report
189 LLaVA-VT (Vicuna-7B) 31.8 – Paper Code 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 2 violated, 1 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified report
190 LRV-Instruction-7B 31.7±0.17B – Paper Code 2023 linked, not harvested report
191 VisionZip (Retain 192 Tokens) 31.7 – Paper Code 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 3 unverified report
192 VisionZip (Retain 64 Tokens) 31.7 – Paper Code 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 3 unverified report
193 LLaVA-1.5-7B (+ SIMA) 31.67B – Paper Code 2024 official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified report
194 LLaMA-Adapter v2-7B 31.4±0.17B – Paper Code 2023 official: not harvested · 0 ran · 1 unverified report
195 MiCo-Chat-7B 31.4 – Paper Code 2024 linked, not harvested report
196 LLaVA-1.5-7B + TeamLoRA 31.2 – Paper Code 2024 linked, not harvested report
197 LLaVA-1.5-7B 31.1±0.27B – Paper Code 2023 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 3 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified report
198 RoboCodeX-13B 31.0 – Paper – 2024 no code linked report
199 HyperLLaVA 31.0 – Paper Code 2024 linked, not harvested report
200 FAST (Vicuna-7B) 31.0 – Paper Code 2024 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
201 JanusFlow 30.9 – Paper Code 2024 linked, not harvested report
202 AlignGPT (Vicuna-7B) 30.8 – Paper – 2024 no code linked report
203 LLaVolta 30.7 – Paper Code 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified report
204 LLaVA-AlignedVQ 30.7 – Paper – 2024 no code linked report
205 LLaVA-1.5-HACL 30.47.3B – Paper Code 2023 official (archive's flag): 1 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified report
206 MaVEn 30.47.2B – Paper – 2024 no code linked report
207 VisionZip (Retain 64 Tokens, fine-tuning) 30.2 – Paper Code 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 3 unverified report
208 H2OVL-Mississippi-0.8B 30.0 – Paper – 2024 no code linked report
209 RoboMamba 29.7 – Paper – 2024 no code linked report
210 LLaVA-TokenPacker (Vicuna-7B) 29.6 – Paper Code 2024 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified report
211 OneLLM-7B 29.17B – Paper Code 2023 official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
212 LLaVA-OneVision-0.5B 29.1 – Paper Code 2024 linked, not harvested report
213 Vary-toy 29.01.8B – Paper – 2024 no code linked report
214 LLaVA-Phi 28.9 – Paper Code 2024 official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified report
215 MM-ReAct-GPT-3.5 27.9±0.1 – Paper Code 2023 official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified report
216 MMAR-7B 27.80 – Paper – 2024 no code linked report
217 SEAL (7B) 27.77B – Paper Code 2023 official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
218 LLaVA-Plus-7B (All Tools) 27.5±0.37B – Paper Code 2023 official (archive's flag): 1 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
219 OtterHD-8B 26.38B – Paper Code 2023 official (archive's flag): 13 ran · 13 ran (of which 0 constructed an object rather than computing a result; 11 with no instrument failure: 0 honoured, 0 violated, 11 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified report
220 TGA-7B 25.6 – Paper – 2024 no code linked report
221 OpenFlamingo-9B (MPT-7B) 24.8±0.29B – Paper Code 2023 linked, not harvested report
222 Otter-9B (MPT-7B) 24.7±0.39B – Paper Code 2023 linked, not harvested report
223 Otter-9B (LLaMA) 24.6±0.29B – Paper Code 2023 linked, not harvested report
224 MiniGPT-4-14B 24.4±0.414B – Paper Code 2023 linked, not harvested report
225 LinVT 23.5 – Paper Code 2024 official (archive's flag): 7 ran · 10 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 3 honoured, 1 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified report
226 BLIP-2-12B 22.4±0.212B – Paper Code 2023 community repositories only · 4 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified; every one of the 4 samples that ran constructed an object rather than computing a result report
227 MiniGPT-4-8B 22.1±0.18B – Paper Code 2023 linked, not harvested report
228 OpenFlamingo-9B (LLaMA-7B) 21.8±0.19B – Paper Code 2023 linked, not harvested report
229 Xmodel-VLM (Xmodel-LM 1.1B) 21.8 – Paper Code 2024 linked, not harvested report
230 TextBind 19.4 – Paper Code 2023 linked, not harvested report
231 MMAR-0.5B 18.49 – Paper – 2024 no code linked report

All 231 rows shown. 223 link to a paper page on this site; 0 are marked as using additional training data in the archive. No GitHub stars are tracked; "Code" is the first repository the archive lists for the row. The archive carries no row tags, review links or community-submitted rows for this table; none are shown. archive 2025-07-28

Syntology Ran reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”: of the code samples Syntology harvested from repositories linked to that row's paper (joined by arXiv id), N executed on a synthesized input and U are unverified (harvested, no recorded run). Of the N, C constructed an object rather than computing a result (a part of N, never taken away from it), K ran with no instrument failure (honoured or violated the contract Syntology drafted, or ran with no contract checked) and I are failures of Syntology's instrument, not of the code (a wrong contract draft, or a contract check that could not run). It counts code from repositories linked to that row's paper, not this result: the row's number was not reproduced and nothing here is a correctness claim. The other cell texts mean no graph line for the row: "linked, not harvested" (the archive links code, Syntology has not harvested it), "no code linked" (no code link in the archive), "not matched" (the row's paper URL matched no paper on this site). 120 rows have a graph line, from 73 distinct papers. Rows whose paper has a sample that ran: 115 (94 with a run with no instrument failure, 21 where every run was a failure of Syntology's instrument). Papers: 69 with a sample that ran: 56 with a run with no instrument failure, 13 where every run was a failure of Syntology's instrument. Whose code ran. When the archive marks a repository official for the row's paper, the cell starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code): “official (archive's flag): N ran” counts the samples from it that ran, instrument failures included; “official: harvested, nothing ran”; when it has no sample for this paper and other samples ran, “community repositories only” if every one came from a repository the archive lists for the paper but neither marks official nor records as mentioned in it, or one Syntology harvested outside the archive's links, and “official: no sample here; runs from other or unrecorded repositories” if any came from a repository named in the paper or found in its text by Syntology, or has no repository recorded; “official: harvested for another paper” (no sample from it is linked to this paper) or “official: not harvested”. No state is printed when the archive marks none official. The cell's tooltip names the repositories the samples that ran came from. Of the 120 rows with a graph line, 111 have an archive-official repository: on 94 a sample from it ran (77 with a run with no instrument failure), on 7 the samples that ran came from community repositories only, and on 2 from other or unrecorded repositories. The filters above hide rows and never renumber them; the first two count, by default, only a run with no instrument failure, and offer any run, instrument failures included. Counting each paper once, of 525 samples: 412 ran (of which 22 constructed an object rather than computing a result; 194 with no instrument failure: 47 honoured, 43 violated, 104 with no contract checked; 218 where Syntology's instrument failed) · 113 unverified. Separately, 207 of those 525 are pointer-only (licence): the site points at that code rather than redistributing it, a licence property recorded for ran and unverified samples alike; each cell's tooltip carries the row's own pointer-only count. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. Per-sample status is on the paper page; each Ran cell links to that paper's sample list. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.

Since the archive Syntology

Of the 8,888 newer papers whose tables Syntology has read (8,886 checked in full and 2 with a leaderboard still awaiting a verdict), 6 had this board among the leaderboards the extractor considered for their tables (it keeps at most 10 boards per paper and 4 per dataset; the lists of 1,596 of the 8,888 were already full without this board, so if it was crowded out of them it is not counted), and 3 proposed a result; of those 3, 2 gave no cell that passed the extractor's own checks, 1 was rejected by the independent check, 0 await a verdict and 0 were placed. Syntology has checked 8,886 of the 18,637 papers on this site that are newer than the archive (for 3,717 of them no archive leaderboard matched the paper's tables, so there was nothing further to check); 733 were read and have nothing to place (arXiv has no HTML version of the paper, or that version has no tables), 41 could not be read (the extractor's reply could not be parsed), and results from the other 8,977 appear after they are checked.

Newer papers whose tables Claude Sonnet 4.5 judged to report on this benchmark (a model's judgement, not a result; newest first by arXiv id)

Paper arXiv date Ran Syntology Outcome
Localize and Neutralize: Gradient-guided Token Suppression against Visual Prompt... arXiv:2605.25194 24 May 2026 7 harvested: 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified refused by a rule: the row label the model quoted is not the row's label
FlexAC: Towards Flexible Control of Associative Reasoning in Multimodal Large Language Models arXiv:2510.11190 13 Oct 2025 6 harvested: 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified refused by a rule: the table's column header does not name the leaderboard's metric
Discrete Guidance Matching: Exact Guidance for Discrete Flow Matching arXiv:2509.21912 26 Sep 2025 6 harvested: 6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified rejected by the independent check

Syntology 3 newer papers judged to report on this board: the papers counted above as having proposed a result (0 placed); every one is listed here and in the JSON twin. The order is the arXiv identifier's, newest first, not a rank. The judgement is the model's, read from the paper's own tables; the outcome is what Syntology's checks did with it, and no value from a paper that was not placed is printed. “refused by a rule” names the extractor's rule; “no cell passed the extractor's checks” means the extractor's record holds no cell for this board, passed or refused. The 2 counted above as giving no cell that passed the extractor's own checks are the 2 marked “refused by a rule”. Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections