</> Code with Papers by Syntology

Papers › Composing Ensembles of Pre-trained Models via Iterative Consensus

Composing Ensembles of Pre-trained Models via Iterative Consensus

20 Oct 2022arXiv:2210.11522archive 2025-07-28

Shuang Li, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Igor Mordatch

Large pre-trained models exhibit distinct and complementary capabilities dependent on the data they are trained on. Language models such as GPT-3 are capable of textual reasoning but cannot understand visual information, while vision models such as DALL-E can generate photorealistic photos but fail to understand complex language descriptions. In this work, we propose a unified framework for composing ensembles of different pre-trained models -- combining the strengths of each individual model to solve various multimodal problems in a zero-shot manner. We use pre-trained models as "generators" or "scorers" and compose them via closed-loop iterative consensus optimization. The generator constructs proposals and the scorers iteratively provide feedback to refine the generated result. Such closed-loop communication enables models to correct errors caused by other models, significantly boosting performance on downstream tasks, e.g. improving accuracy on grade school math problems by 7.5%, without requiring any model finetuning. We demonstrate that consensus achieved by an ensemble of scorers outperforms the feedback of a single scorer, by leveraging the strengths of each expert model. Results show that the proposed method can be used as a general purpose framework for a wide range of zero-shot multimodal tasks, such as image generation, video question answering, mathematical reasoning, and robotic manipulation. Project page: https://energy-based-model.github.io/composing-pretrained-models.

PaperPDF

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Arithmetic ReasoningImage GenerationMathMathematical ReasoningQuestion AnsweringVideo Question Answering

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Arithmetic Reasoning GSM8K GPT-2-Medium 355M + question-solution classifier (BS=5) Accuracy 20.8 #149 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K GPT-2-Medium 355M + question-solution classifier (BS=5) Parameters (Billion) 0.355 #149 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K GPT-2-Medium 355M (fine-tuned, BS=5) Accuracy 18.3 #151 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K GPT-2-Medium 355M (fine-tuned, BS=5) Parameters (Billion) 0.355 #151 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K GPT-2-Medium 355M + question-solution classifier (BS=1) Accuracy 16.8 #156 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K GPT-2-Medium 355M + question-solution classifier (BS=1) Parameters (Billion) 0.355 #156 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K GPT-2-Medium 355M (BS=5) Accuracy 12.2 #158 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K GPT-2-Medium 355M (BS=5) Parameters (Billion) 0.355 #158 of 164 Archive leaderboard report
Image Generation ImageNet 64x64 GLIDE + CLIP + CLS + CLS-FREE FID 29.184 #31 of 65 Archive leaderboard report
Image Generation ImageNet 64x64 GLIDE + CLIP + CLS + CLS-FREE Inception Score 34.952 #31 of 65 Archive leaderboard report
Image Generation ImageNet 64x64 GLIDE + CLIP + CLS + CLS-FREE KID 3.766 #31 of 65 Archive leaderboard report
Image Generation ImageNet 64x64 GLIDE + CLS-FREE FID 29.219 #32 of 65 Archive leaderboard report
Image Generation ImageNet 64x64 GLIDE + CLS-FREE Inception Score 25.926 #32 of 65 Archive leaderboard report
Image Generation ImageNet 64x64 GLIDE + CLS-FREE KID 5.325 #32 of 65 Archive leaderboard report
Image Generation ImageNet 64x64 GLIDE + CLIP FID 30.462 #33 of 65 Archive leaderboard report
Image Generation ImageNet 64x64 GLIDE + CLIP Inception Score 25.017 #33 of 65 Archive leaderboard report
Image Generation ImageNet 64x64 GLIDE + CLIP KID 6.174 #33 of 65 Archive leaderboard report
Image Generation ImageNet 64x64 GLIDE + CLS FID 30.871 #34 of 65 Archive leaderboard report
Image Generation ImageNet 64x64 GLIDE + CLS Inception Score 22.077 #34 of 65 Archive leaderboard report
Image Generation ImageNet 64x64 GLIDE +CLS KID 7.952 #65 of 65 Archive leaderboard report
Video Question Answering ActivityNet-QA GPT-2 + CLIP-14 + CLIP-multilingual (Zero-Shot) Accuracy 61.2 #1 of 36 Archive leaderboard report
Video Question Answering ActivityNet-QA GPT-2 + CLIP-32 (Zero-Shot) Accuracy 58.4 #2 of 36 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

AdamAttentionAttention DropoutBPECosine AnnealingDense ConnectionsDropoutGPT-3Layer NormalizationLinear LayerLinear Warmup With Cosine AnnealingMulti-Head AttentionResidual ConnectionSoftmaxWeight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections