{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/beyond-task-performance-evaluating-and","title":"Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning","arxiv_id":"2310.00647","date":"2023-10-01","proceeding":null,"authors":["Mustafa Shukor","Alexandre Rame","Corentin Dancette","Matthieu Cord"],"abstract":"Following the success of Large Language Models (LLMs), Large Multimodal Models (LMMs), such as the Flamingo model and its subsequent competitors, have started to emerge as natural steps towards generalist agents. However, interacting with recent LMMs reveals major limitations that are hardly captured by the current evaluation benchmarks. Indeed, task performances (e.g., VQA accuracy) alone do not provide enough clues to understand their real capabilities, limitations, and to which extent such models are aligned to human expectations. To refine our understanding of those flaws, we deviate from the current evaluation paradigm, and (1) evaluate 10 recent open-source LMMs from 3B up to 80B parameter scale, on 5 different axes; hallucinations, abstention, compositionality, explainability and instruction following. Our evaluation on these axes reveals major flaws in LMMs. While the current go-to solution to align these models is based on training, such as instruction tuning or RLHF, we rather (2) explore the training-free in-context learning (ICL) as a solution, and study how it affects these limitations. Based on our ICL study, (3) we push ICL further and propose new multimodal ICL variants such as; Multitask-ICL, Chain-of-Hindsight-ICL, and Self-Correcting-ICL. Our findings are as follows. (1) Despite their success, LMMs have flaws that remain unsolved with scaling alone. (2) The effect of ICL on LMMs flaws is nuanced; despite its effectiveness for improved explainability, answer abstention, ICL only slightly improves instruction following, does not improve compositional abilities, and actually even amplifies hallucinations. (3) The proposed ICL variants are promising as post-hoc approaches to efficiently tackle some of those flaws. The code is available here: https://github.com/mshukor/EvALign-ICL.","url_abs":"https://arxiv.org/abs/2310.00647v2","url_pdf":"https://arxiv.org/pdf/2310.00647v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"beyond-task-performance-evaluating-and","repo_url":"https://github.com/mshukor/EvALign-ICL","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.00647","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.00647"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/mshukor/EvALign-ICL","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"ran":8,"ran_violates":1},"by_repo_kind":{"official":{"samples":11,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"4d75570b4e91d0b4","entry":"FeedForward","repo":"mshukor/EvALign-ICL","repo_kind":"official","path":"open_flamingo/src/helpers.py","file_url":"https://github.com/mshukor/EvALign-ICL/blob/HEAD/open_flamingo/src/helpers.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d75570b4e91d0b4"}},{"code_sha256_prefix":"ec8c233fb6abbe44","entry":"compute_classification_accuracy","repo":"mshukor/EvALign-ICL","repo_kind":"official","path":"open_flamingo/eval/classification.py","file_url":"https://github.com/mshukor/EvALign-ICL/blob/HEAD/open_flamingo/eval/classification.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec8c233fb6abbe44"}},{"code_sha256_prefix":"4fbef42363d6fcd6","entry":"compute_shifted_logits_and_labels","repo":"mshukor/EvALign-ICL","repo_kind":"official","path":"open_flamingo/eval/classification.py","file_url":"https://github.com/mshukor/EvALign-ICL/blob/HEAD/open_flamingo/eval/classification.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4fbef42363d6fcd6"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"mshukor/EvALign-ICL","repo_kind":"official","path":"open_flamingo/src/helpers.py","file_url":"https://github.com/mshukor/EvALign-ICL/blob/HEAD/open_flamingo/src/helpers.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"48d71ff25acc020d","entry":"get_prompt","repo":"mshukor/EvALign-ICL","repo_kind":"official","path":"open_flamingo/eval/caption_utils.py","file_url":"https://github.com/mshukor/EvALign-ICL/blob/HEAD/open_flamingo/eval/caption_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48d71ff25acc020d"}},{"code_sha256_prefix":"c5eb3a8dc84a42d6","entry":"get_prompt_caption_objects","repo":"mshukor/EvALign-ICL","repo_kind":"official","path":"open_flamingo/eval/caption_utils.py","file_url":"https://github.com/mshukor/EvALign-ICL/blob/HEAD/open_flamingo/eval/caption_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c5eb3a8dc84a42d6"}},{"code_sha256_prefix":"fe68fff801021b23","entry":"getattr_recursive","repo":"mshukor/EvALign-ICL","repo_kind":"official","path":"open_flamingo/src/utils.py","file_url":"https://github.com/mshukor/EvALign-ICL/blob/HEAD/open_flamingo/src/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe68fff801021b23"}},{"code_sha256_prefix":"fc91265a186bcbd4","entry":"postprocess_captioning_generation","repo":"mshukor/EvALign-ICL","repo_kind":"official","path":"open_flamingo/eval/coco_metric.py","file_url":"https://github.com/mshukor/EvALign-ICL/blob/HEAD/open_flamingo/eval/coco_metric.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc91265a186bcbd4"}},{"code_sha256_prefix":"a666b1cfb752e8e7","entry":"postprocess_classification_generation","repo":"mshukor/EvALign-ICL","repo_kind":"official","path":"open_flamingo/eval/classification.py","file_url":"https://github.com/mshukor/EvALign-ICL/blob/HEAD/open_flamingo/eval/classification.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a666b1cfb752e8e7"}},{"code_sha256_prefix":"d54937bcadecf500","entry":"postprocess_generation","repo":"mshukor/EvALign-ICL","repo_kind":"official","path":"open_flamingo/eval/caption_utils.py","file_url":"https://github.com/mshukor/EvALign-ICL/blob/HEAD/open_flamingo/eval/caption_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d54937bcadecf500"}},{"code_sha256_prefix":"86f1da2d2e3774cd","entry":"word_count","repo":"mshukor/EvALign-ICL","repo_kind":"official","path":"open_flamingo/eval/eval_datasets.py","file_url":"https://github.com/mshukor/EvALign-ICL/blob/HEAD/open_flamingo/eval/eval_datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"86f1da2d2e3774cd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}