{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/an-in-depth-look-at-gemini-s-language","title":"An In-depth Look at Gemini's Language Abilities","arxiv_id":"2312.11444","date":"2023-12-18","proceeding":null,"authors":["Syeda Nahida Akter","Zichun Yu","Aashiq Muhamed","Tianyue Ou","Alex Bäuerle","Ángel Alexander Cabrera","Krish Dholakia","Chenyan Xiong","Graham Neubig"],"abstract":"The recently released Google Gemini class of models are the first to comprehensively report results that rival the OpenAI GPT series across a wide variety of tasks. In this paper, we do an in-depth exploration of Gemini's language abilities, making two contributions. First, we provide a third-party, objective comparison of the abilities of the OpenAI GPT and Google Gemini models with reproducible code and fully transparent results. Second, we take a closer look at the results, identifying areas where one of the two model classes excels. We perform this analysis over 10 datasets testing a variety of language abilities, including reasoning, answering knowledge-based questions, solving math problems, translating between languages, generating code, and acting as instruction-following agents. From this analysis, we find that Gemini Pro achieves accuracy that is close but slightly inferior to the corresponding GPT 3.5 Turbo on all tasks that we benchmarked. We further provide explanations for some of this under-performance, including failures in mathematical reasoning with many digits, sensitivity to multiple-choice answer ordering, aggressive content filtering, and others. We also identify areas where Gemini demonstrates comparably high performance, including generation into non-English languages, and handling longer and more complex reasoning chains. Code and data for reproduction can be found at https://github.com/neulab/gemini-benchmark","url_abs":"https://arxiv.org/abs/2312.11444v2","url_pdf":"https://arxiv.org/pdf/2312.11444v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"an-in-depth-look-at-gemini-s-language","repo_url":"https://github.com/neulab/gemini-benchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"math","task_name":"Math"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.11444","atlas_url":"https://app.syntology.ai/?focus=2312.11444","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.11444"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/neulab/gemini-benchmark","reach":{"status":"ok"}}],"summary":{"ran":11,"unverified":1},"by_repo_kind":{"official":{"samples":12,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"fff28a1235bb78e3","entry":"add_indent","repo":"neulab/gemini-benchmark","repo_kind":"official","path":"benchmarking/Code/prompt.py","file_url":"https://github.com/neulab/gemini-benchmark/blob/HEAD/benchmarking/Code/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fff28a1235bb78e3"}},{"code_sha256_prefix":"cd9b3073b198bcf2","entry":"clean_prediction","repo":"neulab/gemini-benchmark","repo_kind":"official","path":"benchmarking/Code/processor.py","file_url":"https://github.com/neulab/gemini-benchmark/blob/HEAD/benchmarking/Code/processor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd9b3073b198bcf2"}},{"code_sha256_prefix":"c2df81b084b3c449","entry":"create_entry_point","repo":"neulab/gemini-benchmark","repo_kind":"official","path":"benchmarking/Code/prompt.py","file_url":"https://github.com/neulab/gemini-benchmark/blob/HEAD/benchmarking/Code/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c2df81b084b3c449"}},{"code_sha256_prefix":"ea91f5626d5fa702","entry":"get_prediction_dir","repo":"neulab/gemini-benchmark","repo_kind":"official","path":"benchmarking/Code/utils.py","file_url":"https://github.com/neulab/gemini-benchmark/blob/HEAD/benchmarking/Code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea91f5626d5fa702"}},{"code_sha256_prefix":"2cae86ab34075bc3","entry":"get_prediction_name","repo":"neulab/gemini-benchmark","repo_kind":"official","path":"benchmarking/Code/utils.py","file_url":"https://github.com/neulab/gemini-benchmark/blob/HEAD/benchmarking/Code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2cae86ab34075bc3"}},{"code_sha256_prefix":"4932749f4b820198","entry":"get_test_path","repo":"neulab/gemini-benchmark","repo_kind":"official","path":"benchmarking/Code/utils.py","file_url":"https://github.com/neulab/gemini-benchmark/blob/HEAD/benchmarking/Code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4932749f4b820198"}},{"code_sha256_prefix":"f6d679fa0ea419ac","entry":"remove_indent","repo":"neulab/gemini-benchmark","repo_kind":"official","path":"benchmarking/Code/prompt.py","file_url":"https://github.com/neulab/gemini-benchmark/blob/HEAD/benchmarking/Code/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f6d679fa0ea419ac"}},{"code_sha256_prefix":"cb87547454883316","entry":"select_fewshot_examples","repo":"neulab/gemini-benchmark","repo_kind":"official","path":"benchmarking/Code/run_code.py","file_url":"https://github.com/neulab/gemini-benchmark/blob/HEAD/benchmarking/Code/run_code.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb87547454883316"}},{"code_sha256_prefix":"6acf3db996d8c4f5","entry":"test_answer_mmlu_","repo":"neulab/gemini-benchmark","repo_kind":"official","path":"benchmarking/MMLU/utils.py","file_url":"https://github.com/neulab/gemini-benchmark/blob/HEAD/benchmarking/MMLU/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6acf3db996d8c4f5"}},{"code_sha256_prefix":"db42ba0a29b6cff8","entry":"test_answer_mmlu_claude","repo":"neulab/gemini-benchmark","repo_kind":"official","path":"benchmarking/MMLU/utils.py","file_url":"https://github.com/neulab/gemini-benchmark/blob/HEAD/benchmarking/MMLU/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db42ba0a29b6cff8"}},{"code_sha256_prefix":"ff7e01003fa98d8a","entry":"test_answer_mmlu_claude_instant","repo":"neulab/gemini-benchmark","repo_kind":"official","path":"benchmarking/MMLU/utils.py","file_url":"https://github.com/neulab/gemini-benchmark/blob/HEAD/benchmarking/MMLU/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff7e01003fa98d8a"}},{"code_sha256_prefix":"78ca0bb4c28987ab","entry":"get_response","repo":"neulab/gemini-benchmark","repo_kind":"official","path":"benchmarking/Code/run_code.py","file_url":"https://github.com/neulab/gemini-benchmark/blob/HEAD/benchmarking/Code/run_code.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"78ca0bb4c28987ab"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}