{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/law-of-the-weakest-link-cross-capabilities-of","title":"Law of the Weakest Link: Cross Capabilities of Large Language Models","arxiv_id":"2409.19951","date":"2024-09-30","proceeding":null,"authors":["Ming Zhong","Aston Zhang","Xuewei Wang","Rui Hou","Wenhan Xiong","Chenguang Zhu","Zhengxing Chen","Liang Tan","Chloe Bi","Mike Lewis","Sravya Popuri","Sharan Narang","Melanie Kambadur","Dhruv Mahajan","Sergey Edunov","Jiawei Han","Laurens van der Maaten"],"abstract":"The development and evaluation of Large Language Models (LLMs) have largely focused on individual capabilities. However, this overlooks the intersection of multiple abilities across different types of expertise that are often required for real-world tasks, which we term cross capabilities. To systematically explore this concept, we first define seven core individual capabilities and then pair them to form seven common cross capabilities, each supported by a manually constructed taxonomy. Building on these definitions, we introduce CrossEval, a benchmark comprising 1,400 human-annotated prompts, with 100 prompts for each individual and cross capability. To ensure reliable evaluation, we involve expert annotators to assess 4,200 model responses, gathering 8,400 human ratings with detailed explanations to serve as reference examples. Our findings reveal that, in both static evaluations and attempts to enhance specific abilities, current LLMs consistently exhibit the \"Law of the Weakest Link,\" where cross-capability performance is significantly constrained by the weakest component. Specifically, across 58 cross-capability scores from 17 models, 38 scores are lower than all individual capabilities, while 20 fall between strong and weak, but closer to the weaker ability. These results highlight the under-performance of LLMs in cross-capability tasks, making the identification and improvement of the weakest capabilities a critical priority for future research to optimize performance in complex, multi-dimensional scenarios.","url_abs":"https://arxiv.org/abs/2409.19951v2","url_pdf":"https://arxiv.org/pdf/2409.19951v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"law-of-the-weakest-link-cross-capabilities-of","repo_url":"https://github.com/facebookresearch/llm-cross-capabilities","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2409.19951","atlas_url":"https://app.syntology.ai/?focus=2409.19951","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.19951"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/llm-cross-capabilities","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":7,"unverified":5},"by_repo_kind":{"official":{"samples":12,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"9f763c09d7528083","entry":"calculate_capability_correlations","repo":"facebookresearch/llm-cross-capabilities","repo_kind":"official","path":"correlation/calculate_correlations.py","file_url":"https://github.com/facebookresearch/llm-cross-capabilities/blob/HEAD/correlation/calculate_correlations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9f763c09d7528083"}},{"code_sha256_prefix":"cfc32396982754b4","entry":"calculate_correlations","repo":"facebookresearch/llm-cross-capabilities","repo_kind":"official","path":"correlation/calculate_correlations.py","file_url":"https://github.com/facebookresearch/llm-cross-capabilities/blob/HEAD/correlation/calculate_correlations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cfc32396982754b4"}},{"code_sha256_prefix":"81d218b2cceaedf1","entry":"is_file_empty","repo":"facebookresearch/llm-cross-capabilities","repo_kind":"official","path":"evaluation/evaluate_response.py","file_url":"https://github.com/facebookresearch/llm-cross-capabilities/blob/HEAD/evaluation/evaluate_response.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"81d218b2cceaedf1"}},{"code_sha256_prefix":"39c45495a9652f3c","entry":"is_file_empty","repo":"facebookresearch/llm-cross-capabilities","repo_kind":"official","path":"generate_response/generate.py","file_url":"https://github.com/facebookresearch/llm-cross-capabilities/blob/HEAD/generate_response/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"39c45495a9652f3c"}},{"code_sha256_prefix":"6e2463f0db7375ea","entry":"parse_principles","repo":"facebookresearch/llm-cross-capabilities","repo_kind":"official","path":"principle_prompting/generate_principles.py","file_url":"https://github.com/facebookresearch/llm-cross-capabilities/blob/HEAD/principle_prompting/generate_principles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6e2463f0db7375ea"}},{"code_sha256_prefix":"f73ce4685c268b2d","entry":"parse_score","repo":"facebookresearch/llm-cross-capabilities","repo_kind":"official","path":"evaluation/evaluate_response.py","file_url":"https://github.com/facebookresearch/llm-cross-capabilities/blob/HEAD/evaluation/evaluate_response.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f73ce4685c268b2d"}},{"code_sha256_prefix":"b4878d3dddc25ebb","entry":"resize_image","repo":"facebookresearch/llm-cross-capabilities","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/facebookresearch/llm-cross-capabilities/blob/HEAD/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b4878d3dddc25ebb"}},{"code_sha256_prefix":"ddff27ededc0c501","entry":"get_tinyurl","repo":"facebookresearch/llm-cross-capabilities","repo_kind":"official","path":"generate_response/models/utils.py","file_url":"https://github.com/facebookresearch/llm-cross-capabilities/blob/HEAD/generate_response/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ddff27ededc0c501"}},{"code_sha256_prefix":"45df32d170a47f3b","entry":"token_count","repo":"facebookresearch/llm-cross-capabilities","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/facebookresearch/llm-cross-capabilities/blob/HEAD/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"45df32d170a47f3b"}},{"code_sha256_prefix":"99aea99f3b3e7eac","entry":"token_count","repo":"facebookresearch/llm-cross-capabilities","repo_kind":"official","path":"generate_response/models/utils.py","file_url":"https://github.com/facebookresearch/llm-cross-capabilities/blob/HEAD/generate_response/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"99aea99f3b3e7eac"}},{"code_sha256_prefix":"66e63322b5a434b5","entry":"truncate","repo":"facebookresearch/llm-cross-capabilities","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/facebookresearch/llm-cross-capabilities/blob/HEAD/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"66e63322b5a434b5"}},{"code_sha256_prefix":"aba025460db71689","entry":"truncate","repo":"facebookresearch/llm-cross-capabilities","repo_kind":"official","path":"generate_response/models/utils.py","file_url":"https://github.com/facebookresearch/llm-cross-capabilities/blob/HEAD/generate_response/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"aba025460db71689"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}