{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/irt-router-effective-and-interpretable-multi","title":"IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory","arxiv_id":"2506.01048","date":"2025-06-01","proceeding":null,"authors":["Wei Song","Zhenya Huang","Cheng Cheng","Weibo Gao","Bihan Xu","Guanhao Zhao","Fei Wang","Runze Wu"],"abstract":"Large language models (LLMs) have demonstrated exceptional performance across a wide range of natural language tasks. However, selecting the optimal LLM to respond to a user query often necessitates a delicate balance between performance and cost. While powerful models deliver better results, they come at a high cost, whereas smaller models are more cost-effective but less capable. To address this trade-off, we propose IRT-Router, a multi-LLM routing framework that efficiently routes user queries to the most suitable LLM. Inspired by Item Response Theory (IRT), a psychological measurement methodology, IRT-Router explicitly models the relationship between LLM capabilities and user query attributes. This not only enables accurate prediction of response performance but also provides interpretable insights, such as LLM abilities and query difficulty. Additionally, we design an online query warm-up technique based on semantic similarity, further enhancing the online generalization capability of IRT-Router. Extensive experiments on 20 LLMs and 12 datasets demonstrate that IRT-Router outperforms most baseline methods in terms of effectiveness and interpretability. Its superior performance in cold-start scenarios further confirms the reliability and practicality of IRT-Router in real-world applications. Code is available at https://github.com/Mercidaiha/IRT-Router.","url_abs":"https://arxiv.org/abs/2506.01048v1","url_pdf":"https://arxiv.org/pdf/2506.01048v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"irt-router-effective-and-interpretable-multi","repo_url":"https://github.com/Mercidaiha/IRT-Router","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2506.01048","atlas_url":"https://app.syntology.ai/?focus=2506.01048","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.01048"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Mercidaiha/IRT-Router","reach":null}],"summary":{"ran":2,"ran_violates":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"b7f90ecce4c1423f","entry":"MIRT","repo":"Mercidaiha/IRT-Router","repo_kind":"official","path":"router/MIRT.py","file_url":"https://github.com/Mercidaiha/IRT-Router/blob/HEAD/router/MIRT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b7f90ecce4c1423f"}},{"code_sha256_prefix":"d645c1d516dc74e5","entry":"MIRTNet","repo":"Mercidaiha/IRT-Router","repo_kind":"official","path":"router/MIRT.py","file_url":"https://github.com/Mercidaiha/IRT-Router/blob/HEAD/router/MIRT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d645c1d516dc74e5"}},{"code_sha256_prefix":"0d01306fd4de58a5","entry":"irt2pl","repo":"Mercidaiha/IRT-Router","repo_kind":"official","path":"router/MIRT.py","file_url":"https://github.com/Mercidaiha/IRT-Router/blob/HEAD/router/MIRT.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0d01306fd4de58a5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}