{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/uncalibrated-models-can-improve-human-ai","title":"Uncalibrated Models Can Improve Human-AI Collaboration","arxiv_id":"2202.05983","date":"2022-02-12","proceeding":null,"authors":["Kailas Vodrahalli","Tobias Gerstenberg","James Zou"],"abstract":"In many practical applications of AI, an AI model is used as a decision aid for human users. The AI provides advice that a human (sometimes) incorporates into their decision-making process. The AI advice is often presented with some measure of \"confidence\" that the human can use to calibrate how much they depend on or trust the advice. In this paper, we present an initial exploration that suggests showing AI models as more confident than they actually are, even when the original AI is well-calibrated, can improve human-AI performance (measured as the accuracy and confidence of the human's final prediction after seeing the AI advice). We first train a model to predict human incorporation of AI advice using data from thousands of human-AI interactions. This enables us to explicitly estimate how to transform the AI's prediction confidence, making the AI uncalibrated, in order to improve the final human prediction. We empirically validate our results across four different tasks--dealing with images, text and tabular data--involving hundreds of human participants. We further support our findings with simulation analysis. Our findings suggest the importance of jointly optimizing the human-AI system as opposed to the standard paradigm of optimizing the AI model alone.","url_abs":"https://arxiv.org/abs/2202.05983v3","url_pdf":"https://arxiv.org/pdf/2202.05983v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"uncalibrated-models-can-improve-human-ai","repo_url":"https://github.com/kailas-v/human-ai-interactions","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2202.05983","atlas_url":"https://app.syntology.ai/?focus=2202.05983","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.05983"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kailas-v/human-ai-interactions","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2a7638d718164190","entry":"load_dataset","repo":"kailas-v/human-ai-interactions","repo_kind":"official","path":"haiid.py","file_url":"https://github.com/kailas-v/human-ai-interactions/blob/HEAD/haiid.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a7638d718164190"}},{"code_sha256_prefix":"91c5f21f9fa6e786","entry":"load_task","repo":"kailas-v/human-ai-interactions","repo_kind":"official","path":"haiid.py","file_url":"https://github.com/kailas-v/human-ai-interactions/blob/HEAD/haiid.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91c5f21f9fa6e786"}},{"code_sha256_prefix":"a6b4ca4905abac6d","entry":"load_tasks","repo":"kailas-v/human-ai-interactions","repo_kind":"official","path":"haiid.py","file_url":"https://github.com/kailas-v/human-ai-interactions/blob/HEAD/haiid.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6b4ca4905abac6d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}