{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-fit-to-human-reading-times-via","title":"Temperature-scaling surprisal estimates improve fit to human reading times -- but does it do so for the \"right reasons\"?","arxiv_id":"2311.09325","date":"2023-11-15","proceeding":null,"authors":["Tong Liu","Iza Škrjanec","Vera Demberg"],"abstract":"A wide body of evidence shows that human language processing difficulty is predicted by the information-theoretic measure surprisal, a word's negative log probability in context. However, it is still unclear how to best estimate these probabilities needed for predicting human processing difficulty -- while a long-standing belief held that models with lower perplexity would provide more accurate estimates of word predictability, and therefore lead to better reading time predictions, recent work has shown that for very large models, psycholinguistic predictive power decreases. One reason could be that language models might be more confident of their predictions than humans, because they have had exposure to several magnitudes more data. In this paper, we test what effect temperature-scaling of large language model (LLM) predictions has on surprisal estimates and their predictive power of reading times of English texts. Firstly, we show that calibration of large language models typically improves with model size, i.e. poorer calibration cannot account for poorer fit to reading times. Secondly, we find that temperature-scaling probabilities lead to a systematically better fit to reading times (up to 89% improvement in delta log likelihood), across several reading time corpora. Finally, we show that this improvement in fit is chiefly driven by words that are composed of multiple subword tokens.","url_abs":"https://arxiv.org/abs/2311.09325v2","url_pdf":"https://arxiv.org/pdf/2311.09325v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-fit-to-human-reading-times-via","repo_url":"https://github.com/TongLiu-github/TemperatureSaling4RTs","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.09325","atlas_url":"https://app.syntology.ai/?focus=2311.09325","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.09325"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/TongLiu-github/TemperatureSaling4RTs","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"0b50bb10a44fd3b3","entry":"generate_surprisals","repo":"TongLiu-github/TemperatureSaling4RTs","repo_kind":"official","path":"PPP_calculation.py","file_url":"https://github.com/TongLiu-github/TemperatureSaling4RTs/blob/HEAD/PPP_calculation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b50bb10a44fd3b3"}},{"code_sha256_prefix":"4f610d31cb5a6ccd","entry":"surprisal_calculate","repo":"TongLiu-github/TemperatureSaling4RTs","repo_kind":"official","path":"PPP_calculation.py","file_url":"https://github.com/TongLiu-github/TemperatureSaling4RTs/blob/HEAD/PPP_calculation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4f610d31cb5a6ccd"}},{"code_sha256_prefix":"65beb14bd0cbb3a8","entry":"surprisal_calculate","repo":"TongLiu-github/TemperatureSaling4RTs","repo_kind":"official","path":"utils.py","file_url":"https://github.com/TongLiu-github/TemperatureSaling4RTs/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"65beb14bd0cbb3a8"}},{"code_sha256_prefix":"0dc0f4e35de062ea","entry":"valid_data_name","repo":"TongLiu-github/TemperatureSaling4RTs","repo_kind":"official","path":"PPP_calculation.py","file_url":"https://github.com/TongLiu-github/TemperatureSaling4RTs/blob/HEAD/PPP_calculation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0dc0f4e35de062ea"}},{"code_sha256_prefix":"a14d867dc8b2786a","entry":"get_surprisal_per_article","repo":"TongLiu-github/TemperatureSaling4RTs","repo_kind":"official","path":"utils.py","file_url":"https://github.com/TongLiu-github/TemperatureSaling4RTs/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a14d867dc8b2786a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}