{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/entropy-and-distance-based-predictors-from","title":"Entropy- and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisal","arxiv_id":"2212.11185","date":"2022-12-21","proceeding":null,"authors":["Byung-Doh Oh","William Schuler"],"abstract":"Transformer-based large language models are trained to make predictions about the next word by aggregating representations of previous tokens through their self-attention mechanism. In the field of cognitive modeling, such attention patterns have recently been interpreted as embodying the process of cue-based retrieval, in which attention over multiple targets is taken to generate interference and latency during retrieval. Under this framework, this work first defines an entropy-based predictor that quantifies the diffuseness of self-attention, as well as distance-based predictors that capture the incremental change in attention patterns across timesteps. Moreover, following recent studies that question the informativeness of attention weights, we also experiment with alternative methods for incorporating vector norms into attention weights. Regression experiments using predictors calculated from the GPT-2 language model show that these predictors deliver a substantially better fit to held-out self-paced reading and eye-tracking data over a rigorous baseline including GPT-2 surprisal. Additionally, the distance-based predictors generally demonstrated higher predictive power, with effect sizes of up to 6.59 ms per standard deviation on self-paced reading times (compared to 2.82 ms for surprisal) and 1.05 ms per standard deviation on eye-gaze durations (compared to 3.81 ms for surprisal).","url_abs":"https://arxiv.org/abs/2212.11185v1","url_pdf":"https://arxiv.org/pdf/2212.11185v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"entropy-and-distance-based-predictors-from","repo_url":"https://github.com/byungdoh/attn_dist","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"informativeness","task_name":"Informativeness"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-2","method_name":"GPT-2"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2212.11185","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.11185"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/byungdoh/attn_dist","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"unverified":8},"by_repo_kind":{"official":{"samples":11,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a727df5fd2f5599a","entry":"check_number_comma","repo":"byungdoh/attn_dist","repo_kind":"official","path":"huggingface/src/transformers/convert_slow_tokenizer.py","file_url":"https://github.com/byungdoh/attn_dist/blob/HEAD/huggingface/src/transformers/convert_slow_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a727df5fd2f5599a"}},{"code_sha256_prefix":"d770e1dd1d799a47","entry":"ensure_valid_input","repo":"byungdoh/attn_dist","repo_kind":"official","path":"huggingface/src/transformers/convert_graph_to_onnx.py","file_url":"https://github.com/byungdoh/attn_dist/blob/HEAD/huggingface/src/transformers/convert_graph_to_onnx.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d770e1dd1d799a47"}},{"code_sha256_prefix":"7efe3db5c9e2f94d","entry":"generate_identified_filename","repo":"byungdoh/attn_dist","repo_kind":"official","path":"huggingface/src/transformers/convert_graph_to_onnx.py","file_url":"https://github.com/byungdoh/attn_dist/blob/HEAD/huggingface/src/transformers/convert_graph_to_onnx.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7efe3db5c9e2f94d"}},{"code_sha256_prefix":"82e5a32c9b2414f3","entry":"convert_slow_tokenizer","repo":"byungdoh/attn_dist","repo_kind":"official","path":"huggingface/src/transformers/convert_slow_tokenizer.py","file_url":"https://github.com/byungdoh/attn_dist/blob/HEAD/huggingface/src/transformers/convert_slow_tokenizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"82e5a32c9b2414f3"}},{"code_sha256_prefix":"c551a79d5bb1b062","entry":"detect_overflow","repo":"byungdoh/attn_dist","repo_kind":"official","path":"huggingface/src/transformers/debug_utils.py","file_url":"https://github.com/byungdoh/attn_dist/blob/HEAD/huggingface/src/transformers/debug_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c551a79d5bb1b062"}},{"code_sha256_prefix":"37a5eed2dbd663ca","entry":"gelu_fast","repo":"byungdoh/attn_dist","repo_kind":"official","path":"huggingface/src/transformers/activations_tf.py","file_url":"https://github.com/byungdoh/attn_dist/blob/HEAD/huggingface/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37a5eed2dbd663ca"}},{"code_sha256_prefix":"b5db92a796ef38f1","entry":"get_abs_min_max","repo":"byungdoh/attn_dist","repo_kind":"official","path":"huggingface/src/transformers/debug_utils.py","file_url":"https://github.com/byungdoh/attn_dist/blob/HEAD/huggingface/src/transformers/debug_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5db92a796ef38f1"}},{"code_sha256_prefix":"dd3166a87bc972ff","entry":"get_configuration_file","repo":"byungdoh/attn_dist","repo_kind":"official","path":"huggingface/src/transformers/configuration_utils.py","file_url":"https://github.com/byungdoh/attn_dist/blob/HEAD/huggingface/src/transformers/configuration_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd3166a87bc972ff"}},{"code_sha256_prefix":"0da835817b8a57bc","entry":"infer_shapes","repo":"byungdoh/attn_dist","repo_kind":"official","path":"huggingface/src/transformers/convert_graph_to_onnx.py","file_url":"https://github.com/byungdoh/attn_dist/blob/HEAD/huggingface/src/transformers/convert_graph_to_onnx.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0da835817b8a57bc"}},{"code_sha256_prefix":"cc8c8c3ebf0c343f","entry":"mish","repo":"byungdoh/attn_dist","repo_kind":"official","path":"huggingface/src/transformers/activations_tf.py","file_url":"https://github.com/byungdoh/attn_dist/blob/HEAD/huggingface/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc8c8c3ebf0c343f"}},{"code_sha256_prefix":"e2d56cb91f999bef","entry":"quick_gelu","repo":"byungdoh/attn_dist","repo_kind":"official","path":"huggingface/src/transformers/activations_tf.py","file_url":"https://github.com/byungdoh/attn_dist/blob/HEAD/huggingface/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e2d56cb91f999bef"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}