{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/roles-of-scaling-and-instruction-tuning-in","title":"Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention","arxiv_id":"2310.19084","date":"2023-10-29","proceeding":null,"authors":["Changjiang Gao","ShuJian Huang","Jixing Li","Jiajun Chen"],"abstract":"Recent large language models (LLMs) have revealed strong abilities to understand natural language. Since most of them share the same basic structure, i.e. the transformer block, possible contributors to their success in the training process are scaling and instruction tuning. However, how these factors affect the models' language perception is unclear. This work compares the self-attention of several existing LLMs (LLaMA, Alpaca and Vicuna) in different sizes (7B, 13B, 30B, 65B), together with eye saccade, an aspect of human reading attention, to assess the effect of scaling and instruction tuning on language perception. Results show that scaling enhances the human resemblance and improves the effective attention by reducing the trivial pattern reliance, while instruction tuning does not. However, instruction tuning significantly enhances the models' sensitivity to instructions. We also find that current LLMs are consistently closer to non-native than native speakers in attention, suggesting a sub-optimal language perception of all models. Our code and data used in the analysis is available on GitHub.","url_abs":"https://arxiv.org/abs/2310.19084v1","url_pdf":"https://arxiv.org/pdf/2310.19084v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"roles-of-scaling-and-instruction-tuning-in","repo_url":"https://github.com/rivergao/human_llm_attention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2310.19084","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.19084"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rivergao/human_llm_attention","reach":null}],"summary":{"ran_fixture":3,"ran_honours":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"66c3753e2440cd42","entry":"draw_attention_graph","repo":"rivergao/human_llm_attention","repo_kind":"official","path":"utils/attention_flow.py","file_url":"https://github.com/rivergao/human_llm_attention/blob/HEAD/utils/attention_flow.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"66c3753e2440cd42"}},{"code_sha256_prefix":"c36998ed51eb1e9e","entry":"get_adjmat","repo":"rivergao/human_llm_attention","repo_kind":"official","path":"utils/attention_flow.py","file_url":"https://github.com/rivergao/human_llm_attention/blob/HEAD/utils/attention_flow.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c36998ed51eb1e9e"}},{"code_sha256_prefix":"7c84ea3a7b48cde7","entry":"preprocess","repo":"rivergao/human_llm_attention","repo_kind":"official","path":"utils/attention_flow.py","file_url":"https://github.com/rivergao/human_llm_attention/blob/HEAD/utils/attention_flow.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c84ea3a7b48cde7"}},{"code_sha256_prefix":"9200c54483f3cae0","entry":"tril_idx","repo":"rivergao/human_llm_attention","repo_kind":"official","path":"utils/regression/fmri_noise_ceiling.py","file_url":"https://github.com/rivergao/human_llm_attention/blob/HEAD/utils/regression/fmri_noise_ceiling.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9200c54483f3cae0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}