{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/speech-language-models-lack-important-brain","title":"Speech language models lack important brain-relevant semantics","arxiv_id":"2311.04664","date":"2023-11-08","proceeding":null,"authors":["Subba Reddy Oota","Emin Çelik","Fatma Deniz","Mariya Toneva"],"abstract":"Despite known differences between reading and listening in the brain, recent work has shown that text-based language models predict both text-evoked and speech-evoked brain activity to an impressive degree. This poses the question of what types of information language models truly predict in the brain. We investigate this question via a direct approach, in which we systematically remove specific low-level stimulus features (textual, speech, and visual) from language model representations to assess their impact on alignment with fMRI brain recordings during reading and listening. Comparing these findings with speech-based language models reveals starkly different effects of low-level features on brain alignment. While text-based models show reduced alignment in early sensory regions post-removal, they retain significant predictive power in late language regions. In contrast, speech-based models maintain strong alignment in early auditory regions even after feature removal but lose all predictive power in late language regions. These results suggest that speech-based models provide insights into additional information processed by early auditory regions, but caution is needed when using them to model processing in late language regions. We make our code publicly available. [https://github.com/subbareddy248/speech-llm-brain]","url_abs":"https://arxiv.org/abs/2311.04664v2","url_pdf":"https://arxiv.org/pdf/2311.04664v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"speech-language-models-lack-important-brain","repo_url":"https://github.com/subbareddy248/speech-llm-brain","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.04664","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.04664"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/subbareddy248/speech-llm-brain","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a5357d613fccb792","entry":"R2","repo":"subbareddy248/speech-llm-brain","repo_kind":"official","path":"Brain_preditictions/residuals_text_speech.py","file_url":"https://github.com/subbareddy248/speech-llm-brain/blob/HEAD/Brain_preditictions/residuals_text_speech.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5357d613fccb792"}},{"code_sha256_prefix":"fcc1bc3f3b56a6ba","entry":"R2r","repo":"subbareddy248/speech-llm-brain","repo_kind":"official","path":"Brain_preditictions/residuals_text_speech.py","file_url":"https://github.com/subbareddy248/speech-llm-brain/blob/HEAD/Brain_preditictions/residuals_text_speech.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fcc1bc3f3b56a6ba"}},{"code_sha256_prefix":"4f8557633b1c0d9a","entry":"gaussianize","repo":"subbareddy248/speech-llm-brain","repo_kind":"official","path":"Brain_preditictions/SemanticModel.py","file_url":"https://github.com/subbareddy248/speech-llm-brain/blob/HEAD/Brain_preditictions/SemanticModel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f8557633b1c0d9a"}},{"code_sha256_prefix":"e213cc185a8bbe6e","entry":"gaussianize_mat","repo":"subbareddy248/speech-llm-brain","repo_kind":"official","path":"Brain_preditictions/SemanticModel.py","file_url":"https://github.com/subbareddy248/speech-llm-brain/blob/HEAD/Brain_preditictions/SemanticModel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e213cc185a8bbe6e"}},{"code_sha256_prefix":"ba8bba9251c87d26","entry":"interpdata","repo":"subbareddy248/speech-llm-brain","repo_kind":"official","path":"Brain_preditictions/interpdata.py","file_url":"https://github.com/subbareddy248/speech-llm-brain/blob/HEAD/Brain_preditictions/interpdata.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba8bba9251c87d26"}},{"code_sha256_prefix":"fcb81c1314d5e642","entry":"sincinterp1D","repo":"subbareddy248/speech-llm-brain","repo_kind":"official","path":"Brain_preditictions/interpdata.py","file_url":"https://github.com/subbareddy248/speech-llm-brain/blob/HEAD/Brain_preditictions/interpdata.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fcb81c1314d5e642"}},{"code_sha256_prefix":"a7ce946251ada3ec","entry":"sincinterp2D","repo":"subbareddy248/speech-llm-brain","repo_kind":"official","path":"Brain_preditictions/interpdata.py","file_url":"https://github.com/subbareddy248/speech-llm-brain/blob/HEAD/Brain_preditictions/interpdata.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7ce946251ada3ec"}},{"code_sha256_prefix":"b1d67e320cdbeecd","entry":"zscore","repo":"subbareddy248/speech-llm-brain","repo_kind":"official","path":"Brain_preditictions/SemanticModel.py","file_url":"https://github.com/subbareddy248/speech-llm-brain/blob/HEAD/Brain_preditictions/SemanticModel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1d67e320cdbeecd"}},{"code_sha256_prefix":"8cc15e414ba9c48a","entry":"corr","repo":"subbareddy248/speech-llm-brain","repo_kind":"official","path":"Brain_preditictions/residuals_text_speech.py","file_url":"https://github.com/subbareddy248/speech-llm-brain/blob/HEAD/Brain_preditictions/residuals_text_speech.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8cc15e414ba9c48a"}},{"code_sha256_prefix":"39fba6ccf7ab807c","entry":"load_low_level_speech_features","repo":"subbareddy248/speech-llm-brain","repo_kind":"official","path":"Brain_preditictions/brain_predictions_residuals.py","file_url":"https://github.com/subbareddy248/speech-llm-brain/blob/HEAD/Brain_preditictions/brain_predictions_residuals.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39fba6ccf7ab807c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}