{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mapping-between-fmri-responses-to-movies-and","title":"Mapping Between fMRI Responses to Movies and their Natural Language Annotations","arxiv_id":"1610.03914","date":"2016-10-13","proceeding":null,"authors":["Kiran Vodrahalli","Po-Hsuan Chen","YIngyu Liang","Christopher Baldassano","Janice Chen","Esther Yong","Christopher Honey","Uri Hasson","Peter Ramadge","Ken Norman","Sanjeev Arora"],"abstract":"Several research groups have shown how to correlate fMRI responses to the\nmeanings of presented stimuli. This paper presents new methods for doing so\nwhen only a natural language annotation is available as the description of the\nstimulus. We study fMRI data gathered from subjects watching an episode of BBCs\nSherlock [1], and learn bidirectional mappings between fMRI responses and\nnatural language representations. We show how to leverage data from multiple\nsubjects watching the same movie to improve the accuracy of the mappings,\nallowing us to succeed at a scene classification task with 72% accuracy (random\nguessing would give 4%) and at a scene ranking task with average rank in the\ntop 4% (random guessing would give 50%). The key ingredients are (a) the use of\nthe Shared Response Model (SRM) and its variant SRM-ICA [2, 3] to aggregate\nfMRI data from multiple subjects, both of which are shown to be superior to\nstandard PCA in producing low-dimensional representations for the tasks in this\npaper; (b) a sentence embedding technique adapted from the natural language\nprocessing (NLP) literature [4] that produces semantic vector representation of\nthe annotations; (c) using previous timestep information in the featurization\nof the predictor data.","url_abs":"http://arxiv.org/abs/1610.03914v3","url_pdf":"http://arxiv.org/pdf/1610.03914v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mapping-between-fmri-responses-to-movies-and","repo_url":"https://github.com/asprout/CPSC490","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"scene-classification","task_name":"Scene Classification"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"sentence-embedding","task_name":"Sentence Embedding"},{"task_slug":"sentence-embedding-1","task_name":"Sentence-Embedding"}],"methods":[{"method_slug":"pca","method_name":"PCA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1610.03914","atlas_url":"https://app.syntology.ai/?focus=1610.03914","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}