{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/retrospective-for-the-dynamic-sensorium","title":"Retrospective for the Dynamic Sensorium Competition for predicting large-scale mouse primary visual cortex activity from videos","arxiv_id":"2407.09100","date":"2024-07-12","proceeding":null,"authors":["Polina Turishcheva","Paul G. Fahey","Michaela Vystrčilová","Laura Hansel","Rachel Froebe","Kayla Ponder","Yongrong Qiu","Konstantin F. Willeke","Mohammad Bashiri","Ruslan Baikulov","Yu Zhu","Lei Ma","Shan Yu","Tiejun Huang","Bryan M. Li","Wolf De Wulf","Nina Kudryashova","Matthias H. Hennig","Nathalie L. Rochefort","Arno Onken","Eric Wang","Zhiwei Ding","Andreas S. Tolias","Fabian H. Sinz","Alexander S Ecker"],"abstract":"Understanding how biological visual systems process information is challenging because of the nonlinear relationship between visual input and neuronal responses. Artificial neural networks allow computational neuroscientists to create predictive models that connect biological and machine vision. Machine learning has benefited tremendously from benchmarks that compare different model on the same task under standardized conditions. However, there was no standardized benchmark to identify state-of-the-art dynamic models of the mouse visual system. To address this gap, we established the Sensorium 2023 Benchmark Competition with dynamic input, featuring a new large-scale dataset from the primary visual cortex of ten mice. This dataset includes responses from 78,853 neurons to 2 hours of dynamic stimuli per neuron, together with the behavioral measurements such as running speed, pupil dilation, and eye movements. The competition ranked models in two tracks based on predictive performance for neuronal responses on a held-out test set: one focusing on predicting in-domain natural stimuli and another on out-of-distribution (OOD) stimuli to assess model generalization. As part of the NeurIPS 2023 competition track, we received more than 160 model submissions from 22 teams. Several new architectures for predictive models were proposed, and the winning teams improved the previous state-of-the-art model by 50%. Access to the dataset as well as the benchmarking infrastructure will remain online at www.sensorium-competition.net.","url_abs":"https://arxiv.org/abs/2407.09100v1","url_pdf":"https://arxiv.org/pdf/2407.09100v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"retrospective-for-the-dynamic-sensorium","repo_url":"https://github.com/ecker-lab/sensorium_2023","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"retrospective-for-the-dynamic-sensorium","repo_url":"https://github.com/bryanlimy/ViV1T","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"pupil-dilation","task_name":"Pupil Dilation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.09100","atlas_url":"https://app.syntology.ai/?focus=2407.09100","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.09100"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ecker-lab/sensorium_2023","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/lRomul/sensorium","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/zhuyu-cs/Dynamic-VFM","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/bryanlimy/ViV1T","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":16,"ran_fixture":1,"unverified":4},"by_repo_kind":{"official":{"samples":12,"ran":10,"repositories":2},"found_in_text":{"samples":9,"ran":7,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"f6b87b79a83d122c","entry":"check_hyperparam_for_layers","repo":"zhuyu-cs/Dynamic-VFM","repo_kind":"found_in_text","path":"models/conv3d_core.py","file_url":"https://github.com/zhuyu-cs/Dynamic-VFM/blob/HEAD/models/conv3d_core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f6b87b79a83d122c"}},{"code_sha256_prefix":"cf761ce0ca742b66","entry":"check_output","repo":"bryanlimy/ViV1T","repo_kind":"official","path":"src/viv1t/utils/utils.py","file_url":"https://github.com/bryanlimy/ViV1T/blob/HEAD/src/viv1t/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf761ce0ca742b66"}},{"code_sha256_prefix":"3bbc3729e03c6d6c","entry":"correlation","repo":"bryanlimy/ViV1T","repo_kind":"official","path":"src/viv1t/metrics.py","file_url":"https://github.com/bryanlimy/ViV1T/blob/HEAD/src/viv1t/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3bbc3729e03c6d6c"}},{"code_sha256_prefix":"971ae8d2d8313e30","entry":"drop_path","repo":"lRomul/sensorium","repo_kind":"found_in_text","path":"src/models/dwiseneuro.py","file_url":"https://github.com/lRomul/sensorium/blob/HEAD/src/models/dwiseneuro.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"971ae8d2d8313e30"}},{"code_sha256_prefix":"b98b672b95d88ebe","entry":"get_device","repo":"bryanlimy/ViV1T","repo_kind":"official","path":"src/viv1t/utils/utils.py","file_url":"https://github.com/bryanlimy/ViV1T/blob/HEAD/src/viv1t/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b98b672b95d88ebe"}},{"code_sha256_prefix":"aaf130cf62109aaa","entry":"get_dims_for_loader_dict","repo":"ecker-lab/sensorium_2023","repo_kind":"official","path":"sensorium/models/utility.py","file_url":"https://github.com/ecker-lab/sensorium_2023/blob/HEAD/sensorium/models/utility.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aaf130cf62109aaa"}},{"code_sha256_prefix":"325390bf74ae81ee","entry":"get_dims_for_loader_dict","repo":"zhuyu-cs/Dynamic-VFM","repo_kind":"found_in_text","path":"models/utility.py","file_url":"https://github.com/zhuyu-cs/Dynamic-VFM/blob/HEAD/models/utility.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"325390bf74ae81ee"}},{"code_sha256_prefix":"4d1711af341a89b8","entry":"get_io_dims","repo":"ecker-lab/sensorium_2023","repo_kind":"official","path":"sensorium/models/utility.py","file_url":"https://github.com/ecker-lab/sensorium_2023/blob/HEAD/sensorium/models/utility.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d1711af341a89b8"}},{"code_sha256_prefix":"d3f6c644aed06d3f","entry":"get_io_dims","repo":"zhuyu-cs/Dynamic-VFM","repo_kind":"found_in_text","path":"models/utility.py","file_url":"https://github.com/zhuyu-cs/Dynamic-VFM/blob/HEAD/models/utility.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3f6c644aed06d3f"}},{"code_sha256_prefix":"7486444a4b765a39","entry":"get_module_output","repo":"zhuyu-cs/Dynamic-VFM","repo_kind":"found_in_text","path":"models/make_model_adaptive.py","file_url":"https://github.com/zhuyu-cs/Dynamic-VFM/blob/HEAD/models/make_model_adaptive.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7486444a4b765a39"}},{"code_sha256_prefix":"2387de91debc2c22","entry":"get_output_shape","repo":"bryanlimy/ViV1T","repo_kind":"official","path":"src/viv1t/model/critic.py","file_url":"https://github.com/bryanlimy/ViV1T/blob/HEAD/src/viv1t/model/critic.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2387de91debc2c22"}},{"code_sha256_prefix":"8185ec50f5bd1985","entry":"poisson_loss","repo":"bryanlimy/ViV1T","repo_kind":"official","path":"src/viv1t/metrics.py","file_url":"https://github.com/bryanlimy/ViV1T/blob/HEAD/src/viv1t/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8185ec50f5bd1985"}},{"code_sha256_prefix":"4a3eb40ef4963538","entry":"prepare_grid","repo":"ecker-lab/sensorium_2023","repo_kind":"official","path":"sensorium/models/utility.py","file_url":"https://github.com/ecker-lab/sensorium_2023/blob/HEAD/sensorium/models/utility.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4a3eb40ef4963538"}},{"code_sha256_prefix":"3dfa8cee58b9b8ad","entry":"prepare_grid","repo":"zhuyu-cs/Dynamic-VFM","repo_kind":"found_in_text","path":"models/utility.py","file_url":"https://github.com/zhuyu-cs/Dynamic-VFM/blob/HEAD/models/utility.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3dfa8cee58b9b8ad"}},{"code_sha256_prefix":"3c702d811d125a17","entry":"register","repo":"bryanlimy/ViV1T","repo_kind":"official","path":"src/viv1t/criterions.py","file_url":"https://github.com/bryanlimy/ViV1T/blob/HEAD/src/viv1t/criterions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c702d811d125a17"}},{"code_sha256_prefix":"4f1b2ed2a3391b71","entry":"shifted_window_attention_3d","repo":"zhuyu-cs/Dynamic-VFM","repo_kind":"found_in_text","path":"models/swin_core.py","file_url":"https://github.com/zhuyu-cs/Dynamic-VFM/blob/HEAD/models/swin_core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4f1b2ed2a3391b71"}},{"code_sha256_prefix":"657ce70f94e81f45","entry":"sse","repo":"bryanlimy/ViV1T","repo_kind":"official","path":"src/viv1t/metrics.py","file_url":"https://github.com/bryanlimy/ViV1T/blob/HEAD/src/viv1t/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"657ce70f94e81f45"}},{"code_sha256_prefix":"9448aaf59e5342b1","entry":"get_model_info","repo":"bryanlimy/ViV1T","repo_kind":"official","path":"src/viv1t/model/model.py","file_url":"https://github.com/bryanlimy/ViV1T/blob/HEAD/src/viv1t/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9448aaf59e5342b1"}},{"code_sha256_prefix":"8b8ce82c52c4e78c","entry":"make_video_model_adaptive","repo":"zhuyu-cs/Dynamic-VFM","repo_kind":"found_in_text","path":"models/make_model_adaptive.py","file_url":"https://github.com/zhuyu-cs/Dynamic-VFM/blob/HEAD/models/make_model_adaptive.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8b8ce82c52c4e78c"}},{"code_sha256_prefix":"a244a5150de05c50","entry":"make_video_model_mean","repo":"zhuyu-cs/Dynamic-VFM","repo_kind":"found_in_text","path":"models/make_model_mean.py","file_url":"https://github.com/zhuyu-cs/Dynamic-VFM/blob/HEAD/models/make_model_mean.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a244a5150de05c50"}},{"code_sha256_prefix":"e07554b34923da1b","entry":"support_bf16","repo":"bryanlimy/ViV1T","repo_kind":"official","path":"src/viv1t/utils/utils.py","file_url":"https://github.com/bryanlimy/ViV1T/blob/HEAD/src/viv1t/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e07554b34923da1b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}