{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/programmable-3d-snapshot-microscopy-with","title":"FourierNets enable the design of highly non-local optical encoders for computational imaging","arxiv_id":"2104.10611","date":"2021-04-21","proceeding":null,"authors":["Diptodip Deb","Zhenfei Jiao","Ruth Sims","Alex B. Chen","Michael Broxton","Misha B. Ahrens","Kaspar Podgorski","Srinivas C. Turaga"],"abstract":"Differentiable simulations of optical systems can be combined with deep learning-based reconstruction networks to enable high performance computational imaging via end-to-end (E2E) optimization of both the optical encoder and the deep decoder. This has enabled imaging applications such as 3D localization microscopy, depth estimation, and lensless photography via the optimization of local optical encoders. More challenging computational imaging applications, such as 3D snapshot microscopy which compresses 3D volumes into single 2D images, require a highly non-local optical encoder. We show that existing deep network decoders have a locality bias which prevents the optimization of such highly non-local optical encoders. We address this with a decoder based on a shallow neural network architecture using global kernel Fourier convolutional neural networks (FourierNets). We show that FourierNets surpass existing deep network based decoders at reconstructing photographs captured by the highly non-local DiffuserCam optical encoder. Further, we show that FourierNets enable E2E optimization of highly non-local optical encoders for 3D snapshot microscopy. By combining FourierNets with a large-scale multi-GPU differentiable optical simulation, we are able to optimize non-local optical encoders 170$\\times$ to 7372$\\times$ larger than prior state of the art, and demonstrate the potential for ROI-type specific optical encoding with a programmable microscope.","url_abs":"https://arxiv.org/abs/2104.10611v6","url_pdf":"https://arxiv.org/pdf/2104.10611v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"programmable-3d-snapshot-microscopy-with","repo_url":"https://github.com/turagalab/snapshotscope","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":null,"task_name":"GPU"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.10611","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.10611"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/turagalab/snapshotscope","reach":null}],"summary":{"ran":4,"ran_honours":2,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b89fbfc92cb47718","entry":"DoubleConv","repo":"turagalab/snapshotscope","repo_kind":"official","path":"snapshotscope/networks/unet/model.py","file_url":"https://github.com/turagalab/snapshotscope/blob/HEAD/snapshotscope/networks/unet/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b89fbfc92cb47718"}},{"code_sha256_prefix":"caf37f4d286cdb2f","entry":"SingleConv","repo":"turagalab/snapshotscope","repo_kind":"official","path":"snapshotscope/networks/unet/model.py","file_url":"https://github.com/turagalab/snapshotscope/blob/HEAD/snapshotscope/networks/unet/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"caf37f4d286cdb2f"}},{"code_sha256_prefix":"dfb19fd861ab462d","entry":"Upsampling","repo":"turagalab/snapshotscope","repo_kind":"official","path":"snapshotscope/networks/unet/model.py","file_url":"https://github.com/turagalab/snapshotscope/blob/HEAD/snapshotscope/networks/unet/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfb19fd861ab462d"}},{"code_sha256_prefix":"08526b48215dcceb","entry":"_list","repo":"turagalab/snapshotscope","repo_kind":"official","path":"snapshotscope/networks/unet/model.py","file_url":"https://github.com/turagalab/snapshotscope/blob/HEAD/snapshotscope/networks/unet/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08526b48215dcceb"}},{"code_sha256_prefix":"4b9801b55ba77727","entry":"conv3d","repo":"turagalab/snapshotscope","repo_kind":"official","path":"snapshotscope/networks/unet/model.py","file_url":"https://github.com/turagalab/snapshotscope/blob/HEAD/snapshotscope/networks/unet/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b9801b55ba77727"}},{"code_sha256_prefix":"1c5e8fae51b1defa","entry":"create_conv","repo":"turagalab/snapshotscope","repo_kind":"official","path":"snapshotscope/networks/unet/model.py","file_url":"https://github.com/turagalab/snapshotscope/blob/HEAD/snapshotscope/networks/unet/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c5e8fae51b1defa"}},{"code_sha256_prefix":"c2d0bde8dded026c","entry":"initialize_optimizer","repo":"turagalab/snapshotscope","repo_kind":"official","path":"experiments/dlmd/fouriernet_mse_lpips/exp.py","file_url":"https://github.com/turagalab/snapshotscope/blob/HEAD/experiments/dlmd/fouriernet_mse_lpips/exp.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2d0bde8dded026c"}},{"code_sha256_prefix":"432a763e2bd52224","entry":"number_of_features_per_level","repo":"turagalab/snapshotscope","repo_kind":"official","path":"snapshotscope/networks/unet/model.py","file_url":"https://github.com/turagalab/snapshotscope/blob/HEAD/snapshotscope/networks/unet/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"432a763e2bd52224"}},{"code_sha256_prefix":"ef2c227915690bbb","entry":"Decoder","repo":"turagalab/snapshotscope","repo_kind":"official","path":"snapshotscope/networks/unet/model.py","file_url":"https://github.com/turagalab/snapshotscope/blob/HEAD/snapshotscope/networks/unet/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef2c227915690bbb"}},{"code_sha256_prefix":"441bb51000f08ada","entry":"FUNet3D","repo":"turagalab/snapshotscope","repo_kind":"official","path":"snapshotscope/networks/unet/model.py","file_url":"https://github.com/turagalab/snapshotscope/blob/HEAD/snapshotscope/networks/unet/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"441bb51000f08ada"}},{"code_sha256_prefix":"f589bc544a0c13a8","entry":"MultiscaleFourierConv2D","repo":"turagalab/snapshotscope","repo_kind":"official","path":"snapshotscope/networks/unet/model.py","file_url":"https://github.com/turagalab/snapshotscope/blob/HEAD/snapshotscope/networks/unet/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f589bc544a0c13a8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}