{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dirty-pixels-optimizing-image-classification","title":"Dirty Pixels: Towards End-to-End Image Processing and Perception","arxiv_id":"1701.06487","date":"2017-01-23","proceeding":null,"authors":["Steven Diamond","Vincent Sitzmann","Frank Julca-Aguilar","Stephen Boyd","Gordon Wetzstein","Felix Heide"],"abstract":"Real-world imaging systems acquire measurements that are degraded by noise, optical aberrations, and other imperfections that make image processing for human viewing and higher-level perception tasks challenging. Conventional cameras address this problem by compartmentalizing imaging from high-level task processing. As such, conventional imaging involves processing the RAW sensor measurements in a sequential pipeline of steps, such as demosaicking, denoising, deblurring, tone-mapping and compression. This pipeline is optimized to obtain a visually pleasing image. High-level processing, on the other hand, involves steps such as feature extraction, classification, tracking, and fusion. While this siloed design approach allows for efficient development, it also dictates compartmentalized performance metrics, without knowledge of the higher-level task of the camera system. For example, today's demosaicking and denoising algorithms are designed using perceptual image quality metrics but not with domain-specific tasks such as object detection in mind. We propose an end-to-end differentiable architecture that jointly performs demosaicking, denoising, deblurring, tone-mapping, and classification. The architecture learns processing pipelines whose outputs differ from those of existing ISPs optimized for perceptual quality, preserving fine detail at the cost of increased noise and artifacts. We demonstrate on captured and simulated data that our model substantially improves perception in low light and other challenging conditions, which is imperative for real-world applications. Finally, we found that the proposed model also achieves state-of-the-art accuracy when optimized for image reconstruction in low-light conditions, validating the architecture itself as a potentially useful drop-in network for reconstruction and analysis tasks beyond the applications demonstrated in this work.","url_abs":"https://arxiv.org/abs/1701.06487v2","url_pdf":"https://arxiv.org/pdf/1701.06487v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dirty-pixels-optimizing-image-classification","repo_url":"https://github.com/princeton-computational-imaging/DirtyPixels","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"deblurring","task_name":"Deblurring"},{"task_slug":"demosaicking","task_name":"Demosaicking"},{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-reconstruction","task_name":"Image Reconstruction"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"scene-segmentation","task_name":"Scene Segmentation"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"tone-mapping","task_name":"Tone Mapping"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1701.06487","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1701.06487"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/princeton-computational-imaging/DirtyPixels","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":6},"by_repo_kind":{"official":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b6e06004f7efe660","entry":"wrapped_partial","repo":"princeton-computational-imaging/DirtyPixels","repo_kind":"official","path":"nets/mobilenet_isp.py","file_url":"https://github.com/princeton-computational-imaging/DirtyPixels/blob/HEAD/nets/mobilenet_isp.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6e06004f7efe660"}},{"code_sha256_prefix":"841ffb13a2de6c74","entry":"anscombe","repo":"princeton-computational-imaging/DirtyPixels","repo_kind":"official","path":"nets/isp.py","file_url":"https://github.com/princeton-computational-imaging/DirtyPixels/blob/HEAD/nets/isp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"841ffb13a2de6c74"}},{"code_sha256_prefix":"c976f03875653135","entry":"build_number_to_synset","repo":"princeton-computational-imaging/DirtyPixels","repo_kind":"official","path":"datasets/build_imagenet_data.py","file_url":"https://github.com/princeton-computational-imaging/DirtyPixels/blob/HEAD/datasets/build_imagenet_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c976f03875653135"}},{"code_sha256_prefix":"074194387a3b5b47","entry":"inv_anscombe","repo":"princeton-computational-imaging/DirtyPixels","repo_kind":"official","path":"nets/isp.py","file_url":"https://github.com/princeton-computational-imaging/DirtyPixels/blob/HEAD/nets/isp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"074194387a3b5b47"}},{"code_sha256_prefix":"f406422099514e4c","entry":"lrelu","repo":"princeton-computational-imaging/DirtyPixels","repo_kind":"official","path":"nets/unet.py","file_url":"https://github.com/princeton-computational-imaging/DirtyPixels/blob/HEAD/nets/unet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f406422099514e4c"}},{"code_sha256_prefix":"b118e995c81f2377","entry":"optimize_clones","repo":"princeton-computational-imaging/DirtyPixels","repo_kind":"official","path":"deployment/model_deploy.py","file_url":"https://github.com/princeton-computational-imaging/DirtyPixels/blob/HEAD/deployment/model_deploy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b118e995c81f2377"}},{"code_sha256_prefix":"99be22ebba093e63","entry":"upsample_and_concat","repo":"princeton-computational-imaging/DirtyPixels","repo_kind":"official","path":"nets/unet.py","file_url":"https://github.com/princeton-computational-imaging/DirtyPixels/blob/HEAD/nets/unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99be22ebba093e63"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}