{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/beyond-appearances-material-segmentation-with","title":"Beyond Appearances: Material Segmentation with Embedded Spectral Information from RGB-D imagery","arxiv_id":null,"date":"2024-05-17","proceeding":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops 2024 5","authors":["Fabian Perez","Hoover Rueda-Chacón"],"abstract":"In the realm of computer vision material segmentation of natural scenes represents a challenge driven by the complex and diverse appearances of materials. Traditional approaches often rely on RGB images which can be deceptive given the variability in appearances due to different lighting conditions. Other methods that employ polarization or spectral imagery offer a more reliable material differentiation but their cost and accessibility restrict their everyday usage. In this work we propose a deep learning framework that bridges the gap between high-fidelity material segmentation and the practical constraints of data acquisition. Our approach leverages a training strategy that employs a paired RGBD-spectral data to incorporate spectral information directly within the neural network. This encoding process is facilitated by a Spectral Feature Mapper (SFM) layer a novel module that embeds unique spectral characteristics into the network thus enabling the network to infer materials from standard RGB-D images. Once trained the model allows to conduct material segmentation on widely available devices without the need for direct spectral data input. In addition we generate the 3D point cloud from the RGB-D image pair to provide a richer spatial context for scene understanding. Through simulations using available datasets and real experiments conducted with an iPad Pro our method demonstrates superior performance in material segmentation compared to other methods. Code is available at: https://github.com/Factral/Spectral-material-segmentation","url_abs":"https://openaccess.thecvf.com/content/CVPR2024W/LXCV/html/Perez_Beyond_Appearances_Material_Segmentation_with_Embedded_Spectral_Information_from_RGB-D_CVPRW_2024_paper.html","url_pdf":"https://openaccess.thecvf.com/content/CVPR2024W/LXCV/papers/Perez_Beyond_Appearances_Material_Segmentation_with_Embedded_Spectral_Information_from_RGB-D_CVPRW_2024_paper.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"beyond-appearances-material-segmentation-with","repo_url":"https://github.com/Factral/Spectral-Material-Segmentation","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"material-classification","task_name":"Material Classification"},{"task_slug":"material-recognition","task_name":"Material Recognition"},{"task_slug":"material-segmentation","task_name":"Material Segmentation"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"segmentation","task_name":"Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}