{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-touch-vision-and-language-dataset-for","title":"A Touch, Vision, and Language Dataset for Multimodal Alignment","arxiv_id":"2402.13232","date":"2024-02-20","proceeding":null,"authors":["Letian Fu","Gaurav Datta","Huang Huang","William Chung-Ho Panitch","Jaimyn Drake","Joseph Ortiz","Mustafa Mukadam","Mike Lambeta","Roberto Calandra","Ken Goldberg"],"abstract":"Touch is an important sensing modality for humans, but it has not yet been incorporated into a multimodal generative language model. This is partially due to the difficulty of obtaining natural language labels for tactile data and the complexity of aligning tactile readings with both visual observations and language descriptions. As a step towards bridging that gap, this work introduces a new dataset of 44K in-the-wild vision-touch pairs, with English language labels annotated by humans (10%) and textual pseudo-labels from GPT-4V (90%). We use this dataset to train a vision-language-aligned tactile encoder for open-vocabulary classification and a touch-vision-language (TVL) model for text generation using the trained encoder. Results suggest that by incorporating touch, the TVL model improves (+29% classification accuracy) touch-vision-language alignment over existing models trained on any pair of those modalities. Although only a small fraction of the dataset is human-labeled, the TVL model demonstrates improved visual-tactile understanding over GPT-4V (+12%) and open-source vision-language models (+32%) on a new touch-vision understanding benchmark. Code and data: https://tactile-vlm.github.io.","url_abs":"https://arxiv.org/abs/2402.13232v1","url_pdf":"https://arxiv.org/pdf/2402.13232v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-touch-vision-and-language-dataset-for","repo_url":"https://github.com/Max-Fu/tvl","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[{"slug":"tvl-dataset","name":"TVL Dataset","full_name":"Touch-Vision-Language Dataset"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.13232","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.13232"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Max-Fu/tvl","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ff9c8cb4ab823fdf","entry":"tac_padding","repo":"Max-Fu/tvl","repo_kind":"official","path":"tvl_enc/tacvis.py","file_url":"https://github.com/Max-Fu/tvl/blob/HEAD/tvl_enc/tacvis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff9c8cb4ab823fdf"}},{"code_sha256_prefix":"ea1a61a34df40d38","entry":"to_pil","repo":"Max-Fu/tvl","repo_kind":"official","path":"tvl_enc/tacvis.py","file_url":"https://github.com/Max-Fu/tvl/blob/HEAD/tvl_enc/tacvis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ea1a61a34df40d38"}},{"code_sha256_prefix":"9a73cbb66f31231c","entry":"unnormalize_fn","repo":"Max-Fu/tvl","repo_kind":"official","path":"tvl_enc/tacvis.py","file_url":"https://github.com/Max-Fu/tvl/blob/HEAD/tvl_enc/tacvis.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9a73cbb66f31231c"}},{"code_sha256_prefix":"4b3575a3a594d46a","entry":"construct_top_k_mask","repo":"Max-Fu/tvl","repo_kind":"official","path":"tvl_enc/loss.py","file_url":"https://github.com/Max-Fu/tvl/blob/HEAD/tvl_enc/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4b3575a3a594d46a"}},{"code_sha256_prefix":"22df917b46daf58b","entry":"handle_flash_attn","repo":"Max-Fu/tvl","repo_kind":"official","path":"tvl_enc/transformer_utils.py","file_url":"https://github.com/Max-Fu/tvl/blob/HEAD/tvl_enc/transformer_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"22df917b46daf58b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}