{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sherlock-a-deep-learning-approach-to-semantic","title":"Sherlock: A Deep Learning Approach to Semantic Data Type Detection","arxiv_id":"1905.10688","date":"2019-05-25","proceeding":null,"authors":["Madelon Hulsebos","Kevin Hu","Michiel Bakker","Emanuel Zgraggen","Arvind Satyanarayan","Tim Kraska","Çağatay Demiralp","César Hidalgo"],"abstract":"Correctly detecting the semantic type of data columns is crucial for data science tasks such as automated data cleaning, schema matching, and data discovery. Existing data preparation and analysis systems rely on dictionary lookups and regular expression matching to detect semantic types. However, these matching-based approaches often are not robust to dirty data and only detect a limited number of types. We introduce Sherlock, a multi-input deep neural network for detecting semantic types. We train Sherlock on $686,765$ data columns retrieved from the VizNet corpus by matching $78$ semantic types from DBpedia to column headers. We characterize each matched column with $1,588$ features describing the statistical properties, character distributions, word embeddings, and paragraph vectors of column values. Sherlock achieves a support-weighted F$_1$ score of $0.89$, exceeding that of machine learning baselines, dictionary and regular expression benchmarks, and the consensus of crowdsourced annotations.","url_abs":"https://arxiv.org/abs/1905.10688v1","url_pdf":"https://arxiv.org/pdf/1905.10688v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sherlock-a-deep-learning-approach-to-semantic","repo_url":"https://github.com/mitmedialab/sherlock-project","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"sherlock-a-deep-learning-approach-to-semantic","repo_url":"https://github.com/pidahbus/DCoM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"column-type-annotation","task_name":"Column Type Annotation"},{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"table-annotation","task_name":"Table annotation"},{"task_slug":"type","task_name":"Vocal Bursts Type Prediction"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1905.10688","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1905.10688"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pidahbus/DCoM","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mitmedialab/sherlock-project","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":12},"by_repo_kind":{"official":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4d63572e07a8b26e","entry":"as_py_str","repo":"mitmedialab/sherlock-project","repo_kind":"official","path":"sherlock/functional.py","file_url":"https://github.com/mitmedialab/sherlock-project/blob/HEAD/sherlock/functional.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d63572e07a8b26e"}},{"code_sha256_prefix":"6cc16f28f1f61667","entry":"compute_stats","repo":"mitmedialab/sherlock-project","repo_kind":"official","path":"sherlock/features/stats_helper.py","file_url":"https://github.com/mitmedialab/sherlock-project/blob/HEAD/sherlock/features/stats_helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6cc16f28f1f61667"}},{"code_sha256_prefix":"7a01b569dcb0749c","entry":"count_pattern_in_cells","repo":"mitmedialab/sherlock-project","repo_kind":"official","path":"sherlock/features/bag_of_words.py","file_url":"https://github.com/mitmedialab/sherlock-project/blob/HEAD/sherlock/features/bag_of_words.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a01b569dcb0749c"}},{"code_sha256_prefix":"1b86d95ff04ca356","entry":"count_pattern_in_cells_with_non_zero_count","repo":"mitmedialab/sherlock-project","repo_kind":"official","path":"sherlock/features/bag_of_words.py","file_url":"https://github.com/mitmedialab/sherlock-project/blob/HEAD/sherlock/features/bag_of_words.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b86d95ff04ca356"}},{"code_sha256_prefix":"738c569048b50355","entry":"escape_for_regex","repo":"mitmedialab/sherlock-project","repo_kind":"official","path":"sherlock/features/helpers.py","file_url":"https://github.com/mitmedialab/sherlock-project/blob/HEAD/sherlock/features/helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"738c569048b50355"}},{"code_sha256_prefix":"142adc15ad2b37a7","entry":"keys_to_csv","repo":"mitmedialab/sherlock-project","repo_kind":"official","path":"sherlock/features/helpers.py","file_url":"https://github.com/mitmedialab/sherlock-project/blob/HEAD/sherlock/features/helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"142adc15ad2b37a7"}},{"code_sha256_prefix":"46c8f6a744c32f72","entry":"literal_eval_as_str","repo":"mitmedialab/sherlock-project","repo_kind":"official","path":"sherlock/features/helpers.py","file_url":"https://github.com/mitmedialab/sherlock-project/blob/HEAD/sherlock/features/helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"46c8f6a744c32f72"}},{"code_sha256_prefix":"4bd791ca6a3ba1b5","entry":"load_parquet_values","repo":"mitmedialab/sherlock-project","repo_kind":"official","path":"sherlock/features/preprocessing.py","file_url":"https://github.com/mitmedialab/sherlock-project/blob/HEAD/sherlock/features/preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4bd791ca6a3ba1b5"}},{"code_sha256_prefix":"f771a29940687aa4","entry":"mode","repo":"mitmedialab/sherlock-project","repo_kind":"official","path":"sherlock/features/stats_helper.py","file_url":"https://github.com/mitmedialab/sherlock-project/blob/HEAD/sherlock/features/stats_helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f771a29940687aa4"}},{"code_sha256_prefix":"8654506268afc092","entry":"random_sample","repo":"mitmedialab/sherlock-project","repo_kind":"official","path":"sherlock/functional.py","file_url":"https://github.com/mitmedialab/sherlock-project/blob/HEAD/sherlock/functional.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8654506268afc092"}},{"code_sha256_prefix":"03e3b47abb11dfd6","entry":"tokenise","repo":"mitmedialab/sherlock-project","repo_kind":"official","path":"sherlock/features/paragraph_vectors.py","file_url":"https://github.com/mitmedialab/sherlock-project/blob/HEAD/sherlock/features/paragraph_vectors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03e3b47abb11dfd6"}},{"code_sha256_prefix":"eec887fe5005331e","entry":"transpose","repo":"mitmedialab/sherlock-project","repo_kind":"official","path":"sherlock/features/word_embeddings.py","file_url":"https://github.com/mitmedialab/sherlock-project/blob/HEAD/sherlock/features/word_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eec887fe5005331e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}