{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/observatory-characterizing-embeddings-of","title":"Observatory: Characterizing Embeddings of Relational Tables","arxiv_id":"2310.07736","date":"2023-10-05","proceeding":null,"authors":["Tianji Cong","Madelon Hulsebos","Zhenjie Sun","Paul Groth","H. V. Jagadish"],"abstract":"Language models and specialized table embedding models have recently demonstrated strong performance on many tasks over tabular data. Researchers and practitioners are keen to leverage these models in many new application contexts; but limited understanding of the strengths and weaknesses of these models, and the table representations they generate, makes the process of finding a suitable model for a given task reliant on trial and error. There is an urgent need to gain a comprehensive understanding of these models to minimize inefficiency and failures in downstream usage. To address this need, we propose Observatory, a formal framework to systematically analyze embedding representations of relational tables. Motivated both by invariants of the relational data model and by statistical considerations regarding data distributions, we define eight primitive properties, and corresponding measures to quantitatively characterize table embeddings for these properties. Based on these properties, we define an extensible framework to evaluate language and table embedding models. We collect and synthesize a suite of datasets and use Observatory to analyze nine such models. Our analysis provides insights into the strengths and weaknesses of learned representations over tables. We find, for example, that some models are sensitive to table structure such as column order, that functional dependencies are rarely reflected in embeddings, and that specialized table embedding models have relatively lower sample fidelity. Such insights help researchers and practitioners better anticipate model behaviors and select appropriate models for their downstream tasks, while guiding researchers in the development of new models.","url_abs":"https://arxiv.org/abs/2310.07736v3","url_pdf":"https://arxiv.org/pdf/2310.07736v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"observatory-characterizing-embeddings-of","repo_url":"https://github.com/superctj/observatory","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.07736","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.07736"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/superctj/observatory","reach":{"status":"ok"}}],"summary":{"ran":11,"unverified":4},"by_repo_kind":{"official":{"samples":15,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"ca81b47b86c16354","entry":"cell_based_process_table","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/hugging_face_cell_embeddings.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/hugging_face_cell_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca81b47b86c16354"}},{"code_sha256_prefix":"46feca30244200ef","entry":"cell_embedding_collate_fn","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/doduo_entity_embeddings.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/doduo_entity_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46feca30244200ef"}},{"code_sha256_prefix":"b8053f2c1ee08ab9","entry":"column_based_process_table","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/hugging_face_column_embeddings.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/hugging_face_column_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b8053f2c1ee08ab9"}},{"code_sha256_prefix":"eeb0d1c39beb711b","entry":"get_tapas_cell_embeddings","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/hugging_face_cell_embeddings.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/hugging_face_cell_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eeb0d1c39beb711b"}},{"code_sha256_prefix":"aea20ae08149b553","entry":"get_tapas_column_embeddings","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/hugging_face_column_embeddings.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/hugging_face_column_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aea20ae08149b553"}},{"code_sha256_prefix":"4922a6012c4066dd","entry":"process_one_table","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/example_doduo_entity_embeddings.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/example_doduo_entity_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4922a6012c4066dd"}},{"code_sha256_prefix":"bdc5cba97b33c234","entry":"read_jsonl","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/example_doduo_entity_embeddings.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/example_doduo_entity_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bdc5cba97b33c234"}},{"code_sha256_prefix":"85f43f90d81a96a3","entry":"row2strList","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/hugging_face_row_embeddings.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/hugging_face_row_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85f43f90d81a96a3"}},{"code_sha256_prefix":"06af7c088164f804","entry":"row_based_process_table","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/hugging_face_row_embeddings.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/hugging_face_row_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"06af7c088164f804"}},{"code_sha256_prefix":"b893b0fab2d36452","entry":"table2colList","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/hugging_face_cell_embeddings.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/hugging_face_cell_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b893b0fab2d36452"}},{"code_sha256_prefix":"009ccc8065ba8c75","entry":"table2colList","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/hugging_face_column_embeddings.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/hugging_face_column_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"009ccc8065ba8c75"}},{"code_sha256_prefix":"1c09f09aefb31bca","entry":"load_transformers_model","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/huggingface_models.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/huggingface_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1c09f09aefb31bca"}},{"code_sha256_prefix":"f3f78d8cbe6d2359","entry":"load_transformers_tokenizer","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/huggingface_models.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/huggingface_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3f78d8cbe6d2359"}},{"code_sha256_prefix":"3192b166b26a6da2","entry":"load_transformers_tokenizer_and_max_length","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/huggingface_models.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/huggingface_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3192b166b26a6da2"}},{"code_sha256_prefix":"17490ffb4881d31b","entry":"tapex_inference","repo":"superctj/observatory","repo_kind":"official","path":"observatory/models/tapex.py","file_url":"https://github.com/superctj/observatory/blob/HEAD/observatory/models/tapex.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17490ffb4881d31b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}