{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bring-your-own-data-self-supervised","title":"Bring Your Own Data! Self-Supervised Evaluation for Large Language Models","arxiv_id":"2306.13651","date":"2023-06-23","proceeding":null,"authors":["Neel Jain","Khalid Saifullah","Yuxin Wen","John Kirchenbauer","Manli Shu","Aniruddha Saha","Micah Goldblum","Jonas Geiping","Tom Goldstein"],"abstract":"With the rise of Large Language Models (LLMs) and their ubiquitous deployment in diverse domains, measuring language model behavior on realistic data is imperative. For example, a company deploying a client-facing chatbot must ensure that the model will not respond to client requests with profanity. Current evaluations approach this problem using small, domain-specific datasets with human-curated labels. These evaluation sets are often sampled from a narrow and simplified distribution, and data sources can unknowingly be leaked into the training set which can lead to misleading evaluations. To bypass these drawbacks, we propose a framework for self-supervised evaluation of LLMs by analyzing their sensitivity or invariance to transformations on the input text. Self-supervised evaluation can directly monitor LLM behavior on datasets collected in the wild or streamed during live model deployment. We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence, in addition to sensitivity to grammatical structure and tokenization errors. When comparisons to similar human-labeled benchmarks are available, we find strong correlations between self-supervised and human-supervised evaluations. The self-supervised paradigm complements current evaluation strategies that rely on labeled data.","url_abs":"https://arxiv.org/abs/2306.13651v2","url_pdf":"https://arxiv.org/pdf/2306.13651v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bring-your-own-data-self-supervised","repo_url":"https://github.com/neelsjain/byod","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"chatbot","task_name":"Chatbot"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"sensitivity","task_name":"Sensitivity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.13651","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.13651"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/neelsjain/byod","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":15},"by_repo_kind":{"official":{"samples":15,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"df6cd920ad87a22e","entry":"JSD","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/utils/JSD.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/utils/JSD.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df6cd920ad87a22e"}},{"code_sha256_prefix":"e5b60d3b4decd6d0","entry":"apply_transformation","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/negations.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/negations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5b60d3b4decd6d0"}},{"code_sha256_prefix":"0d4050f370c9da3d","entry":"chop_string_every_n_characters","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/tokenization_robustness.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/tokenization_robustness.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d4050f370c9da3d"}},{"code_sha256_prefix":"fadf2d5f80771cdc","entry":"filter_dataset","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/negations.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/negations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fadf2d5f80771cdc"}},{"code_sha256_prefix":"7f993e2595ba4016","entry":"filter_dataset","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/tokenization_robustness.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/tokenization_robustness.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f993e2595ba4016"}},{"code_sha256_prefix":"6745c66fbab09884","entry":"get_dataset","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/utils/hf_utils.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/utils/hf_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6745c66fbab09884"}},{"code_sha256_prefix":"2580b711e477a223","entry":"get_model_n_tokenizer","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/utils/hf_utils.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/utils/hf_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2580b711e477a223"}},{"code_sha256_prefix":"2699058fffaedefa","entry":"get_sent_pair_sens_score","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/word_order.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/word_order.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2699058fffaedefa"}},{"code_sha256_prefix":"a0aeb6547e602c03","entry":"llama_loading","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/utils/hf_utils.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/utils/hf_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0aeb6547e602c03"}},{"code_sha256_prefix":"04d514cc43e49457","entry":"lrs_metric","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/context_sensitivity.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/context_sensitivity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"04d514cc43e49457"}},{"code_sha256_prefix":"60628928ee38052d","entry":"negation_metric","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/negations.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/negations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60628928ee38052d"}},{"code_sha256_prefix":"678bf292ea6f7843","entry":"swap_words_in_sentence","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/word_order.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/word_order.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"678bf292ea6f7843"}},{"code_sha256_prefix":"af60a64429281b7c","entry":"tokenization_metric","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/tokenization_robustness.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/tokenization_robustness.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af60a64429281b7c"}},{"code_sha256_prefix":"22fc2c0a24b6d693","entry":"toxicity_metric","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/toxicity.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/toxicity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"22fc2c0a24b6d693"}},{"code_sha256_prefix":"e40cd3414e82866f","entry":"word_order_metric","repo":"neelsjain/byod","repo_kind":"official","path":"BYOD/word_order.py","file_url":"https://github.com/neelsjain/byod/blob/HEAD/BYOD/word_order.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e40cd3414e82866f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}