{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/serval-synergy-learning-between-vertical","title":"SERVAL: Synergy Learning between Vertical Models and LLMs towards Oracle-Level Zero-shot Medical Prediction","arxiv_id":"2403.01570","date":"2024-03-03","proceeding":null,"authors":["Jiahuan Yan","Jintai Chen","Chaowen Hu","Bo Zheng","Yaojun Hu","Jimeng Sun","Jian Wu"],"abstract":"Recent development of large language models (LLMs) has exhibited impressive zero-shot proficiency on generic and common sense questions. However, LLMs' application on domain-specific vertical questions still lags behind, primarily due to the humiliation problems and deficiencies in vertical knowledge. Furthermore, the vertical data annotation process often requires labor-intensive expert involvement, thereby presenting an additional challenge in enhancing the model's vertical capabilities. In this paper, we propose SERVAL, a synergy learning pipeline designed for unsupervised development of vertical capabilities in both LLMs and small models by mutual enhancement. Specifically, SERVAL utilizes the LLM's zero-shot outputs as annotations, leveraging its confidence to teach a robust vertical model from scratch. Reversely, the trained vertical model guides the LLM fine-tuning to enhance its zero-shot capability, progressively improving both models through an iterative process. In medical domain, known for complex vertical knowledge and costly annotations, comprehensive experiments show that, without access to any gold labels, SERVAL with the synergy learning of OpenAI GPT-3.5 and a simple model attains fully-supervised competitive performance across ten widely used medical datasets. These datasets represent vertically specialized medical diagnostic scenarios (e.g., diabetes, heart diseases, COVID-19), highlighting the potential of SERVAL in refining the vertical capabilities of LLMs and training vertical models from scratch, all achieved without the need for annotations.","url_abs":"https://arxiv.org/abs/2403.01570v2","url_pdf":"https://arxiv.org/pdf/2403.01570v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"diagnostic","task_name":"Diagnostic"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2403.01570","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.01570"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jyansir/sersal","reach":null}],"summary":{"ran":1,"ran_draft_wrong":5,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1050129a2986d571","entry":"MultiheadGEAttention","repo":"jyansir/sersal","repo_kind":"found_in_text","path":"models/t2g.py","file_url":"https://github.com/jyansir/sersal/blob/HEAD/models/t2g.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1050129a2986d571"}},{"code_sha256_prefix":"d7bace13fb369137","entry":"geglu","repo":"jyansir/sersal","repo_kind":"found_in_text","path":"models/t2g.py","file_url":"https://github.com/jyansir/sersal/blob/HEAD/models/t2g.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7bace13fb369137"}},{"code_sha256_prefix":"7627d2c02619ab87","entry":"get_activation_fn","repo":"jyansir/sersal","repo_kind":"found_in_text","path":"models/t2g.py","file_url":"https://github.com/jyansir/sersal/blob/HEAD/models/t2g.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7627d2c02619ab87"}},{"code_sha256_prefix":"44d9a7feaba1dc68","entry":"get_nonglu_activation_fn","repo":"jyansir/sersal","repo_kind":"found_in_text","path":"models/t2g.py","file_url":"https://github.com/jyansir/sersal/blob/HEAD/models/t2g.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44d9a7feaba1dc68"}},{"code_sha256_prefix":"2fb665ac3ecd7c20","entry":"reglu","repo":"jyansir/sersal","repo_kind":"found_in_text","path":"models/t2g.py","file_url":"https://github.com/jyansir/sersal/blob/HEAD/models/t2g.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fb665ac3ecd7c20"}},{"code_sha256_prefix":"dc107686161af683","entry":"tanglu","repo":"jyansir/sersal","repo_kind":"found_in_text","path":"models/t2g.py","file_url":"https://github.com/jyansir/sersal/blob/HEAD/models/t2g.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc107686161af683"}},{"code_sha256_prefix":"0bbf2ee0b377a20d","entry":"T2GFormer","repo":"jyansir/sersal","repo_kind":"found_in_text","path":"models/t2g.py","file_url":"https://github.com/jyansir/sersal/blob/HEAD/models/t2g.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0bbf2ee0b377a20d"}},{"code_sha256_prefix":"d0e255c6c68af4b2","entry":"Tokenizer","repo":"jyansir/sersal","repo_kind":"found_in_text","path":"models/t2g.py","file_url":"https://github.com/jyansir/sersal/blob/HEAD/models/t2g.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0e255c6c68af4b2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}