{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/zephyr-direct-distillation-of-lm-alignment","title":"Zephyr: Direct Distillation of LM Alignment","arxiv_id":"2310.16944","date":"2023-10-25","proceeding":null,"authors":["Lewis Tunstall","Edward Beeching","Nathan Lambert","Nazneen Rajani","Kashif Rasul","Younes Belkada","Shengyi Huang","Leandro von Werra","Clémentine Fourrier","Nathan Habib","Nathan Sarrazin","Omar Sanseviero","Alexander M. Rush","Thomas Wolf"],"abstract":"We aim to produce a smaller language model that is aligned to user intent. Previous research has shown that applying distilled supervised fine-tuning (dSFT) on larger models significantly improves task accuracy; however, these models are unaligned, i.e. they do not respond well to natural prompts. To distill this property, we experiment with the use of preference data from AI Feedback (AIF). Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment. The approach requires only a few hours of training without any additional sampling during fine-tuning. The final result, Zephyr-7B, sets the state-of-the-art on chat benchmarks for 7B parameter models, and requires no human annotation. In particular, results on MT-Bench show that Zephyr-7B surpasses Llama2-Chat-70B, the best open-access RLHF-based model. Code, models, data, and tutorials for the system are available at https://github.com/huggingface/alignment-handbook.","url_abs":"https://arxiv.org/abs/2310.16944v1","url_pdf":"https://arxiv.org/pdf/2310.16944v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"zephyr-direct-distillation-of-lm-alignment","repo_url":"https://github.com/huggingface/alignment-handbook","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"zephyr-direct-distillation-of-lm-alignment","repo_url":"https://github.com/Savannah120/alignment-handbook-PoFT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"2d-cyclist-detection","task_name":"2D Cyclist Detection"},{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/few-shot-learning-on-medconceptsqa","task":"Few-Shot Learning","dataset":"MedConceptsQA","model":"HuggingFaceH4/zephyr-7b-beta","rank_in_archive_order":9,"of":12,"metrics":{"Accuracy":"25.058"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-learning-on-medconceptsqa","task":"Zero-Shot Learning","dataset":"MedConceptsQA","model":"HuggingFaceH4/zephyr-7b-beta","rank_in_archive_order":7,"of":13,"metrics":{"Accuracy":"25.538"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.16944","atlas_url":"https://app.syntology.ai/?focus=2310.16944","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.16944"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Savannah120/alignment-handbook-PoFT","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/huggingface/alignment-handbook","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"d1e156f54d1adeec","entry":"compute_loss","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"d1e156f54d1adeec"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}