Methods › Natural Language Processing › Language Models › Galactica
Galactica
Introduced by Ross Taylor et al. in Galactica: A Large Language Model for Science
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Galactica is a language model which uses a Transformer architecture in a decoder-only setup with the following modifications:
- It uses GeLU activations on all model sizes
- It uses a 2048 length context window for all model sizes
- It does not use biases in any of the dense kernels or layer norms
- It uses learned positional embeddings for the model
- A vocabulary of 50k tokens was constructed using BPE. The vocabulary was generated from a randomly selected 2% subset of the training data
Papers archive 2025-07-28
10 shown of 10, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
GeoGalactica: A Scientific Large Language Model in Geoscience 31 Dec 2023 · 1 repository · arXiv:2401.00434
-
TOP-Training: Target-Oriented Pretraining for Medical Extractive Question Answering 25 Oct 2023 · 1 repository · arXiv:2310.16995
-
Unlocking Model Insights: A Dataset for Automated Model Card Generation 22 Sep 2023 · 0 repositories · arXiv:2309.12616
-
Mitigating the Alignment Tax of RLHF 12 Sep 2023 · 1 repository · arXiv:2309.06256Syntology ran 1 of 1 samples · 0 unverified
-
Soft-prompt Tuning for Large Language Models to Evaluate Bias 7 Jun 2023 · 0 repositories · arXiv:2306.04735
-
How well do Large Language Models perform in Arithmetic tasks? 16 Mar 2023 · 1 repository · arXiv:2304.02015
-
Complex QA and language models hybrid architectures, Survey 17 Feb 2023 · 0 repositories · arXiv:2302.09051
-
ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots 8 Feb 2023 · 0 repositories · arXiv:2302.06466
-
ChatGPT is not all you need. A State of the Art Review of large Generative AI models 11 Jan 2023 · 0 repositories · arXiv:2301.04655
-
Galactica: A Large Language Model for Science 16 Nov 2022 · 1 repository · arXiv:2211.09085Syntology ran 0 of 2 samples · 2 unverified
Tasks archive 2025-07-28
20 shown of 43 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections