GPT-MoE

GPT-MoE (Generative Pretrained Transformer-Mixture of Experts) is a model architecture where only a subset of the model's parameters (the 'experts') are activated for any given input, allowing for massive scale with manageable computational costs.