Microsoft Develops New Large Language Model MAI-Base-2
Microsoft Corp. has introduced a new large language model (LLM) that it claims is significantly more efficient than current leading open-source alternatives. The company unveiled details about the algorithm through its artificial intelligence research group on Thursday.
Named MAI-Base-2, this model serves as a foundational base model rather than a fully-fledged LLM. Base models provide a minimalist prototype that developers can enhance through subsequent rounds of training, focusing on incorporating reasoning capabilities and safety measures.
The most advanced open-source base model currently available is DeepSeek-V4-Pro-Base, which Microsoft asserts MAI-Base-2 has surpassed in internal benchmarks that evaluate science, programming, and mathematical questions. Notably, MAI-Base-2 demonstrates up to 12 times greater hardware efficiency under certain metrics.
MAI-Base-2 employs a mixture of experts (MoE) architecture, which allows it to activate only a subset of its artificial neurons for processing prompts rather than employing the entire neural network. This architecture features 2 trillion parameters, with 80 billion activated for prompt processing, contributing to its enhanced efficiency.
In a language model, artificial neurons are structured into layers, and Microsoft previously blended MoE layers with a less efficient design called feedforward neural networks (FFN) in earlier models. However, MAI-Base-2 consists exclusively of MoE layers, marking a significant advancement in performance.
To train MAI-Base-2, Microsoft utilized 16,000 Nvidia Corp. GB200 chips over a 38-day period. The training dataset comprised 50 trillion tokens, with approximately a fifth allocated for a computer vision module integrated within the model. This integration enhances MAI-Base-2’s ability to interpret scientific charts more effectively than its predecessor.
During the training process, developers provide insight into task completion, allowing for iterative learning through feedback manifested as gradients. These gradients, which recalibrate each layer of the model to elevate output quality, can sometimes lead to bugs that hinder the training workflow. To mitigate this risk, Microsoft employed a technique known as gradient clipping while also leveraging Muon, a well-regarded open-source AI tool, to expedite the training process.
Microsoft plans to license the weights of MAI-Base-2 to “trusted partners” via its Azure cloud platform and intends to utilize the model in a new series of LLMs. More information regarding these upcoming models is expected to be shared in the near future.