On October 12, 2026 (US time), Microsoft announced Azure Maia 300, its in-house accelerator for AI inference. According to the company's announcement, production deployment will begin in 2027, and from that year Microsoft plans to expand the Maia series to multi-gigawatt scale. Over time, the company intends to make it a mainstay of the AI inference infrastructure behind its own products and services. As AI agents that understand long contexts and repeat multi-step reasoning and tool use become more common, the speed at which data reaches the compute units and the power available have become key factors in AI processing capability. Maia 300 is designed to run more AI workloads within a limited power budget by rethinking everything from the memory configuration to rack-to-rack communication.
Memory bandwidth improves far more than compute
The notable point in Maia 300's specifications is the large gap between the improvement in compute performance and in memory bandwidth. According to Microsoft's technical deep dive, compute performance rises 36% compared with the previous-generation Maia 200, while memory bandwidth grows by roughly 2.5x. Memory capacity also increases by 33%, but the design prioritizes faster data transfer over sheer capacity.
| Item | Maia 200 | Maia 300 |
|---|---|---|
| Memory | HBM3e | HBM4, 8 stacks |
| Memory capacity | 216GB | 288GB |
| Peak memory bandwidth | 7TB/s | 18TB/s |
| SoC thermal design power (TDP) | 750W | 950W |
The specifications above are based on Microsoft's Maia 200 announcement and its Maia 300 technical deep dive. Note that TDP is a benchmark for thermal design and should be distinguished from the power actually consumed during processing.
To understand the thinking behind this design, it helps to divide AI inference into two stages.
First, in the stage that processes the input text in bulk (prefill), a large amount of computation can be run in parallel. In the stage that generates the response one token at a time (decode), by contrast, the model data and context must be consulted every time the next token is generated. Memory bandwidth and the communication speed between accelerators can therefore become the constraint on performance. In a technical article explaining the design philosophy of Maia 200, Microsoft likewise emphasized the differing characteristics of these two stages.
Speeding up the compute units alone does not eliminate waiting time if the required data does not arrive fast enough. Maia 300's substantial boost in memory bandwidth appears intended to ease this bottleneck. However, the load profile changes with the length of the input text and the number of requests processed at once, so not every inference workload will speed up by the same proportion.
What matters to users is likewise not the theoretical performance of the compute units themselves. It is how many users can be served at the same time while keeping the wait before a response begins short and the generation speed steady.
To turn high memory bandwidth into real gains in processing capability, data must be placed appropriately for each type of workload so that compute units spend as little time as possible waiting for data.
Chiplets and NUMA to optimize data placement
Maia 300 adopts a chiplet architecture that splits functions such as compute, memory control and I/O across multiple small semiconductor dies. The compute chiplets use TSMC's N2P process. Rather than implementing every function on a single manufacturing process, this approach applies a design suited to each role.
The chip is also moving to a NUMA (Non-Uniform Memory Access) configuration, in which memory access conditions differ depending on location.
With NUMA, placing the data a compute unit uses in the nearest possible memory reduces the burden of transferring it over long distances within the package. The data locality that Microsoft stresses in its Maia 300 technical deep dive is closely tied to the chiplet architecture.
Supporting this data placement is a design philosophy called SDLA, carried over from Maia 200.
In SDLA, software explicitly manages where data is placed and how it is transferred, and runs computation and data transfer in parallel. The fast on-chip SRAM is also used as a working area managed directly by software. Rather than leaving it to automatic cache control, the mechanism places the necessary data near the compute units at the necessary time. Microsoft also explains this approach in detail in its Maia 200 technical article.
If the right data can be placed nearby, fast memory regions can be reused efficiently. On the other hand, if the placement does not suit the characteristics of the workload, unnecessary data transfers between compute units and memory can occur, and performance may fall.
In other words, getting the most out of the hardware also requires optimizing the compiler and the kernels that handle individual computations. Maia 300's specialized design presupposes sophisticated control by software.
To reduce the burden of data transfer, Maia 300 also adopts techniques that shrink the data itself. It supports low-precision formats such as an improved MXFP4, and also introduces formats that store data in fewer than 3 bits.
However, the number of bits used to store data and the precision actually used in computation are separate concepts. Support for storage formats below 3 bits does not mean that all calculations are run at that precision.
In AI that handles long contexts, the "KV cache," which holds intermediate data generated along the way, also consumes a large amount of memory.
In its explanation of its AI infrastructure strategy published in September 2026, Microsoft pointed to the importance of combining model-side optimization and data compression with proper management of the memory hierarchy.
If compression lets the same memory capacity hold longer contexts or raises the number of requests that can be processed at once, the benefit could exceed that of physically adding memory. Whether compression degrades response quality, however, has to be verified for each model.
Up to 12,288 accelerators connected, with prefill and decode separated
Maia 300 is designed to put 96 accelerators in a single rack and to connect up to 128 racks, for a total of 12,288 accelerators.
For communication, it uses a proprietary AI-oriented protocol built on Ethernet, with the same communication method used within and between racks. According to Microsoft's explanation of the connectivity architecture, data can be transferred directly between accelerators without passing through the host CPU or separate scale-out network equipment.
This connectivity is expected to be especially effective in configurations that separate input processing from response generation.
Prefill, which reads and processes long text, and decode, which generates the response sequentially, are assigned to different accelerators. When prefill finishes, the generated KV cache is transferred to the decode side, which takes over response generation. With Maia 300, this handoff of data can be handled over the shared communication fabric.
For example, the compute resources required differ between a situation where requests that read long documents are concentrated and one where many requests generate long answers to short questions.
If prefill and decode can be managed separately, accelerators can be allocated flexibly according to the load. However, separating the processing requires transferring the KV cache, so placement must be arranged so that the efficiency gained from separation outweighs the communication load.
Being able to use idle accelerators across racks is also important for raising the utilization of the facility as a whole.
If processing allocation is limited to a single rack, it is hard to meet processing demand arising in another rack even when accelerators sit unused in one. For Maia 300, Microsoft also describes workload placement across racks and a mechanism for using standby units in other racks when a failure occurs.
The aim is not only to deploy large-scale facilities but to keep them running with as little waste as possible.
That said, using the same communication method does not mean communication latency is the same between all accelerators. The data takes different paths for communication within a rack and communication across racks.
Workloads still need to be matched to their nature: latency-sensitive processing placed on nearby accelerators, and large-scale parallel processing spread across multiple racks.
Being able to connect up to 12,288 accelerators and being able to use all of them efficiently for a single workload need to be considered separately.
What "40% better power efficiency" and "up to 7x" mean
Microsoft projects that Maia 300 will process 40% more tokens per megawatt than leading GPU-based systems.
However, the October 12 announcement does not publish detailed evaluation results with unified conditions, such as the model number of the GPU used for comparison, the model used, or response speed. This figure therefore cannot be taken as a universal performance advantage over any particular GPU, or as an independent third-party measurement.
If, hypothetically, throughput per unit of power rose 40% under the same model and response conditions, the power needed to run the same workload would fall by about 28.6% in calculation.
This is converted from Microsoft's projected figure using (1 − 1 ÷ 1.4) × 100, and is not a measured power reduction rate. Note also that an improvement in power efficiency is not the same as a reduction in power consumption.
Meanwhile, the "up to 7x" performance gain Microsoft cites uses a different comparison target and a different unit of measurement.
It refers to token throughput for an entire rack of 96 accelerators, compared with Maia 200.
With Maia 300, the accelerator density per rack is itself double that of the previous generation. The "up to 7x" figure therefore cannot be interpreted as the performance improvement of a single chip. Even dividing by the number of installed units would not make it possible to accurately separate the effects of improvements in memory bandwidth, communication performance and workload placement.
A rack with 96 Maia 300 accelerators is allotted 167kW of power and is designed to remove more than 80% of the heat generated with a liquid cooling system. Both high-density packaging and cooling efficiency are addressed, including housing the host system in the same tray as the accelerators.
Maia 300 is a system designed not only to increase memory capacity and bandwidth, but also to run large numbers of accelerators efficiently within the constraints of power supply and cooling capacity.
If power efficiency improves, more AI processing can be run within the same power ceiling. The effect is especially large in data centers where power supply is constrained.
However, the operating cost of AI services also includes the cost of procuring accelerators and memory. Actual service pricing also depends on facility utilization and the provider's pricing strategy, so improved power efficiency does not translate directly into lower usage fees for Copilot or Azure.
Toward full deployment in 2027: software and operational efficiency are key
The previous-generation Maia 200 is already used in production environments such as Microsoft Copilot, running OpenAI models as well as Microsoft's own MAI models and open-source models.
According to Microsoft's October 12 announcement, rollout to GitHub Copilot and Microsoft Foundry is also planned by the end of 2026. Maia 300 is the next-generation product that incorporates the knowledge gained from such real-world operation, intended to substantially expand the scale of Maia deployment from 2027.
Work on the software environment has also continued from the previous generation.
When it announced Maia 200 in January 2026, Microsoft announced a preview of its SDK and introduced PyTorch integration, a Triton compiler and low-level programming options.
The aim is to make existing AI models easier to port while giving developers an environment in which they can finely optimize data transfer and computation for performance-critical parts.
However, a model running and a model running with high efficiency are separate matters.
To make the most of Maia 300's wide memory bandwidth and its NUMA configuration, which is aware of where data is placed, kernel optimization tailored to each model and appropriate workload allocation are required.
Whether response quality can be maintained when using lower precision and data compression, and whether the number of simultaneous requests can be increased without sacrificing response speed, are also important. Only when such conditions are met can the hardware's theoretical performance be turned into real reductions in operating costs.
Microsoft has also indicated that it will continue to combine multiple types of hardware and AI models, so the arrival of Maia 300 does not mean it will wholly replace existing GPUs.
The announcement also did not make clear what specifications or pricing general Azure customers will get for Maia 300. What is certain at this point is that Microsoft aims to overhaul, at large scale, the computing foundation that supports its own AI services.
After production deployment in 2027, the point to watch will be how far throughput per unit of power can be increased while maintaining the same model and response quality and meeting response-speed conditions.
If those gains hold across a variety of AI models and in real high-load environments, Microsoft will be able to deliver more AI services within a limited power supply.