Maia 300 Buys 36% More Compute and 2.5x the Memory Bandwidth. The Ratio Is the Argument

Microsoft's third custom AI accelerator, announced on 12 October at the OCP Global Summit, gets 36% more compute than the chip it replaces and 2.5 times the memory bandwidth. Those two numbers sit in Microsoft's own materials, one after the other, and the gap between them is the design. Maia 300 is not an attempt to out-multiply Nvidia. It is a bet that the thing standing between a datacentre and more tokens per second is not arithmetic but the speed at which weights and KV cache can be dragged in front of the arithmetic.

That bet is not new — it is why high-bandwidth memory has been the scarce commodity of the last three years — but Maia 300 is the first Maia built around it from the floor plan up.

What changed in the silicon

Maia 300 is Microsoft's first multi-chiplet Maia. The compute dies move to TSMC's N2P process, with memory-controller and I/O functions split onto separate dies that Microsoft says are optimised for bandwidth and cost rather than density. Memory addressing shifts from uniform to NUMA across the chiplets, so data can sit next to the compute that wants it — a structural admission that on a part this wide, where a tensor lives matters as much as how fast it can be multiplied.

Specifications beyond Microsoft's own blog posts come from StorageReview's reading of the launch material; Microsoft has not published a full datasheet and says more will come at Ignite in November.

The rack is where the ambition shows. Ninety-six accelerators go into 24 four-accelerator trays at 167 kW, with host processor, memory, storage and cooling folded into each tray and the separate host trays deleted. Liquid cooling carries more than 80% of the thermal load. In-rack networking is half-switched: accelerator pairs in a tray are wired directly to each other, and a switched Ethernet fabric reaches all 96. A second, optical tier joins up to 128 racks into a superpod of up to 12,288 accelerators, and an accelerator can start a transfer without involving the host or a discrete NIC. Microsoft's framing — "one communication model to software" from kernel to datacentre — is the part a competitor cannot buy off a shelf.

The claims, and what they rest on

Microsoft says Maia 300 delivers 40% more tokens per megawatt than "leading GPU-based systems," up to seven times the rack-scale token throughput of Maia 200, and more than 2.5 times the performance per watt. All three are Microsoft's own figures. The 40% number names no GPU, no model and no serving configuration, which makes it unfalsifiable as published; the rack-scale figure is partly an artefact of doubling the accelerator count. Nothing here has been independently benchmarked.

The honest comparison is the one Reuters drew: 96 Maia 300 accelerators in 167 kW against 72 GPUs in 142 kW for an Nvidia GB300 rack. Microsoft is buying density and paying for it in power per rack, which only works if you also own the datacentre, the busbar and the power-allocation algorithms — which Microsoft does.

Why Microsoft is doing this

Rani Borkar, who runs Azure hardware systems and infrastructure, told Reuters the goal is "delivering more intelligence for every watt, every dollar, and every byte of memory." The dollar is the operative word. Microsoft expects to spend more than $175 billion on capital expenditure this calendar year, a meaningful share of it with Nvidia. Maia 300 is described as becoming "the primary AI inference system for Microsoft products and services" — Copilot, Microsoft 365, GitHub Copilot, Foundry. Microsoft told Reuters it could deploy more than a million Maia 300 chips next year, with multi-gigawatt deployments beginning in 2027.

Note what is not claimed. Nvidia and AMD stay in the training fleet. Maia is not sold as an Azure instance type, so no customer will rent one and run their own benchmark on it. This is Microsoft taking its own inference bill off the GPU market, not competing for anyone else's.

That is the narrower but harder strategy. A merchant accelerator has to win on a datasheet against Nvidia's software ecosystem. A captive one only has to beat the GPU on Microsoft's own workloads, at Microsoft's own power prices, with Microsoft's own compiler team pointed at it — and the first-party workload is the one case where a vertically integrated design can be tuned to the model and the model to the design. Whether the 40% holds is unknowable from here. Whether Microsoft can build a million of them in a year, with HBM4 supply as tight as it is, is the question Ignite should answer.

Sources: Microsoft: Microsoft to expand Azure Maia system to multi-gigawatt deployments · Microsoft Azure: Deep dive into Azure Maia 300 architecture · Microsoft: Maia 300 newsroom · Reuters: Microsoft plans huge deployment of its own AI chips, with an eye on saving costs · StorageReview: Microsoft Maia 300 puts 288GB of HBM4 at 18TB/s on an N2P chiplet · HPCwire: Microsoft announces Maia 300, its next-gen custom AI chip