The GB200, codenamed Oberon and Bianca solves many of the issues from the GH200.

First off, it halves the number of CPUs that are required, so that greatly improves TCO. Currently, GH200 is just far too expensive, and most large scale AI workloads do not require the big Nvidia tax for Grace. After all, Grace in GH200 for most models simply acts as the most expensive memory controller in the world. Basically, every major firm involved with AI has ran the TCO calculation and chose to purchase more GPUs, and not buy the CPUs from Nvidia.

This potentially changes with GB200. Nvidia is halving the number of CPUs to GPUs, which is critical for TCO. On the flipside, Nvidia is trying to sell integrated racks with liquid cooling. They do not want to sell the GPU compute trays and NVSwitch trays separately. This is something hyperscalers are probably not happy with, but necessary due to the level of integration required with the switch backplane, power busbar, and cooling. The CPUs, GPUs, NVSwitches, and ConnectX7 NICs are all watercooled.

There are two versions of the rack here. One is 120kW, which is an incredible amount of power for 72 GPUs. It has 10 compute trays on top, 9 NVSwitch trays in the middle, and 8 compute trays on the bottom. Each compute tray contains 4 GPUs, 4 NICs, 2 CPUs, 768GB of HBM, and 1,024GB of LPDDR5X. This version is actually later though.

Nvidia has also opted for a lower power rack that has 36 GPUs and 9 compute trays, which can have NVLink scale up connect across two racks. This is likely much easier for most firms to deploy. Nvidia has told the primary ODM for these racks to prepare capacity for 50,000+ racks next year. A gargantuan volume of CPU+GPU hybrid versus the miniscule <1% of GPUs shipped with GH200.

COGS & Pricing

Despite the huge improvements in TCO the B100 offers to end users, the cost to manufacture more than doubles. Contact us for our full BOM model which breaks out COGS by GPU SKU Nvidia will offer. This includes wafer pricing, Bose Einstein yield model, recoverable/repairable area, die cost, CoWoS-L packaging, ABF substrate, assembly, test, HBM, SLT, and much more.

Our understanding is that the ASP for the B100 baseboard will come in at only ~$240,000 per GPU baseboard for high volume hyperscaler level pricing. This is a surprise for many as this is much less than the expected 50% increase in ASP gen on gen, which was the expectation.

With the B100 still offering tremendous increase in performance and TCO, this raises the question why? The simple answer is competition is emerging. The AMD MI300 has emerged as a credible competitor for inference use cases although AMD had to pull out all stops in terms of taking big technology risks and pricing aggressively. AMD secured major orders from Nvidia’s two largest customers, Meta and Microsoft.

On the custom front all of Nvidia’s major customers are designing their own chips. Only Google have been successful to date, but Amazon continues to ramp Inferentia and Trainium, even though the current generation is not great, Meta is betting big on MTIA long term, and Microsoft is starting their silicon journey as well. As the hyperscalers dramatically increase their capital expenditure to defend their moats in the Gen AI world, the more hyperscaler dollars are ending up as Nvidia’s gross profits. There is an extreme sense of urgency to find alternatives.

Our accelerator model shares units, volumes, and ASP for all 8 SKUs of the Blackwell family. The initial B100 is of course the lowest of the bunch.

Margin Impact – Nvidia’s Benevolence

With production cost doubling, but ASP only increasing a fraction of that, it’s clear that Nvidia’s margins on the B100 won’t be as good as the H100.

Therefore we believe Nvidia’s margins have peaked. We expect B100 and future families to have slightly lower margins, but furthermore, over the next few quarters H100 margins will also come down due to the H200 and H20.

The H200 will be the same ASP as H100 but with significantly more HBM, adding to the BOM. The H20 is even worse from a financial perspective: Nvidia is offering it for significantly lower prices as it is shipping with far fewer FLOPS, even though it is the same GPU silicon as the H100. Furthermore, the HBM capacity is increased from 80GB to 96GB so the overall BOM cost actually increases here.

Nvidia decided it would be difficult to justify for Chinese customers to pay more for significantly fewer FLOPs, as they also have to compete with massive reimportation schemes of H100/H200 amounting to hundreds of thousands of GPUs. Nvidia would prefer China buys the China specific GPUs as they have no way of tracking or preventing the reimportation schemes. If they can sell more H20 and reduce the reimportation volumes, they probably draw less ire from the US Government.

While these margin decreases sounds dramatic, those who care about the finances can breathe a sigh of relief as the financial impact is far from disastrous, gross margins are only going down a few points from here, due to the product markup and gross margin being already so high in the first place.

Our take is simple, Nvidia cares more about gross profit and market share than gross margin. Obsessively worrying about a few percentage points is what bean counters do, not visionaries like Jensen Huang, who want to rule the world as benevolent compute dictators.

Some of this margin deterioration is blunted though due to the mix of datacenter revenue continuing to soar. Furthermore, Nvidia is diversifying their supply chain from 800G transceivers to power delivery components, which helps blunt the margin decrease as well.

Despite this, management guided to lower margins for the fiscal year compared to the previous 2 quarters.

GAAP and non-GAAP gross margins are expected to be 76.3% and 77%, respectively, plus or minus 50 basis points. Similar to Q4, Q1 gross margins are benefiting from favourable component costs. Beyond Q1, for the remainder of the year, we expect gross margins to return to the mid-70s percent range.…

We highlighted in our opening remarks, really about our Q4 results and our outlook for Q1. Both of those quarters are unique. Those 2 quarters are unique in their gross margin as they include some benefit from favourable component cost in the supply chain kind of across both our compute and networking, and also in several different stages of our manufacturing process. So looking forward, we have visibility into a mid-70s gross margin for the rest of the fiscal year, taking us back to where we were before this Q4 and Q1 peak that we’ve had here. So we’re really looking at just a balance of our mix. Mix is always going to be our largest driver of what we will be shipping for the rest of the year and those are really just the drivers.

Colette Kress (Q4 FY2024 earnings call)

However, on the other hand there could be larger qualitative impact on the narrative around Nvidia. Nvidia is perceived as the dominant player in accelerated AI compute and rightly so. One of the bear cases for Nvidia is competition: that other players will chip away at their software moat and then Nvidia will no longer be able to enjoy their current levels of profitability, which have never been seen before from physical hardware.

Nvidia starting to take a margin hit is the right defensive move as it tries to take away the oxygen from some initially promising attempts from AMD and hyperscaler in-house silicon trying to break into the market. However, some people may perceive this as worrying that Nvidia feels the need for defense as the moat is no longer impenetrable.