GPT-4 is more than 10x the size of GPT-3. We believe it has a total of ~1.8 trillion parameters across 120 layers versus the ~175 billion parameters of GPT-3.

OpenAI was able to keep costs reasonable by utilizing a mixture of experts (MoE) model. If you are unfamiliar with MoE, read our post about the broad GPT-4 architecture and training cost from 6 months ago.

Furthermore, OpenAI utilizes 16 experts within their model, each is about ~111B parameters for MLP. 2 of these experts are routed to per forward pass.

While the literature talks a lot about advanced routing algorithms for choosing which experts to route each token to, OpenAI’s is allegedly quite simple, for the current GPT-4 model.

Furthermore, there are roughly ~55B shared parameters for attention.

Each forward pass inference (generation of 1 token) only utilizes ~280B parameters and ~560 GFLOPs. This contrasts with the ~1.8 trillion parameters and ~3,700 GFLOP that would be required per forward pass of a purely dense model.

Dataset Composition

OpenAI trained GPT-4 on ~13 trillion tokens. This makes sense, given CommonCrawl for RefinedWeb contains ~5 trillion tokens high-quality tokens. For reference, Deepmind’s Chinchilla and Google’s PaLM model were trained with ~1.4 trillion tokens and ~0.78 trillion tokens, respectively. Even PaLM 2 is allegedly trained on ~5 trillion tokens.

This dataset is not 13 trillion unique tokens. Instead, the dataset contains multiple epochs due to a lack of high-quality tokens. There were 2 epochs for text-based data and 4 for code-based data. Interestingly, this is far short of Chinchilla optimal, indicating the need to train the model on double the token count. This indicates there is a lack of easy-to-source tokens on the web. There are 1,000x more high-quality text tokens out there and even more audio and visual, but sourcing them isn’t as simple as a web scrape.

There is millions of rows of instruction fine-tuning data from ScaleAI as well as internally. Unfortunately, we could not find much out on their RLHF data.

There was an 8k context length (seqlen) for the pre-training phase. The 32k seqlen version of GPT-4 is based on fine-tuning of the 8k after the pre-training.

The batch size was gradually ramped up over a number of days on the cluster, but by the end, OpenAI was using a batch size of 60 million! This, of course, is “only” a batch size of 7.5 million tokens per expert due to not every expert seeing all tokens.

Parallelism Strategies

The strategy for parallelizing across all their A100 GPUs is critically important. They utilized 8-way tensor parallelism as that is the limit for NVLink. Beyond that, we hear they are using 15-way pipeline parallelism. Theoretically, this is too many pipelines when considering data-comm vs compute time, but if they are memory capacity bound, then it makes sense.

It is ~30GB just for the parameters at FP16 per GPU when purely pipeline + tensor parallel. Once you add on KV cache and overhead, this theoretically makes sense if a large portion of OpenAI’s GPUs are 40GB A100s. They likely used ZeRo Stage 1. It is possible they used block-level FSDP, or hybrid shared data parallel.

As for why they didn’t use full model FSDP, it could be because of the higher communication overhead. While OpenAI has high-speed networking between most nodes, it may not be between all their nodes. We believe at least some of the clusters are connected at much lower bandwidth than the other clusters.

We don’t understand how they avoid having huge bubbles for every batch with such high pipeline parallelism. It’s likely they just ate the cost.

Training Cost

OpenAI’s training FLOPS for GPT-4 is ~2.15e25, on ~25,000 A100s for 90 to 100 days at about 32% to 36% MFU. Part of this extremely low utilization is due to an absurd number of failures requiring checkpoints that needed to be restarted from. The above mentioned bubbles are extremely costly.

Another reason is that all-reduce between that many GPUs is extremely costly. The is especially true if, as we suspect, the cluster is really a bunch of smaller clusters with much weaker networking between them IE 800G/1.6T non-blocking between segments of the cluster, but those segments are only connected at 200G/400G.

If their cost in the cloud was about $1 per A100 hour, the training costs for this run alone would be about $63 million. This ignores all the experimentation, failed training runs, and other costs such as data gathering, RLHF, staff, etc. The true cost is much higher due to these factors. Furthermore, it implies you have someone to buy the chips/networking/datacenter, absorb the Capex, and rent them to you.

Today, the pre-training could be done with ~8,192 H100 in ~55 days for $21.5 million at $2 per H100 hour. Note that we believe there are 9 firms that will have more H100s by the end of this year. Not all of these firms will dedicate all of them to a single training run, but those that do will have a much larger model. Meta will have over 100,000 H100’s by the end of the year, but a significant number will be distributed across their datacenters for inference. Their largest individual cluster will still be well over 25k H100s.

Many firms will have the compute resources to train a GPT-4 size model by the end of this year.

Mixture of Expert Tradeoffs

MoE is a great way to reduce parameter count during inference while still pumping up parameter count, which is required to encode more information per training token. This is necessary as acquiring enough high-quality tokens is extremely difficult. OpenAI would have had to train on 2x the tokens if they were actually trying to go Chinchilla optimal.

With that said, there are multiple tradeoffs that OpenAI has made. For example, MoE is incredibly difficult to deal with on inference because not every part of the model is utilized on every token generation. This means parts may sit dormant when other parts are being used. When serving users, this really hurts utilization rates.

Researchers have shown that using 64 to 128 experts achieves better loss than 16 experts, but that’s purely research. There are multiple reasons to go with fewer experts. One reason for OpenAI choosing 16 experts is because more experts are difficult to generalize at many tasks. More experts can also be more difficult to achieve convergence with. With such a large training run, OpenAI instead chose to be more conservative on the number of experts.

Furthermore, running with fewer experts also helps with their inference infrastructure. There are a variety of difficult tradeoffs when moving to a mixture of experts’ inference architecture. Let’s start with the basic tradeoffs with inference for LLMs before moving to what OpenAI faces and the choices they made.

Inference Tradeoffs

Before starting, as an aside, we want to point out that every LLM company we have spoken with thinks Nvidia’s FasterTransformer inference library is quite bad, and that TensorRT is even worse. The lack of ability to take Nvidia’s template and modify it means that people create their own solutions from scratch. For those of you at Nvidia reading this, you need to get on this ASAP for LLM inference, or else the defacto will become an open tool, which can add 3^rd party hardware support much more easily. A wave of huge models is coming. If there is no software advantage in inference, and handwritten kernels are required anyways, then there is a much larger market for AMD’s MI300 and other hardware.

There are 3 main tradeoffs for inference of large language models that occur along the batch size (number of concurrent users served) dimension and the number of chips used.

1. Latency – The model must respond in reasonable latency. Humans don’t want to wait many seconds before waiting for their output to start streaming in a chat application. Prefill (input tokens) and decode (output tokens) take varying amounts of time to process.

2. Throughput – The model must output a certain number tokens outputted per second. Somewhere around 30 tokens per second is what is needed for humans use. Lower and higher throughput are okay for various other usecases.

3. Utilization – The hardware running the model must achieve high utilization, or else it will be too costly. While higher latency and lower throughput can be used to group more user requests together and achieve much higher utilization rates, they make it more difficult.

LLM inference is all about balancing 2 main points, memory bandwidth and compute. In the most oversimplified terms, each parameter must be read, and it has 2 FLOPs associated to it. As such, the ratio of most chips, (H100 SXM only has 3TB/s of memory bandwidth, but 2,000 TFLOP/s of FP8), is completely imbalanced for inference at batch size 1. If only 1 user is being served, batch size 1, then the memory bandwidth required to stream in every parameter for every token generation dominates inference time. Compute time is near nothing.

To efficiently scale a large language model out to many users, batch size must exceed 1. Multiple users amortize the parameter read cost. For example at batch size of 256 or 512, there are 512 FLOP/s or 1024 FLOP/s for each byte of memory that read in. This ratio more closely matches the H100’s memory bandwidth versus FLOPS. This helps achieve much higher utilization, but that comes the drawback of higher latency.

Many point to memory capacity as a major bottleneck for LLM inference due to the size of the model that can fit on a number of chips, but that is incorrect. While large models require multiple chips to inference and higher memory capacity leads to them fitting on fewer chips, it is actually better to use more chips than is required capacity-wise so latency can be brought back down, throughput can be increased, and larger batch sizes can be used for increasingly higher utilization rates.

Google showed off these tradeoffs in their PaLM inference paper. However, it’s noteworthy that this was for a dense model like PaLM, not a sparse model like GPT-4.

If an application requires the lowest possible latency, we need to apply more chips and partition the model in as many ways as we profitably can. Lower latency can often be achieved with smaller batch sizes, but smaller batch sizes also result in worse MFU [utilization], resulting in a higher total cost (in terms of chip-seconds or dollars) per token.

If an applications requires offline inference and latency is not a concern, the primary goal is to maximize per-chip throughput (i.e., minimize total cost per token). It is most efficient to increase the batch size because larger batches typically result in better MFU [utilization], but certain partitioning strategies that are not efficient for small batch sizes become efficient as the batch size grows larger.

More chips and higher batch sizes are the cheapest because the increase utilization, but then that also introduces a 3^rd variable, networking time. Some methods of splitting a model across chips are more efficient for latency, but tradeoff with utilization.

Both the weight loading part of the memory time and the non-attention compute time are proportional to the model size and inversely proportional to the number of chips. However, for a given partitioning layout, the time needed for chip-to-chip communication decreases less quickly (or not at all) with the number of chips used, so it becomes an increasingly important bottleneck as the chip count grows.

While we will only briefly discuss it today, it should be noted that the memory requirements for KV cache explode in capacity as batch size and seqlen grow.

If an application requires generating text with long attention contexts, it substantially increases the inference time. For a 500B+ model with multihead attention, the attention KV cache grows large: for batch size 512 and context length 2048, the KV cache totals 3TB, which is 3 times the size of the model’s parameters. The on-chip memory needs to load this KV cache from off-chip memory once for every token generated during which the computational core of the chip is essentially idle.

Longer sequence lengths are particularly nasty on memory bandwidth and memory capacity. OpenAI’s 16k seqlen GPT 3.5 turbo and 32k seqlen GPT 4 are so much more expensive because they cannot utilize larger batch sizes due to memory constraints. Lower batch sizes leads to lower hardware utilization. Furthermore, with larger sequence length, the KV cache balloons. KV cache cannot be shared between users, so that requires individual memory reads, further bottlenecking memory bandwidth. More on MQA in a bit.

GPT-4 Inference Tradeoffs And Infrastructure

All of the above is difficult with GPT-4 inference, but the model architecture being a Mixture of Experts (MoE) introduces a whole new set of difficulties. Each token generation forward pass can be routed to a different set of experts. This throws a wrench into the tradeoffs that are achieved along the axis of throughput, latency, and utilization at higher batch sizes.

OpenAI’s GPT-4 has 16 experts, with 2 routed to per forward pass. This means that if there is a batch size of 8, the parameter read for each expert could be at only batch size 1. Worse yet, it could mean 1 expert could be at a batch size of 8 and others could be at 4 or 1 or 0. Every single token generation, the routing algorithm will send the forward pass in a different direction, leading to significant variation in token-to-token latency as well as expert batch size.

Inference infrastructure is of the primary reasons OpenAI went with a much smaller number of experts. If they went with an even larger number of experts, memory bandwidth would bottleneck inference even more. OpenAI regularly hits a batch size of 4k+ on their inference clusters, which means even with optimal load balancing between experts, the experts only have batch sizes of ~500. This requires very large amounts of usage to achieve.

Our understanding is that OpenAI runs inference on a cluster of 128 GPUs. They have multiple of these clusters in multiple datacenters and geographies. The inference is done at 8-way tensor parallelism and 16-way pipeline parallelism. Each node of 8 GPUs has only ~130B parameters, or less than 30GB per GPU at FP16 and less than 15GB at FP8/int8. This enables inference to be run on 40GB A100’s as long as the KV cache size across all batches doesn’t balloon too large.

Individual layers containing various experts are not broken up across different nodes because that would make network traffic too irregular and recalculating KV cache between each token generation would be far too costly. The biggest difficulty for any future MoE model scaling and conditional routing is how to deal with routing around the KV cache.

The layer count is 120, so it is simple to dive that among 15 different nodes, but because the first node needs to do dataloading and embedding, it would make sense to put fewer layers on the head node of the inference cluster. Furthermore, there are some murmurs of speculative decoding, which we will discuss later, but we aren’t sure if we believe them. That would also explain why the head node needs to contain so many fewer layers.

GPT-4 Inference Cost

GPT-4 costs 3x that of the 175B parameter Davinchi model despite being only 1.6x the feed-forward parameters. This is largely due to the larger clusters required for GPT-4 and much lower utilization achieved.

We believe that it costs $0.0049 cents per 1k tokens for 128 A100s to inference GPT-4 8k seqlen and $0.0021 cents per 1k tokens for 128 H100’s to inference GPT-4 8k seqlen. It should be noted, we assume decent high utilization, and keeping batch sizes high.

This may be an erroneous assumption as it’s pretty clear OpenAI sometimes has very poor utilization at times. We assume OpenAI takes clusters down during the trough hours and repurposes those nodes to resume training from checkpoints for smaller test models trying a variety of new techniques. This helps keep inference costs low. If OfpenAI does not do this, their utilization rates would be even lower, and our cost estimates would more than double.

Multi-Query Attention

MQA is something everyone else is doing, but we wanted to note OpenAI is as well. Long story short, only 1 head is needed and memory capacity can be significantly reduced for the KV cache. Even then, the 32k seqlen GPT-4 definitely cannot run on 40GB A100s, and the 8k is capped on max batch size. Without it, the 8k would be significantly capped on max batch size to the point where it’s uneconomical.

Continuous batching

OpenAI implements both variable batch sizes and continuous batching. This is so as to allow some level of maximum latency as well optimizing the inference costs. This page by AnyScale is worth a read if you are unfamiliar with the concept.

Speculative Decoding

We have heard from some reliable folks that OpenAI uses speculative decoding on GPT-4 inference. We aren’t sure if we believe it to be clear. The general variation in token to token latency and difference when doing simple retrieval tasks vs more complex tasks seems to indicate it’s possible, but there are way too many variables to know. Just in case, we will explain it here by using some of the text in “Accelerating LLM Inference with Staged Speculative Decoding” and modifying a bit/adding some color.

Using an LLMs is generally split into two phases,. First, prefill , the prompt is run through the model to generate the KV cache and the first output logits (probability distribution of possible token outputs). This is usually fast, as the entire prompt can be handled in parallel.

The second phase is decoding. A token is selected from the outputted logits and fed back into the model, which produces logits for the following token. This is repeated until the desired number of tokens is produced. Because decoding must be done sequentially weights streamed through the compute units each time in order to generate a single token, the arithmetic intensity (that is, FLOP of compute / byte of memory bandwidth) of this second phase is extremely low when run in small batches. As such, decoding is usually the most expensive part of autoregressive generation.

This is why input tokens are much cheaper than output tokens in OpenAI’s API calls.

The basic idea of speculative decoding is to use a smaller, faster draft model to decode several tokens in advance, and then feeds them into the oracle model as a single batch. If the draft model was right about its predictions – the larger model agrees – one can decode several tokens with a single batch, which saves considerable memory bandwidth, and thus time, per token.

However, if the larger model rejects the tokens predicted by the draft model, then the rest of the batch is discarded and the algorithm naturally reverts to standard token-by-token decoding. Speculative decoding may also be accompanied by a rejection sampling scheme to sample from the original distribution. Note this is only useful in small-batch settings where bandwidth is the bottleneck.

Speculative decoding trades compute for bandwidth. There are two key reasons why speculative decoding is an attractive performance engineering target. First, it does not degrade model quality at all. Second, the gains it provides are generally orthogonal to other methods, because its performance comes from converting sequential execution to parallel execution.

Current speculative methods predict a single sequence for the batch. However, this doesn’t scale well to large batch sizes or low draft model alignments. Intuitively, the probability that two models agree for long consecutive sequences of tokens is exponentially low, which means that speculative decoding has rapidly diminishing returns as one scales its arithmetic intensity.

We believe that if OpenAI is using speculative decoding, they are likely only using it for sequences of ~4 tokens. As an aside the whole conspiracy of lowering quality of GPT-4 could just be because they are letting the oracle model accept lower probability sequences from the speculative decoding model. Another aside is that some folks speculate that bard uses speculative decoding because Google waits for the entire sequence to be produced before sending it to users, but we don’t believe this speculation is true.

Vision Multi-Modal

The vision multimodal capabilities are the least impressive part of GPT-4, at least compared to leading research. Of course, no one has commercialized any of the research into multi-modal LLMs yet.

It is a separate vision encoder from the text encoder, but there is cross-attention. We hear the architecture is similar to Flamingo. This adds more parameters on top of the 1.8T of GPT-4. It is fine-tuned with another ~2 trillion tokens, after the text only pre-training.

On the vision model, OpenAI wanted to train it from scratch, but it wasn’t mature enough, so they wanted to derisk it by starting with text.

The next model, GPT-5, they train will supposedly be from scratch for vision and be able to generate images on its own too. Furthermore, it will be able to do audio as well.

One of the primary purposes of this vision capability is for autonomous agents able to read web pages and transcribe what’s in images and video. Some of the data they train on is joint data (rendered LaTeX/text), screen shots of web page, youtube videos: sampling frames, and run Whisper around it to get transcript.

One interesting thing about all this over-optimization for LLMs is that the IO cost of a vision model differs from that of text. On a text model, it is extremely cheap as we described in the Amazon Cloud Crisis piece. On vision, the data-loading is ~150x higher IO. 600 bytes per token rather than 4 like with text. There is a lot of work being done on image compression.

This is extremely relevant for hardware vendors who are optimizing hardware 2-3 years out from now around the usecase and ratios of LLMs. They may find themselves in a world where every model has robust vision and audio capabilities. They may find their architecture poorly adapted. In general, the architecture will definitely evolve past the current simplified text-based dense and/or MoE models we see today.