How much GPU memory an LLM needs: a worked sizing guide

GPU memory decides which hardware can run a model. The weights, the KV cache for every active request and a margin for runtime buffers must all fit in GPU memory. Memory bandwidth then sets the upper limit on how fast the model generates text.

This article works one example from start to finish. The model is Qwen2.5-72B-Instruct, a 70B class open-weight model with a public model card and config file. The same method works for any model in the open-weight model comparison.

What has to fit in GPU memory

Three things share the memory of each GPU:

1. The model weights, which are fixed for a given model and precision.
2. The KV cache, which grows with the number of active requests and the length of each request.
3. Activations and runtime buffers, which the serving engine needs for each forward pass.

vLLM sets the fraction of GPU memory to be used for the model executor with the --gpu-memory-utilization flag. It loads the weights first and gives most of the remainder to the KV cache. This article assumes 0.9 of each GPU is available to vLLM. Measure the real headroom on your hardware before you commit to a purchase.

Weights by precision

Weight memory is the parameter count multiplied by the bytes per parameter. The model card for Qwen2.5-72B-Instruct lists 72.7B parameters, of which 70.0B are non-embedding.

The BF16 figure matches the published checkpoint. The safetensors files in the Qwen2.5-72B-Instruct repository add up to 145.4 GB. The INT4 figure is a floor. Quantisation stores weights in fewer bits, and the official AWQ checkpoint has safetensors files that add up to 41.6 GB. Its config.json sets 4 bits with a group_size of 128 and a zero point. Each group of 128 weights then carries a 16 bit scale and a 4 bit zero point: 70.0 billion / 128 x 2.5 bytes = 1.4 GB. If the 2.7 billion embedding parameters stay at 16 bits, they add 5.4 GB. The total is 35.0 + 1.4 + 5.4 = 41.8 GB, close to the size on disk. Always size from the files of the checkpoint you will run.

FP8 needs the right hardware. The vLLM documentation states that FP8 computation is supported on NVIDIA GPUs with compute capability >= 8.9 (Ada Lovelace, Hopper, Blackwell). The L40S uses the Ada Lovelace architecture, and the H100 and H200 use Hopper. The A100 uses the older Ampere architecture. Ampere GPUs run FP8 checkpoints as weight-only W8A16 with the FP8 Marlin kernels. In that mode the memory saving stays, but the arithmetic runs at 16 bits.

Lower precision changes the outputs. Score each quantised version on your own evaluation set before you choose. The method is in LLM evaluation and observability on private infrastructure.

KV cache

The KV cache stores the key and value vectors for every token in every active request. The model reads these vectors at each step and does not recompute them. NVIDIA gives the size as batch_size x sequence_length x 2 x num_layers x hidden_size x sizeof(FP16).

That formula assumes every attention head keeps its own keys and values. Qwen2.5-72B uses grouped-query attention, where several query heads share one key and value head. Its model card lists 64 heads for Q and 8 for KV. For a model like this, replace the hidden size with the number of KV heads multiplied by the head dimension:

KV bytes per token = 2 x num_layers x num_kv_heads x head_dim x bytes_per_element

The config.json for Qwen2.5-72B-Instruct lists num_hidden_layers 80, num_key_value_heads 8, hidden_size 8192 and num_attention_heads 64. The head dimension is 8192 / 64 = 128.

KV bytes per token (BF16) = 2 x 80 x 8 x 128 x 2 = 327,680 bytes, about 0.33 MB
One request at 32,768 tokens   = 327,680 x 32,768  = 10.7 GB
32 requests at 8,000 tokens    = 327,680 x 256,000 = 83.9 GB
Same 32 requests with FP8 KV   = 83.9 / 2           = 41.9 GB

The 32 by 8,000 load is an assumption for a retrieval workload. Each request carries retrieved chunks, a question and an answer. Replace it with your own numbers from the gateway logs or from a pilot.

The KV head count has a large effect. With 64 KV heads instead of 8, the same model needs 2 x 80 x 8192 x 2 = 2.6 MB per token. That is eight times more. Read the config file of every candidate model before you size.

vLLM can store the cache at lower precision with --kv-cache-dtype fp8. The default is auto, which will use model data type. vLLM also manages the cache in pages. The PagedAttention paper describes how KV cache memory can be significantly wasted by fragmentation and redundant duplication, which limits the batch size.

The worked example on real hardware

The table applies the 0.9 assumption to each GPU set and subtracts the FP8 weights of 72.7 GB. The last column divides the space left by the FP8 KV cost of one 8,000 token request. That cost is 163,840 bytes x 8,000 = 1.31 GB.

The request counts are upper limits. Activations and buffers take part of the remaining space, and requests longer than 8,000 tokens take more. The BF16 weights of 145.4 GB need at least 2 x H200 or 4 x H100. The INT4 checkpoint of 41.6 GB fits on one H100 SXM and leaves 30.4 GB. That space holds about 23 requests at 8,000 tokens with an FP8 cache.

Throughput and concurrency

Generation has two phases. In prefill, the model processes the input tokens to compute the intermediate states. In decode, the model generates output tokens one at a time, and NVIDIA describes decode as a memory-bound operation. Long retrieved contexts make prefill heavy, and prefill sets the time to the first token.

Memory bandwidth gives a ceiling for decode. Each decode step reads all the weights once. It also reads the KV cache of every request in the batch. The H100 SXM has 3.35 TB/s of memory bandwidth, so two of them in tensor parallel give 6.7 TB/s in total.

One user, FP8 weights, 2 x H100:   6.7 TB/s / 72.7 GB          = about 92 tokens/s ceiling
32 users at 8,000 tokens, FP8 KV:  bytes per step = 72.7 + 41.9 = 114.6 GB
                                   6.7 TB/s / 114.6 GB          = about 58 steps/s
                                   58 x 32 users                = about 1,870 tokens/s ceiling

These are ceilings that ignore compute limits, communication between GPUs and kernel efficiency. Real figures are lower. The arithmetic still shows the trade-off. Batching shares one read of the weights across many users, so total throughput rises with concurrency. Each user then gets fewer tokens per second, and the KV reads grow with context length.

Measure before you buy. vllm bench serve reports percentiles for ttft, tpot and itl, which are time to first token, time per output token and inter-token latency. The --max-concurrency flag caps the number of requests in flight. Run it on a rented instance of the GPU you plan to buy, with prompts taken from your own workload.

Tensor parallelism

Tensor parallelism splits each layer of the model across several GPUs. NVIDIA describes it as sharding (horizontally) individual layers of the model. Each GPU holds a share of the weights and a share of the KV heads. With 8 KV heads, a tensor parallel size of 2, 4 or 8 splits the cache evenly.

The GPUs exchange data at every layer, and the speed of the link between them limits throughput. vLLM advises tensor parallelism if the model is too large for a single GPU but fits on a single node with several GPUs. For GPUs without an NVLink interconnect, such as the L40S, vLLM suggests pipeline parallelism, which splits the model by layers. The H100 SXM has NVLink at 900 GB/s and PCIe Gen5 at 128 GB/s. The L40S lists NVLink support as No.

GPU tiers

All figures come from the NVIDIA product pages as of October 2026.

The H100 NVL and H200 NVL are PCIe dual-slot air-cooled cards, and the L40S and RTX PRO 6000 are also dual slot PCIe cards. The SXM parts ship in HGX systems with 4 or 8 GPUs, with higher power and cooling needs. Power and rack planning are in buying, leasing or renting GPUs in Australia.

Check the unit on every spec sheet. NVIDIA lists the L40S at 48 GB. AWS lists the same GPU in a g6e instance at 44 GiB, and 48 billion bytes is 44.7 GiB.

Sizing checklist

1. Pick the candidate models and read the parameter count from each model card.
2. Read num_hidden_layers, num_key_value_heads and the head dimension from each config.json.
3. Choose a precision and take the weight size from the files of the actual checkpoint.
4. Measure the request lengths and the peak number of requests in flight from a pilot or from the gateway logs.
5. Calculate the KV cache at peak load, at BF16 and at FP8.
6. Add the weights and the KV cache, then compare the total with 0.9 of the GPU memory.
7. Make sure that the GPUs support FP8 if you plan to run FP8.
8. Use NVLink or SXM parts when you need tensor parallelism across more than two GPUs.
9. Run vllm bench serve on a rented instance of the target GPU with your own prompts.
10. Add one spare node, or accept the outage when a GPU fails.

What to decide next

Decide the model and the precision first, because they fix the weight size. Then measure the real concurrency and context length, because they decide whether you need one GPU or eight. The deployment architecture decides where those GPUs sit, and the cost comparison with an API decides whether to own them at all.

Our on-prem AI deployment service sizes the hardware against your measured load before you buy. The LLM integration and RAG service builds the retrieval workload that drives the context length.