Free tool
How much GPU memory does your model need?
Pick an open model, a precision and a context length: the calculator adds up the weights, the KV cache or the training state, and the runtime overhead — then shows the cheapest GPU configuration that holds it, from a single L4 to an 8-GPU B300 node.
- Weights
- 71 GB
- KV cache
- 2.7 GB
- Optimizer & adapters
- 0.0 GB
- Runtime & buffers
- 4.3 GB
Suggested configurations
- 1× RTX PRO 600096 GB total $749/mo
- 2× L40S96 GB total · PCIe $858/mo
- 4× L496 GB total · PCIe $676/mo
- 2× A100160 GB total · NVLink $1,298/mo
- 4× RTX 409096 GB total · PCIe $756/mo
Ranked by monthly price, with a penalty for every extra card a model is split across (30% on PCIe, 10% on NVLink). Estimate only — engines, schedulers and quantisation formats differ; 5% of each GPU is kept free, plus 1.5 GB per GPU for the CUDA context. How it is calculated.
Reference table
VRAM needed by popular open models
Weights at each precision, the KV cache per 1,000 tokens at 16-bit, and the configuration we would suggest here for FP8 inference with an 8k context and one sequence.
| Model | Parameters | Weights 16-bit | Weights FP8 | Weights 4-bit | KV per 1k tokens | FP8, 8k context | Suggested here |
|---|---|---|---|---|---|---|---|
| Llama 3.1 8B | 8B | 16 GB | 8.0 GB | 4.4 GB | 0.13 GB | 11 GB | 1× L4 · $169/mo |
| Llama 3.3 70B | 70.6B | 141 GB | 71 GB | 39 GB | 0.34 GB | 78 GB | 1× RTX PRO 6000 · $749/mo |
| Llama 3.1 405B | 405.9B | 812 GB | 406 GB | 223 GB | 0.53 GB | 428 GB | 2× B300 · $7,580/mo |
| Qwen3 8B | 8.2B | 16 GB | 8.2 GB | 4.5 GB | 0.15 GB | 11 GB | 1× L4 · $169/mo |
| Qwen3 14B | 14.8B | 30 GB | 15 GB | 8.1 GB | 0.17 GB | 18 GB | 1× L4 · $169/mo |
| Qwen3 32B | 32.8B | 66 GB | 33 GB | 18 GB | 0.27 GB | 38 GB | 1× L40S · $429/mo |
| Qwen3 30B-A3B (MoE)3.3B active per token | 30.5B | 61 GB | 31 GB | 17 GB | 0.10 GB | 34 GB | 1× L40S · $429/mo |
| Qwen3 235B-A22B (MoE)22B active per token | 235.1B | 470 GB | 235 GB | 129 GB | 0.20 GB | 248 GB | 4× A100 · $2,596/mo |
| DeepSeek-V3 / R1, 671B (MoE)37B active per token | 671B | 1,342 GB | 671 GB | 369 GB | 0.07 GB | 700 GB | 8× H200 · $10,990/mo |
| Kimi K2, 1T (MoE)32B active per token | 1,026B | 2,052 GB | 1,026 GB | 564 GB | 0.07 GB | 1,069 GB | 8× B200 · $18,900/mo |
| gpt-oss-20b (MoE)3.6B active per token | 20.9B | 42 GB | 21 GB | 11 GB | 0.03 GB | 23 GB | 1× RTX 5090 · $279/mo |
| gpt-oss-120b (MoE)5.1B active per token | 116.8B | 234 GB | 117 GB | 64 GB | 0.04 GB | 123 GB | 2× A100 · $1,298/mo |
| Mistral Small 3.2 24B | 24B | 48 GB | 24 GB | 13 GB | 0.17 GB | 28 GB | 1× RTX 5090 · $279/mo |
| Gemma 3 27B | 27.4B | 55 GB | 27 GB | 15 GB | 0.52 GB | 31 GB | 1× L40S · $429/mo |
| Gemma 4 31B | 31.3B | 63 GB | 31 GB | 17 GB | 0.92 GB | 36 GB | 1× L40S · $429/mo |
| Gemma 4 26B-A4B (MoE)3.8B active per token | 25.8B | 52 GB | 26 GB | 14 GB | 0.23 GB | 29 GB | 1× RTX 5090 · $279/mo |
| Qwen3.6 35B-A3B (MoE)3B active per token | 36B | 72 GB | 36 GB | 20 GB | 0.02 GB | 39 GB | 1× L40S · $429/mo |
| GLM-4.5-Air, 106B (MoE)12B active per token | 106.9B | 214 GB | 107 GB | 59 GB | 0.19 GB | 114 GB | 2× A100 · $1,298/mo |
GB are 109 bytes. Mixture-of-experts models are counted with all their experts resident, as every serving engine loads them. KV figures follow each model's attention layout: grouped-query, latent (DeepSeek, Kimi) or sliding-window layers (Gemma, gpt-oss).
How the estimate is built
- WeightsParameters × bytes per parameter: 2 at 16-bit, 1 at FP8 or INT8, about 0.55 at 4-bit once quantisation scales are included.
- KV cache (inference)2 × layers × KV heads × head dimension × 2 bytes per token, times context length and concurrent sequences — computed from each model's own configuration, with latent attention and sliding windows taken into account.
- Training stateFull fine-tuning adds gradients and Adam states in mixed precision: about 14 bytes per parameter on top of the 2 of the weights. LoRA and QLoRA train small adapters: about 0.05 bytes per parameter.
- Activations (training)With gradient checkpointing, each layer keeps its input: about 2 × hidden size × layers × tokens × 2 bytes.
- Runtime1.5 GB per GPU for the CUDA context and libraries, plus 4% of the weights for buffers and fragmentation; 5% of every GPU is left free. For the B300 the calculation starts from the 270 GB per GPU that NVIDIA rates as usable on HGX B300 boards, not the 288 GB physically fitted.
- SuggestionConfigurations are ranked by monthly price, raised by 30% for every extra PCIe card and 10% for every extra NVLink GPU a model is split across — one big card usually beats four small ones.
From an estimate to a server
Memory decides whether a job runs; bandwidth and compute decide how fast. Once you know the number, the GPU comparison shows what each card does with it, and every GPU page has its price by term.
- Single GPU when you can. A model that fits one card avoids tensor-parallel overhead entirely.
- NVLink when you split. H100, H200, B200 and B300 nodes keep eight GPUs on one fabric; PCIe cards suit replicas and pipeline parallelism.
- Leave room for growth. Longer contexts and more users grow the KV cache, not the weights.
GPU memory: questions answered
How much GPU memory do I need to run a 70B model?
For the weights alone: about 141 GB at 16-bit, 71 GB at FP8 and 39 GB at 4-bit. Add the KV cache — about 0.33 GB per 1,000 tokens of context for Llama 3.3 70B at 16-bit — and a few GB for the runtime. At FP8 with an 8k context, that is about 78 GB in total, which fits on one NVIDIA RTX PRO 6000 Blackwell ($749 a month).
Can I run Llama 3.3 70B on a single GPU?
Yes, with quantisation: at FP8 it needs about 78 GB with an 8k context, which fits a 96 GB RTX PRO 6000 or a 141 GB H200; at 4-bit it fits a 48 GB L40S with a short context. At 16-bit it needs about 151 GB, which means two 80 GB GPUs or one 180 GB B200.
How much memory does DeepSeek-V3 or R1 need?
The full 671B mixture-of-experts model keeps every expert in memory: about 671 GB at FP8, the precision it was released in, plus a small KV cache thanks to its latent attention. That is an 8-GPU node — the cheapest fit here is an 8× H200 node at $10,990 a month.
How much GPU memory does fine-tuning need?
Full fine-tuning with Adam in mixed precision needs about 16 bytes per parameter before activations — some 112 GB for a 7B model. LoRA on a 16-bit base needs about 2 bytes per parameter plus adapters; QLoRA on a 4-bit base about 0.6 bytes per parameter, which puts a 70B model within reach of a single 80 GB GPU.
Why does context length matter so much?
Because the KV cache grows with every token of every sequence in flight. A model that fits comfortably with a 4k context can run out of memory at 128k, or with 32 users at once. Use the calculator's context and concurrency fields: the cache is often bigger than the weights.
How accurate is this calculator?
It is an estimate built from each model's published architecture: exact for weights and KV cache, approximate for runtime buffers and activations, which depend on the engine (vLLM, TensorRT-LLM, llama.cpp), the batch scheduler and the quantisation format. We keep 5% of every GPU free and add 1.5 GB per GPU for the CUDA context. Plan a margin before you run at the limit.
Your estimate fits 1× RTX PRO 6000 — $749 a month
Dedicated, single-tenant and live in under 10 minutes. Pay in BTC, ETH or USDT, with no KYC.