Mistral · 24B parameters
Mistral Small 3.2: GPU memory requirements and which GPU to rent
Mistral's 24-billion-parameter model with its vision encoder: 40 layers and grouped-query attention, and a strong single-GPU model at FP8 or 4-bit.
- Weights at FP824 GB48 GB at 16-bit · 13 GB at 4-bit
- FP8, 8k context, 1 sequence28 GBweights + KV cache + runtime
- Suggested here1× RTX 5090$279 a month · dedicated
Inference
How much GPU memory Mistral Small 3.2 needs to serve
Total memory for one sequence at each context length, and for 16 concurrent sequences at 8k — weights, KV cache at 16-bit and runtime overhead included.
| Weights precision | Weights | 4k context | 32k context | 128k context | 8k × 16 sequences | Suggested here, 8k |
|---|---|---|---|---|---|---|
| 16-bit (BF16 / FP16) | 48 GB | 52 GB | 57 GB | 73 GB | 73 GB | 1× A100 · $649/mo |
| 8-bit (FP8 / INT8) | 24 GB | 27 GB | 32 GB | 48 GB | 48 GB | 1× RTX 5090 · $279/mo |
| 4-bit (GPTQ, AWQ, Q4) | 13 GB | 16 GB | 21 GB | 37 GB | 37 GB | 1× L4 · $169/mo |
GB are 109 bytes. Context lengths are shown for scale where the model supports them; an FP8 KV cache halves the cache part of each figure. Counted with its 0.4B vision encoder, as served.
Hardware
GPU servers that run Mistral Small 3.2
For FP8 inference with an 8k context and one sequence: the configurations we would suggest, in order — one large card ahead of several small ones.
Fine-tuning Mistral Small 3.2
Memory for a 2k-token sequence and a micro-batch of 4, with gradient checkpointing. Full fine-tuning keeps gradients and Adam states for every parameter; LoRA and QLoRA train small adapters on a frozen base.
| Method | Memory | Suggested here |
|---|---|---|
| QLoRA, 4-bit base | 23 GB | 1× RTX 5090 · $279/mo |
| LoRA, 16-bit base | 59 GB | 1× A100 · $649/mo |
| Full fine-tune, mixed precision | 394 GB | 2× B300 · $7,580/mo |
Architecture at a glance
Counted with its 0.4B vision encoder, as served. Figures from the model's published configuration; method on the calculator page.
Mistral Small 3.2 hardware: questions answered
How much VRAM does Mistral Small 3.2 need?
For the weights alone: about 48 GB at 16-bit, 24 GB at FP8 and 13 GB at 4-bit. Serving it at FP8 with an 8k-token context and one sequence takes about 28 GB in total once the KV cache and the runtime are counted; long contexts and many concurrent users add the KV cache on top — about 0.17 GB per 1,000 tokens at 16-bit.
Can Mistral Small 3.2 run on a single GPU?
Yes, at FP8 with a moderate context: one NVIDIA RTX 5090 (32 GB GDDR7) holds it, for $279 a month here. Longer contexts or more concurrent users may need a larger card or a second GPU.
What is the cheapest GPU setup to run Mistral Small 3.2?
For FP8 inference with an 8k context and one sequence, the configuration we would suggest is 1× RTX 5090 · $279/mo; at 4-bit, 1× L4 · $169/mo. Suggestions favour one large card over several small ones: splitting a model across PCIe cards costs speed.
How much GPU memory do I need to fine-tune Mistral Small 3.2?
With a 2k-token sequence and a micro-batch of 4: about 23 GB for QLoRA on a 4-bit base, 59 GB for LoRA on a 16-bit base, and 394 GB for a full fine-tune with Adam in mixed precision. Gradient checkpointing is assumed; sharding the optimizer across GPUs spreads the full fine-tune over several cards.
Run Mistral Small 3.2 on 1× RTX 5090 for $279 a month
Dedicated, single-tenant and live in under 10 minutes — with vLLM, the Hugging Face stack or Ollama preinstalled if you want. Paid in BTC, ETH or USDT, with no KYC.