LLM VRAM Calculator
Estimate the GPU memory (VRAM) needed to run a large language model locally: weights at FP16, INT8 or 4-bit, plus the KV cache for your context length and batch size, plus overhead.
Last updated: October 2026
Estimated VRAM needed
19.08 GB
Formula below · 2 sources (developer.nvidia.com, arxiv.org) · Updated Oct 2026
Compare with similar
About this calculator
Running a model needs memory for three things:
1. Weights: parameters × bytes per parameter. 2 bytes at FP16/BF16, 1 at INT8/FP8, 0.5 at 4-bit. NVIDIA’s example: a 7B model in 16-bit takes about 7B × 2 bytes ≈ 14 GB. 2. KV cache: the attention keys and values kept for every token in every sequence. Following NVIDIA’s formula, per token it is 2 (key and value) × layers × (KV heads × head dimension) × bytes, and in total that times context length times batch size. Models with grouped-query attention store fewer KV heads than attention heads, which is why the KV-head count is asked for. 3. Overhead: the CUDA context, activations, temporary buffers and fragmentation, entered as a percentage of the first two.
VRAM = (weights + KV cache) × (1 + overhead)
Results are in GB of 1,073,741,824 bytes (GiB), the unit GPU memory sizes are quoted in, so a "24 GB" card holds 24 of these. The defaults are Llama 3.1 8B (32 layers, 8 key/value heads, head dimension 4,096 ÷ 32 = 128, per the Llama 3 paper). Real 4-bit formats store scaling factors as well, so they average a little over 4 bits per weight; inference servers such as vLLM also reserve a fixed share of the GPU for the KV cache up front. Treat the result as a planning estimate and keep 1–2 GB spare for the display and drivers.
How to use
Enter the parameter count, choose the weight precision, and take the layer count, KV heads and head dimension from the model’s config.json (num_hidden_layers, num_key_value_heads, and hidden_size ÷ num_attention_heads). Then set the context length and how many sequences run at once. Example (the defaults): Llama 3.1 8B at BF16 with an 8,192-token context. Weights: 8 × 10⁹ × 2 bytes = 16 × 10⁹ bytes = 14.90 GB. KV cache: 2 × 32 × 8 × 128 × 8,192 × 2 bytes = 1.00 GB (0.125 MB per token). With 20% overhead: (14.90 + 1.00) × 1.2 = 19.08 GB, which fits a 24 GB card. Example 2: the same model at 4-bit: weights 3.73 GB, total (3.73 + 1.00) × 1.2 = 5.67 GB. Example 3: Llama 3.1 70B (80 layers, 8 KV heads, head dimension 128) at 4-bit with 8,192 tokens: weights 32.60 GB, KV cache 2.50 GB, total 42.12 GB.
Frequently asked questions
How much VRAM do I need for a 7B or 8B model?
At 16-bit precision the weights alone are about 13–15 GB (7–8 billion × 2 bytes), so a 16 GB card is tight and 24 GB is comfortable once the KV cache and overhead are added. At 4-bit the weights are about 3.5–4 GB and the model runs on an 8 GB card with a moderate context.
Why does a long context use so much memory?
The KV cache grows linearly with context length and batch size. For Llama 3.1 8B at 16-bit it is 0.125 MB per token, so a full 128K-token context needs 16 GB of KV cache on top of the weights. Quantising the KV cache to 8-bit halves it.
Does 4-bit quantization reduce quality?
Usually a little. 8-bit is typically very close to 16-bit quality, and good 4-bit methods keep most of it on large models; smaller models lose more. Test on your own tasks.
Is this enough for fine-tuning?
No. Training also stores gradients and optimizer states, which for full fine-tuning with Adam take several times the memory of the weights. This calculator covers inference only.
Related calculators
Sources & references
Think a result is wrong? Report a problem with this calculator.