How much VRAM do you need to run Llama 7B, 13B and 70B?

In short: a quantised 7B model will fit in 8 GB, 13B needs 16 GB to run comfortably, 34B only just fits in 24 GB, and for 70B you realistically need around 40-48 GB of VRAM, that is, two 24 GB cards or one card with a large amount of memory. Below I explain where these figures come from and why the context can upset all of it.

I write this from the perspective of someone who has set up local LLMs on used, second-hand Teslas: not from a manufacturer's brochure, but from real tests in a home rack. Tesla cards (P4, P40, P100, V100) and their successor the A2 have no video output and no active cooling. They are server-room cards, bought precisely for this kind of application, so "how much VRAM" is the number one question here, before anyone even reaches for their wallet.

Rule of thumb: parameters × bits + context

Before we get to the table, it is worth understanding where these figures come from in the first place, because it lets you work out the requirements of any model that does not appear in any comparison.

The simplest approximation: model size in GB ≈ number of parameters (in billions) × quantisation bits / 8, plus overhead for the format and buffers. In practice, for the popular Q4 quantisation (4-bit, e.g. GGUF Q4_K_M), this comes to roughly 0.55-0.65 GB for every billion parameters. A 7B model in Q4 therefore needs about 4-4.5 GB for the weights alone. In Q8 (8-bit) it is about 7-8 GB, and in full FP16 about 14 GB.

That is not the end of it, however. On top of the model weights comes the KV cache, the memory for the context: the conversation history and the prompt currently being processed. The longer the context (4K, 8K, 32K tokens), the more VRAM the model's "working memory" alone occupies, regardless of the weights. For smaller models and short contexts this is a margin of around 1-2 GB, but with 70B and a long context window it can grow to a dozen or more gigabytes on top. Treat this as a rule of thumb, not a rigid formula: the exact value depends on the model architecture (the number of layers and attention heads) and on the implementation (llama.cpp manages memory differently from native PyTorch). The sizes in this article are model classes rather than exact releases: 7B and 13B are Llama 2, today's Llama 3.x comes as 8B and 70B, and 34B stands for models such as Code Llama 34B; the rule works for any dense model.

Table: model × quantisation → VRAM → card

Model	Q4 (4-bit)	Q8 (8-bit)	FP16	Suggested card
7B	~4-5 GB	~7-8 GB	~14 GB	P4 8GB (Q4, tight) or V100 16GB with headroom
13B	~7-8 GB	~13-14 GB	~26 GB	V100 16GB comfortably in Q4/Q8
34B	~18-20 GB	~34-36 GB	~68 GB	P40 24GB in Q4, only just
70B	~39-43 GB	~70-75 GB	~140 GB	2× V100 32GB or 2× P40 24GB

The values are approximate: the actual size of a GGUF file or checkpoint can differ by a dozen or so per cent depending on the quantisation method (Q4_0 vs Q4_K_M vs Q5_K_M and so on), but the order of magnitude is right, and that is what matters most when planning a purchase.

This is not an exhaustive list: I am showing the cards most commonly used for these applications and currently available from us; if you need a different model (for example, one with more VRAM), we can source it to order, with a quotation on request.

One practical note: the A2 16GB is a low-profile, energy-efficient card (40-60W), excellent for lighter inference tasks or as a second card in a system, but with large models its memory bandwidth (about 200 GB/s, GDDR6) will be noticeably lower than that of the HBM2 in the V100 (900 GB/s) or the P100 (732 GB/s). More on that shortly.

Why the context can upset your plan

You have set up a 13B model in Q4 on a 16 GB card and it seemed you had 8 GB to spare? You switch on a 16K or 32K token context (because you happen to be feeding a long document into the prompt) and suddenly you start getting CUDA out of memory. This is a classic on r/LocalLLaMA: people count only the model weights and forget that the KV cache grows linearly with the context length and with the number of concurrent sessions (the batch).

In practice: if you plan to use a long context (RAG with large documents, long conversations), budget the KV cache explicitly instead of hoping the model fits. For one sequence, FP16 KV cache uses about 0.82 MB per token for Llama 2 13B, which has no grouped-query attention (GQA), and about 0.33 MB per token for Llama 2 and Llama 3.1 70B with GQA. For 70B, that is 1.25 GiB at 4096 tokens and 10 GiB at 32768 tokens, or about 1.34 GB and 10.74 GB respectively. These calculations do not establish native context support: Llama 2 has a native 4096-token window, while Llama 3.1 supports up to 128K. Two 24 GB cards can accommodate 70B Q4 at 4K-8K, depending on the weight format, usable memory and buffers. For Q4_K_M at 32K, plan on two 32 GB cards or consider quantised KV cache, and budget implementation overhead and parallel sequences as well.

Two cards or offloading to RAM

When a model does not fit on one card, there are two ways forward.

Two cards (multi-GPU). llama.cpp and Ollama can split a model's layers between cards (tensor/layer split): for example, 70B in Q4 on 2× P40 24GB or 2× V100 32GB. This works quite well, provided the cards are combined in one system with adequate power (remember that the V100 and P40 use the EPS connector, so you need the right cabling in the chassis or server). The performance loss with a good split is usually acceptable, because during token generation most of the time is dominated by memory bandwidth anyway, not by communication between the cards.

Offloading to system RAM. llama.cpp also lets you keep some of the model's layers in RAM and the rest on the GPU (--n-gpu-layers). It works, and it will run a model you could not otherwise start, but at the cost of speed. Dual-channel DDR4/DDR5 RAM in a typical desktop delivers around 40-100 GB/s, many times less than VRAM (P40: 346 GB/s, V100: 900 GB/s); even a multi-channel server board, at roughly 200-460 GB/s in theory, stays below the faster cards. For large models in partial CPU offload mode, the drop in tokens generated per second can be drastic: a comfortable chat turns into waiting for every answer. This is a fallback, not a target solution.

Ollama and llama.cpp in practice

In Ollama it is simple: you choose a model tag (for example, llama3:70b-instruct-q4_0) and Ollama itself tries to load it into the available VRAM, offloading part of it if necessary. It is worth watching ollama ps, which shows what percentage of the model actually sits on the GPU.

In plain llama.cpp you have more control: with the -ngl (number of GPU layers) flag you set manually how many layers go onto the card. This is useful when you want to deliberately leave a margin for a longer context instead of loading the model "to the brim". Ollama's default context depends on detected GPU memory: its documentation lists 4K below 24 GiB VRAM, 32K at 24-48 GiB, and 256K from 48 GiB. The actual context also depends on the model limit and settings. Set it with OLLAMA_CONTEXT_LENGTH and check the allocated context with ollama ps. With Flash Attention enabled, OLLAMA_KV_CACHE_TYPE=q8_0 roughly halves KV cache memory, usually with very little precision loss.

Pascal cards (P4, P40, P100) have no Tensor Cores; those arrived with Volta. On the P4 and P40, FP16 also runs at only 1/64 of the FP32 rate, so llama.cpp uses integer and FP32 kernels on them, which suits quantised GGUF models, in any case the natural choice for local LLMs. The P100 is the exception: it has fast FP16 (twice its FP32 rate) and full FP64, which matters more for training and fine-tuning than for inference alone. One thing to know before buying: CUDA 13 dropped Pascal and Volta, so the P4, P40, P100 and V100 stay on CUDA 12.9 and the R580 driver branch (supported until mid-2028). llama.cpp and Ollama still run on them, but current vLLM needs compute capability 7.5 or higher and runs on neither generation. The V100 (Volta, first-generation Tensor Cores) and the A2 (Ampere, BF16/TF32) handle FP16 natively, which can be convenient in some pipelines, although GGUF dominates local inference in practice anyway.

If you are buying a used card for this purpose, whatever its generation, put it through a thorough load test (gpu-burn) before you train anything on it or serve anything in production. Passive cards are particularly easy to overheat when there is insufficient airflow in the chassis, and the fault only shows up under full load, not at idle.