Which GPU for LLM inference? (Used accelerators for local models)

The short answer before I go into detail: if you are setting up a local language model for the first time and do not know where to start, in most cases the Tesla P40 with 24 GB of VRAM is the best place to start. Not because it is the fastest, but because it gives you a lot of memory on a single card for reasonable money. The rest of this article explains why VRAM matters more here than raw compute power, and when it is worth departing from this default choice.

Why VRAM, not TFLOPS, matters for inference

When I started experimenting with local models, I instinctively looked for the card with the highest compute performance; after all, this is "AI". I soon realised that this was the wrong way of thinking, at least for inference (simply querying a ready-made model, without training).

A language model has to keep its weights in the card's memory, and it is the weights, not the number of CUDA cores, that decide whether the model can be run at all. If the model does not fit in VRAM, that is the end of it: either an out-of-memory error, or some of the layers end up on the CPU and performance drops drastically. That is why the first question I ask when looking at a card is not "how many Tensor Cores does it have", but "how many gigabytes do I have to work with".

This is where quantisation helps: reducing the precision of the weights from native FP16/BF16 to INT8 or even INT4. It sounds like a loss of quality, and there is indeed some loss, but in practice (with formats such as GGUF, discussed below) the difference can be hard to spot, while the VRAM saving is huge: a 4-bit model takes up roughly a quarter of the space of the FP16 version. Thanks to this, 24 GB of memory can hold models that "on paper" would require much more.

Once the model fits, memory bandwidth is worth checking too. When generating one sequence at a time (batch size 1) with a dense model such as Llama 2 13B, the card reads essentially all of the weights for every new token, so dividing bandwidth by the size of the weights gives a rough ceiling set by reading them alone: a 13B model at Q4 (about 8 GB) gives around 25 tokens/s at 200 GB/s and around 112 tokens/s at 900 GB/s. This is an idealised estimate: real speed also depends on the kernels, the compute cost and KV-cache handling, so bandwidth alone does not settle how the cards rank.

Approximate VRAM for each model size

A few rough figures that I keep in mind when planning:

• 8 GB: 7-8B models in quantised form, with limited headroom for long context.
• 16 GB: 13B comfortably, with room for context and perhaps a second small model alongside.
• 24 GB: 13B with room to spare, and 34B models only just, depending on the quantisation.
• 2×24 GB (48 GB in total): the realistic entry point for 70B, that is, serious 70B-class models such as Llama 3.3 70B in sensible quantisation (Q4 usually needs about 40–48 GB). A single 32 GB card is enough only with aggressive quantisation and/or some layers offloaded to the CPU.

These are approximate values, not hard rules: actual usage depends on the model architecture, the context length and the quantisation format, but as a starting point they work well.

Which used NVIDIA cards make sense for inference

In home labs and in second-hand listings, the cards that turn up most often come from the Tesla/data-centre family: the P4, P40, P100, V100 and the newer A2. They span three architectures (Pascal, Volta and Ampere), and each card is a different compromise. This is not an exhaustive list: these are simply the models we most often have in stock; we source other accelerators to order and quote for them individually on request.

Pascal: P4, P40, P100, an older but still useful generation

All three are based on the Pascal architecture (compute capability 6.x). There is a trap here that I once fell into myself: the P40 and P4 have limited native FP16 support, slower than computing in FP32. That would be a problem for training, but for inference with llama.cpp and GGUF models it makes little difference, because llama.cpp avoids FP16 on these cards and uses integer and FP32 kernels instead. Engines built around FP16, such as ExLlama, run poorly on them, and vLLM does not support Pascal at all. The P100 is a different matter: it has full FP16 and strong FP64, so it is more interesting if you also care about scientific computing, not only about LLMs.

The P40 has 24 GB of GDDR5 and a bandwidth of about 346 GB/s. It is the default starting choice I mentioned at the beginning: plenty of VRAM, a low entry price and a proven position in the community (there are many threads on r/LocalLLaMA and in the llama.cpp GitHub issues and discussions devoted to this very card). I describe this approach in detail in a separate article on the Tesla P40 with Ollama, if you want to see specific settings.

The P4 is in a completely different league: 8 GB of GDDR5, 192 GB/s, and power drawn from the PCIe slot alone (75 W), with no extra cable. It makes sense if you want an energy-efficient small server for genuinely small models, or as a second card to take load off the main one; like the other Teslas it is passive, so it still needs directed airflow.

The P100: 16 GB of HBM2, 732 GB/s, faster memory than the P40, although with less capacity. It makes sense when bandwidth and full FP16 matter to you and 16 GB is enough.

Volta: V100, when it is worth going higher from the start

The V100 is in a different class: the Volta architecture (CC 7.0), first-generation Tensor Cores, HBM2 memory with up to 900 GB/s, in 16 GB or 32 GB versions. If you already know that you are interested in larger models (34B) or in serving several models at once, consider the 32 GB version rather than adding more smaller cards. Note that if you plan to serve with vLLM, current releases require compute capability 7.5 or newer, which excludes both Pascal and the V100; llama.cpp and Ollama run on all the cards listed here. Bear in mind, however, that a 70B model at a sensible quantisation (Q4, about 40-45 GB) still needs two cards; a single 32 GB card only fits it at aggressive 2-3-bit quantisation. As for drivers: CUDA 13 dropped Pascal and Volta. CUDA 12.9 is the last toolkit that builds for them, the R580 driver branch is the last to support them (maintained until mid-2028) and PyTorch 2.14 is the last release with a ready-made wheel. Of the cards discussed here, only the Ampere-based A2 remains on the current support track.

Ampere: A2, modern and low-power, but not for large models

The A2 is an Ampere card (CC 8.6), with third-generation Tensor Cores and BF16/TF32 support. However, it has only 16 GB of GDDR6 at 200 GB/s, a low-profile form factor and a power draw of around 40–60 W. I would choose it not for maximum capacity, but for edge applications: a small server where power consumption and chassis size matter more than whether a 70B model will fit; noise depends on the cooling system, as the A2 is passive and needs directed airflow. It handles suitably quantised models of around 13B, but its 200 GB/s gives a lower theoretical ceiling on generation speed than the P40 or P100; real results depend on the model, quantisation, context and software. For a card running 24/7 the low power draw makes a real difference to the electricity bill.

Comparison table for inference

Card	VRAM	Memory bandwidth	Architecture	Native FP16	Power	Best for
P4	8 GB GDDR5	192 GB/s	Pascal	limited	from the slot, 75 W	small models, low power draw and low cost
A2	16 GB GDDR6	200 GB/s	Ampere	full (+BF16/TF32)	low profile, 40–60 W	edge, low power draw
P100	16 GB HBM2	732 GB/s	Pascal	full	8-pin EPS	bandwidth + FP64
P40	24 GB GDDR5	346 GB/s	Pascal	limited	8-pin EPS	default starting point, large quantised models
V100 32GB	32 GB HBM2	up to 900 GB/s	Volta	full + 1st-gen Tensor	8-pin EPS	training/fine-tuning, 34B (70B as a pair); Volta dropped in CUDA 13 (last toolkit: 12.9)

A practical note on power, which I once had to relearn the hard way: the V100, P100 and P40 have an 8-pin CPU/EPS connector, which differs from PCIe 8-pin. Power it with a CPU/EPS 8-pin cable from the power supply or a compatible adapter (two PCIe 8-pin to one CPU/EPS 8-pin); without correctly connected power the card will not start. The P4 is powered entirely from the slot, and the A2 also has low requirements. The second point: none of these cards has a video output. They are pure compute accelerators, so for display you need the CPU's integrated graphics or a separate graphics card. The third: in compute mode, the P40 exposes a 32 GB memory window (BAR1), so enable "Above 4G Decoding" in the BIOS before installing it; on older boards without this option the card may not initialise at all.

In practice: llama.cpp, Ollama and GGUF quantisation

All this theory about VRAM translates directly into practice. The standard in the local LLM community today is the GGUF format supported by llama.cpp (and the tools built on it, such as Ollama). It lets you keep a model quantised to 4 or 8 bits and load it directly onto the card.

A few things I have verified in practice:

• Q4_K_M quantisation is usually a sensible trade-off between quality and size. If you have spare VRAM, Q5 or Q8 are worth trying, but this is less critical than whether the model fits in memory at all.
• Ollama downloads and runs the model in the background, but under the hood it uses the same mechanism as llama.cpp: both pick the number of layers offloaded to the GPU automatically, but on older cards with limited memory it is worth checking and, if needed, setting it by hand (num_gpu in Ollama, --n-gpu-layers in llama.cpp).
• Linux is the natural environment for these cards: the data-centre drivers and CUDA work better than on Windows. Windows can be configured, but if you have the choice, Linux will save you a good deal of frustration.
• A common myth: you do not need a vGPU licence to compute with CUDA. That restriction applies to GPU virtualisation across virtual machines, not to running computations on bare metal.
• If the card is passive (no fan, like most Teslas), in a desktop you have to add a shroud and a blower fan to force airflow, otherwise sooner or later you will get throttling or the card will shut down from overheating. This is a prerequisite for stable operation, not an optional extra.

I cover in more detail exactly how much VRAM you need for specific model sizes (7B, 13B, 70B) in a separate article on choosing VRAM for Llama, which gives more specifics for different quantisation levels.