Used NVIDIA data-centre GPUs for AI - a practical guide to the Tesla V100, P40, P100, P4 and the A2

I bought my first Tesla P40 because my model would not fit into the 8 GB of VRAM on my "ordinary" graphics card, and new hardware with 24 GB cost as much as a used car. The card arrived in an anti-static bag, with no box, no fan, a bare heatsink and a power connector that matched nothing I had in my drawer. After two evenings of struggle I had a local LLM running on it, and that is how it all began.

This article covers everything I did not know then and wish someone had told me. To put it plainly: server cards can deliver enormous performance for a fraction of the price of new hardware, but they have their quirks, and you will not read about them in the specification.

Why used server cards at all?

The logic is simple and unforgiving. Hardware that until recently ran production AI workloads in data centres reaches the secondary market once a lease ends or a facility is upgraded, and it loses value faster than consumer cards. The same V100 that listed at around USD 10,000 in 2018 is now within the budget of a hobbyist, a start-up or a university laboratory.

You pay for this in three ways: the architecture is several generations behind the current one (Pascal and Volta predate Turing, Ampere, Hopper and Blackwell), there is no manufacturer's warranty, and the hardware is "server-grade" in every inconvenient sense of the word (more on that shortly). But if your task is running inference on a local model, fine-tuning, experimenting with diffusion models or scientific computing, rather than training GPT-5 from scratch, the performance-to-price ratio is unbeatable.

Before you buy: four things that surprise everyone

1. These are passive cards. They have no fan. In a server they are cooled by a powerful airflow from the front of the chassis. Put such a card in an ordinary desktop, load it, and within a few tens of seconds it will start throttling and then shut down to protect itself. This is the most common cause of disappointment. The solution: either install the card in a server chassis with proper airflow, or add your own cooling, most commonly a 3D-printed shroud with a 40 mm blower (turbine) fan. This is not optional. There is no "perhaps the case airflow will be enough". It will not.

2. Power comes through an EPS connector, not PCIe. Cards such as the V100, P40 and P100 draw power through an 8-pin EPS connector (the same type that powers the CPU on the motherboard), not through the 8-pin PCIe connector used by graphics cards. The plugs look similar but are keyed differently, and forcing a PCIe plug in can damage the card. You need a 2× PCIe to EPS adapter (usually supplied with a well-described used card) and a power supply that can handle the load.

3. There is no video output. These are compute accelerators. You cannot connect a monitor to them. In practice you need a second card or the CPU's integrated graphics for display, and the Tesla serves purely as a compute "engine". For some people this is obvious; for others it is an unpleasant surprise after unpacking.

4. Your motherboard must map a very large memory window. In compute mode the P40 asks for a 32 GB BAR1 address window. Enable "Above 4G Decoding" in the BIOS before installing the card; on older consumer boards without this option the card may not initialise at all (Code 12 on Windows).

Architecture matters, more than you might think

Three generations appear in this class of hardware, and they differ not only in performance but in what you can sensibly do with them at all:

• Pascal (P4, P40, P100): compute capability 6.x. Solid and inexpensive, but with one catch: native FP16 on the P40 and P4 is deliberately limited and actually slower than FP32. That is why you run LLMs on them in quantised form (for example 8-bit or 4-bit GGUF models), and then they work very well. The P100 is the exception here, as it offers full-rate FP16 and strong FP64.
• Volta (V100): compute capability 7.0. This is where first-generation Tensor Cores and wide HBM2 memory appear. It is the first of these cards on which training and fine-tuning in FP16 make real sense.
• Ampere (A2): compute capability 8.6. The newest architecture of the group: third-generation Tensor Cores, BF16 and TF32 support, and low power consumption. You pay a premium for the newer architecture, but you get the longest expected period of software support.

Compute capability is not a decorative number. Successive versions of libraries (CUDA, drivers, ML frameworks) drop older architectures over time, and this is already happening: CUDA 13 removed compiler and library support for architectures older than Turing, which affects both Pascal and Volta (V100). Ampere (A2) has the longest expected period of software support here; it is the only architecture in this group that is not yet on the list of those being phased out.

The cards at a glance: which one for what

NVIDIA Tesla P4 (8 GB): quiet, small and remarkably frugal

Single-slot, low-profile, 75 W with no auxiliary power: it draws all its power from the PCIe slot. That makes it ideal for a small case, a home server or video transcoding (Jellyfin uses it out of the box, Plex with a Plex Pass), provided you still give it directed airflow: it is passive like the others. For AI: smaller models, light INT8 inference, experiments. 8 GB of VRAM is a ceiling you will quickly feel with larger LLMs, but as a quiet all-rounder it is excellent.

NVIDIA Tesla P40 (24 GB): the home lab LLM favourite

Ask on r/LocalLLaMA about a first used card for local models and the P40 will come up in every other answer. The reason: 24 GB of VRAM in a used server GPU. With LLMs memory is king, and here you get plenty of it cheaply. The card runs very well with llama.cpp/Ollama on quantised models: at Q4 a 13B model fits with room to spare (even at Q8), a 30-34B model only just with modest context, and two cards open the door to 70B. You already know the catch: weak FP16, so FP16-based engines such as ExLlama run poorly on it, and current vLLM releases support neither Pascal nor the V100 (they need compute capability 7.5 or newer). Stick to GGUF/INT formats and you will be fine. It is not a speed demon; its main advantage is memory capacity.

NVIDIA Tesla P100 (16 GB): the outsider with strong FP64

Somewhat forgotten between the P40 and the V100, it nevertheless has its niche. HBM2 with 732 GB/s of bandwidth (far more than the GDDR5 on the P40) and genuine FP16 make it better suited to memory-intensive tasks and to scientific computing where double precision (FP64) matters. If you run simulations rather than only LLMs, the P100 can be a wiser choice than the seemingly newer P40.

NVIDIA Tesla V100 (16 or 32 GB): where training begins

This is where Tensor Cores and truly fast HBM2 memory (900 GB/s) come in. On the V100, fine-tuning and training in FP16 stop being an ordeal, and inference is fast. The 16 GB version is cheaper, but VRAM will limit you sooner than compute power, so if the budget allows, 32 GB is the difference between "a curiosity" and "a real working tool". It is as close to a "proper AI card" as you can get for a reasonable sum on the secondary market.

NVIDIA A2 (16 GB): modern features in an efficient package

It stands apart from the rest because it is Ampere: BF16, TF32, third-generation Tensor Cores, combined with power consumption of a few dozen watts and a low-profile form factor. On paper it will not beat the V100 in raw performance, but it wins on efficiency and on the fact that software will support it for much longer. For inference at the edge, for a server that needs to be quiet and economical with power, and for newer data formats, it is the most future-proof card on this list.

Summary table

Card	Architecture	VRAM	Bandwidth	Power	Best for
Tesla P4	Pascal	8 GB GDDR5	192 GB/s	75 W	light inference, transcoding, small cases
Tesla P40	Pascal	24 GB GDDR5	346 GB/s	250 W	local LLMs (plenty of inexpensive VRAM, quantisation)
Tesla P100	Pascal	16 GB HBM2	732 GB/s	250 W	HPC, FP64, memory-intensive tasks
Tesla V100 (PCIe)	Volta	16/32 GB HBM2	900 GB/s	250 W	training and fine-tuning, heavy inference
NVIDIA A2	Ampere	16 GB GDDR6	200 GB/s	~40–60 W	efficient inference, edge, long-term support

I have deliberately left out core counts and TFLOPS: with these cards the choice is decided by VRAM, architecture and power consumption, not by a raw figure from the datasheet.

This is not an exhaustive list: I am showing the cards most commonly used for these applications and currently available from us. If a different model suits your case better (newer, with more VRAM, from another family), we can source it to order, with a quotation on request.

How to match a card to the task

• I want to run local LLMs (Llama, Mistral, Qwen) in Ollama/llama.cpp → think in terms of VRAM first. The P40 24 GB is the default choice to start with; for a 70B model in Q4 you realistically need 40–48 GB, that is, two 24 GB cards (or about 48 GB in total). A single 32 GB card is enough only with aggressive quantisation or with some layers offloaded to the CPU.
• I want to train or fine-tune my own models → the V100. Tensor Cores and HBM2 make a difference that Pascal simply cannot offer.
• Stable Diffusion / generative imaging → the P40 copes thanks to its capacity, the V100 thanks to its speed. Both are sensible; the choice depends on whether comfortable VRAM headroom or speed matters more.
• Scientific computing and simulations (FP64) → the P100 on a budget (4.7 TFLOPS FP64), the V100 if you need more (7 TFLOPS).
• A quiet, economical server for 24/7 inference → the A2 or the P4, depending on the size of the models.

Cooling and installation: where people stumble

I am returning to this because it is where most people lose heart. You have two options. The first: a server (rack) chassis with factory airflow, in which the card works as designed, but you pay in noise and size. The second: your own adaptation in a desktop, a printed shroud made for the specific card model plus a fan (a 40 mm blower, or a Noctua on an adapter). It will take you an evening, costs very little, and makes the difference between a card that computes stably for hours and a card that shuts down in the middle of a job. Add a power supply with headroom and an EPS adapter, and only then is the hardware truly "ready".

If you are looking for a complete platform rather than just an accelerator, see the ready-built Dell PowerEdge R740 GPU servers.

Drivers, software and service life

These cards are most at home on Linux: the data-centre drivers install without trouble, and the whole ecosystem (CUDA, containers, llama.cpp, ML libraries) is native there. They will also run on Windows, but if you are serious about it, sooner or later you will end up on Linux. With the standard data-centre driver on bare metal you do not need any licence for CUDA compute. A licence comes into play only with NVIDIA's vGPU software, for example when you split one card between virtual machines, and it covers compute (NVIDIA Virtual Compute Server, now vGPU for Compute) as well as graphics.

As for service life: Pascal and Volta share the same timetable. The R580 driver branch is the last to support them and is maintained until mid-2028, CUDA 12.9 is the last toolkit that can compile for them, and PyTorch 2.14 is the last release with a ready-made wheel (some frameworks dropped Pascal even earlier). Pascal hardware works very well today, but when you buy a P40 or a P4 you are knowingly buying hardware "for now and a few years", not "for a decade". Ampere (A2) gives you the most room for the future. So treat the V100 the same way as the P40 and P4: excellent "for now", not for a decade. That is no reason not to buy Pascal or Volta; it is a reason to know what you are buying.

The certificate, or how not to buy a pig in a poke

The biggest risk with used server hardware is not that it is old, but that it is faulty or has been worn out. A card from a data centre may have run under full load for years. That is why, before buying, you should ask for proof that the unit has passed a load test, ideally gpu-burn running for a few minutes to just under twenty minutes with no computational errors, together with the card's ECC error and retired-page counters from nvidia-smi -q and its temperatures and clocks under load. It is the same philosophy as a test drive before buying a car: not "how does it look", but "how does it behave under pressure". A unit with such a result offers an entirely different level of confidence from a card described as "checked, it worked when I removed it".

---

I write this guide on the Agile Store blog, where we deal in used NVIDIA Tesla accelerators: every unit that passes through our hands undergoes a gpu-burn test and receives a certificate. But wherever you buy, ask for the result of a load test and prepare the cooling before the card arrives. The rest is pure enjoyment.