P40 or P100 for LLMs - the FP16 trap everyone gets wrong

The short answer: for a local LLM in quantised form (GGUF, Ollama/llama.cpp), choose the Tesla P40 24GB: more VRAM, larger models, peace of mind about the context. Choose the Tesla P100 16GB with its trade-offs in mind, and only if you really need fast FP16 (2:1) and high HBM2 bandwidth or strong FP64 for scientific computing, at the cost of 8 GB less memory and no low power-saving states. The P100 does not have low power states like the P40, so at idle it maintains elevated clock speeds and draws around ~30 W (instead of ~10 W like the P40 with no model loaded); that is not 250 W (that is the TDP under load). And most importantly, the thing people get wrong: the P40 has very weak FP16 (around 1/64 of FP32), so native FP16 and ExLlama run terribly on it; on the P40, stick to quantisation. On the P100 it is the other way round: FP16 is its strength.

I am writing this because these two cards are notoriously confused: both "Pascal", both "an old 16/24 GB Tesla", both inexpensive. Yet they differ in a way that determines whether your model will run at a usable speed at all. To be upfront: yes, we sell both (full disclaimer at the end). That is precisely why I want you to buy the right one rather than return the wrong one.

Where the whole confusion comes from

Both cards use the Pascal architecture, but they are two different chips under the hood. The P40 was built as an inference accelerator on the GP102 graphics chip (the same one as in the GeForce GTX 1080 Ti / Titan X of that generation), but without display outputs: it is a server card, and you cannot connect a monitor to it, with 24 GB of GDDR5 memory. The P100 is a pure compute (HPC) chip, with 16 GB of HBM2 memory and, this is the key, fully capable, fast FP16 and FP64.

And here is the core of the confusion: people assume that "newer/more expensive/HBM2 = better for LLMs". Not true in general. For quantised inference what matters above all is the amount of VRAM (so that the model fits), and a typical GGUF stack does not load the P40's weakest path (native FP16), and here the P40's 24 GB wins on usefulness. The P100's fast FP16 helps only when you actually compute in FP16, and with GGUF you usually do not.

The FP16 trap on the P40: the most important paragraph

The P40 has deliberately limited FP16: its FP16 throughput is around 1/64 of its FP32 performance. This is not a fault; it is a Pascal design decision for this particular chip. The practical consequences:

• Native FP16 models (without quantisation) run very slowly on the P40. Not "a bit slower": in FP16 arithmetic the P40 delivers about 0.19 TFLOPS, roughly a hundred times less than the P100's 18.7, so anything that really computes in FP16, prompt processing above all, slows down dramatically.
• ExLlama / ExLlamaV2 and similar backends that compute in FP16 will disappoint on the P40. Not because the card is bad, but because you are using its weakest path.
• The solution: if you are buying the P40 primarily for local LLMs, stick to GGUF/quantisation through llama.cpp/Ollama. In typical GGUF use, the P40's weak FP16 is not the main limitation, because the stack uses paths optimised for quantisation (the exact behaviour depends on the backend, the quantisation, the kernels and the software version), and the P40 then shows its real value: 24 GB of inexpensive memory for a large model.

On the P100 it is the other way round: FP16 is 2:1 (twice as fast as FP32), so the card copes where the P40 chokes. But, and this needs saying plainly, neither of these cards has Tensor Cores. Tensor Cores arrived only with Volta (V100) and later. So if you are dreaming of "tensor" speed-ups, neither the P40 nor the P100 will give them to you; that is what the V100 32GB is for.

VRAM vs bandwidth: what really limits inference

It is worth separating two things that are easy to confuse: the amount of VRAM decides whether the model fits at all, and memory bandwidth affects how fast it generates tokens (generating tokens one at a time is largely limited by memory bandwidth; processing the prompt is limited mainly by compute). On paper the HBM2 in the P100 has more than twice the bandwidth of the GDDR5 in the P40 (732 vs 346 GB/s), which raises the ceiling on generation speed when the model fits in its 16 GB. A ceiling is not a guarantee, though: in our llama.cpp test below, the P100 generated no faster than the P40.

The problem is that in practice you first have to fit. If your model plus context takes up 18–20 GB, it will not fit entirely in a single P100's 16 GB of VRAM. Ollama can run a model partly on the GPU and partly on the CPU and in system RAM; in llama.cpp you can adjust GPU layer placement or use automatic fitting. Running part of the model on the CPU can substantially slow generation, but the size of the slowdown depends on the hardware, model and settings. High HBM2 bandwidth alone does not guarantee an advantage over a card that holds the entire model in VRAM. On a P40 with 24 GB the same model fits with room to spare. That is why, for large models and a long context, the P40's capacity advantage beats the P100's bandwidth advantage, simply because "it fits" beats "it would be faster if it fitted".

Also remember the KV cache: the longer the context, the larger the memory buffer for the conversation history, and it can add several GB. That is another argument for 24 GB if you plan long prompts or a larger batch.

Power states: the P100's quiet drawback

Something price lists say nothing about, but which matters in practice: the P100 does not have low power-saving states like the P40: at idle it maintains elevated clock speeds and draws around ~30 W (instead of ~10 W like the P40). That is not 250 W (that is the TDP under load), but it is still clearly more than the P40 at idle. (Our burn-in: P100 idle average ~29 W; the P40 drops to ~8–10 W with pstate management; the average across the whole sample can be higher, because some cards ran without that management.) For a constantly loaded compute server this is irrelevant (it computes at full tilt anyway). But for a home server that spends most of the day waiting for the occasional request, it is a real cost in electricity and heat. The P40's idle advantage depends on whether a model remains loaded and whether pstate management is used. Community reports on the NVIDIA forums and in the nvidia-pstated project describe the P100 as having only P0, and the P40 as remaining in P0 while a model holds a CUDA context, unless pstate management lowers it between requests. These mechanisms are documented in community sources rather than a primary NVIDIA specification. Unloading the model, including after Ollama's keep-alive period, allows the P40 to return to its low power state.

In addition, both models are passive: without forced airflow they will throttle. I have described how to cool such a card in a desktop separately in the guide to cooling passive server cards.

Table: P40 vs P100 where it hurts

Feature	Tesla P40 24GB	Tesla P100 16GB
Architecture	Pascal	Pascal
VRAM	24 GB GDDR5	16 GB HBM2
Memory bandwidth	lower (GDDR5)	higher (HBM2)
FP16	very weak (~1/64 FP32)	fast (2:1)
FP32	12 TFLOPS	9.3 TFLOPS
FP64	weak	strong (an asset in science)
Tensor Cores	none	none
Power states / idle	low (~8–10 W idle with no model loaded or with pstate management)	no low states: elevated clock speeds, ~30 W idle
Best for	large models in GGUF, inference	fast FP16, HPC/FP64, smaller models
Pitfall	do not compute in native FP16/ExLlama	only 16 GB: a larger model will not fit

Our head-to-head measurement (file Bielik-11B-v2.3-Instruct-Q4_K_M.gguf, llama-bench -ngl 99 -p 512/2048 -n 128 -r 3 = average of 3 runs; full 250 W power limit. Test bench: 2× Xeon E5-2697 v3 (24 cores), 19 GB RAM, Ubuntu 24.04, driver 580.173.02 / CUDA 12.0, PCIe Gen3; the raw CSV and the llama.cpp build log are available on request). All results below are in tok/s (higher = better):

Card	Prompt eval pp512	Prompt eval pp2048	Generation tg128
P40 24GB	593.6	550.8	28.5
P100 16GB	419.0	402.1	27.2

A result that goes beyond the thesis of this article: despite its faster HBM2 memory, the P100 has LOWER prompt eval than the P40, and generation is practically equal, even though this model (about 6.7 GB) fits comfortably in 16 GB. One possible explanation is kernel selection: the P40 supports the INT8 __dp4a instruction (compute capability 6.1), which the P100 lacks (6.0). In llama.cpp, dot products using this instruction have a fallback implementation on the P100, but prompt processing may instead use dequantisation and floating-point matrix multiplication. The P40 also has higher peak FP32 throughput (12 vs 9.3 TFLOPS). Without identifying the build and kernels used, we cannot establish the cause of the difference. In this GGUF test, the P100's HBM2 bandwidth did not produce higher speed, while the P40 offers 8 GB more VRAM. Results with another model, quantisation and context length will be different; this measurement shows the behaviour of both cards in an identical test environment.

Verdict: which card and when

Choose the Tesla P40 24GB if:

• You run a local LLM in quantised form (Bielik, Llama, PLLuM through Ollama/llama.cpp): the most common scenario.
• You want the largest possible model on a single card or a long context: 24 GB gives real headroom that 16 GB does not have.
• You want the best ratio of GB of VRAM to price and better behaviour at idle.
• You accept one condition: you stick to GGUF, not native FP16.

Choose the Tesla P100 16GB if:

• You really need fast FP16 (a specific workflow that computes in half precision, outside llama.cpp/GGUF) with a model that fits in 16 GB.
• You compute in FP64: simulations and scientific computing, where double precision is your bread and butter. This is where the P100 shines brightest and beats the P40 hands down.
• You accept less VRAM and higher idle power draw.

And if you want Tensor Cores or more power than either card → that is the V100 32GB: Volta, HBM2, Tensor Cores and 32 GB for a larger model with a long context. I have written separately about choosing a card strictly for LLM inference in which GPU for LLM inference, and I show a practical P40 setup for a home model in Tesla P40 for Ollama.

Sources

• NVIDIA Tesla P40: technical data (24 GB, 346 GB/s, 12 TFLOPS FP32, 47 TOPS INT8): https://images.nvidia.com/content/pdf/tesla/184427-Tesla-P40-Datasheet-NV-Final-Letter-Web.pdf; GDDR5 memory: https://images.nvidia.com/content/pdf/tesla/Tesla-P40-Product-Brief.pdf; FP16 performance of approximately 0.19 TFLOPS: https://developer.nvidia.com/blog/mixed-precision-programming-cuda-8/; FP16-to-FP32 throughput ratio of 1:64 for compute capability 6.1: https://docs.nvidia.com/cuda/archive/12.6.0/cuda-c-programming-guide/index.html#arithmetic-instructions.
• NVIDIA Tesla P100: technical data (16 GB HBM2): https://images.nvidia.com/content/tesla/pdf/nvidia-tesla-p100-PCIe-datasheet.pdf
• NVIDIA Pascal Architecture Whitepaper: the primary source for the P100/GP100: FP16 2:1 (double throughput) and the FP64 units: https://images.nvidia.com/content/pdf/tesla/whitepaper/pascal-architecture-whitepaper.pdf
• Idle power draw: our burn-in (20× P100, 41× P40; nvidia-smi monitoring): P100 idle average ~29 W (no low states), the P40 drops to ~8–10 W with pstate management. The same lack of low states on DC Teslas is documented for the Tesla V100 (nvidia-pstated #6): https://github.com/sasha0552/nvidia-pstated/issues/6
• NVIDIA Tesla V100: technical data (Tensor Cores, Volta): https://images.nvidia.com/content/technologies/volta/pdf/tesla-volta-v100-datasheet-letter-fnl-web.pdf

---

A disclaimer to finish, as always: yes, we sell both the P40 and the P100, so read this with that in mind. But I do not want you to buy "either one": I want you to buy the right one for your task, because returning the wrong card is a waste of time for both of us. We test every unit under load and include a burn-in certificate (temperature, power draw, ECC errors): with used cards, that is the difference between a promise and proof. For reference, the average of the maximum temperatures under full load in our sample is ~64 °C (P40) and ~62 °C (P100). Not sure whether your workflow is GGUF (→ P40) or FP16/FP64 (→ P100)? Write to us and describe what you want to compute: we will help you choose, rather than sell you something more expensive.

(See also: Which GPU for LLM inference · Tesla P40 for Ollama · Cooling passive server cards)