Short answer: in our measurements the V100 generated text about 2.6–3.1× faster than the P40 with the same model and quant (depending on the quant and the V100 variant), but that is not the question worth asking first. The first question is: is generation actually the bottleneck in your RAG system, or is it retrieval and context processing? Where the context is short, the difference between the P40 and the V100 in prompt processing is smallest; where it is long, it grows.
We sell used Teslas, so treat this post as a seller's measurement: the figures come from our own test bench, not from manufacturers' datasheets. If you are building a local RAG and hesitating between the P40 24 GB and the V100 16/32 GB, below you will find data from one model, one backend and one methodology for all three cards.
One guiding example: Bielik-11B on three cards
I stick to one model throughout the post: Bielik-11B-v2.3-instruct (SpeakLeash), because it is a model we actually use in Polish RAG deployments. I tested it in three quantisations (Q4_K_M, Q5_K_M, Q8_0) on three cards (P40 24 GB, V100 16 GB, V100 32 GB) at four context lengths (512 / 2048 / 4096 / 8192 tokens), through Ollama 0.33.2, at the full power limit. A methodological caveat that applies to the whole post: each data point is a single run, we measured one unit of each card, and in our configuration the Ollama runner handled one active request (we did not log the OLLAMA_NUM_PARALLEL value; a constant VRAM peak independent of the number of parallel requests in another of our tests points to this). Where I write about a "consistent pattern", I mean repeatability within our measurement matrix, not independent repeats.
Generation / decode (tg, tokens/s): here the V100 clearly wins
| Card | Quant | ctx 512 | ctx 2048 | ctx 4096 | ctx 8192 |
|---|---|---|---|---|---|
| P40 | Q4_K_M | 27.90 | 27.54 | 26.63 | 25.53 |
| V100-16 | Q4_K_M | 79.84 | 74.90 | 76.18 | 72.53 |
| V100-32 | Q4_K_M | 73.82 | 73.09 | 71.48 | 68.63 |
| P40 | Q5_K_M | 23.90 | 23.53 | 23.11 | 22.23 |
| V100-16 | Q5_K_M | 67.99 | 70.51 | 69.07 | 66.64 |
| V100-32 | Q5_K_M | 67.98 | 66.92 | 64.93 | 62.31 |
| P40 | Q8_0 | 18.13 | 18.07 | 17.79 | 17.20 |
| V100-16 | Q8_0 | 55.79 | 55.63 | 54.21 | 52.66 |
| V100-32 | Q8_0 | 54.17 | 52.66 | 51.32 | 50.33 |
At Q4_K_M and ctx 512 the V100-16 gives 79.8 tok/s against 27.9 tok/s on the P40: that is 2.86×. The advantage falls very slightly with a growing context (at 8192 it is 2.84×), but it does not disappear: Volta holds it across the whole range of contexts studied. A hypothesis about the cause: the V100 has HBM2 memory with higher bandwidth than the GDDR5 in the P40, and autoregressive decode in this class of workload is often strongly limited by memory bandwidth; this is a mechanism consistent with the typical decode profile and with the observed advantage of the V100, but our test does not isolate it. The nominal bandwidth ratio is about 2.6× (900 vs 346 GB/s), while our measured generation advantage was 2.65–3.08×, so the nominal bandwidth ratio alone does not reproduce the full measured difference. The cards also differ in SM architecture and clocks, and kernels may use their memory with different efficiency. For single-stream generation with quantised weights, llama.cpp commonly uses MMVQ kernels for weight matrix-vector multiplication; those kernels do not use Tensor Cores, but the exact path depends on the version and configuration. We did not establish the kernels used in this measurement or the exact contribution of memory bandwidth to the observed difference.
Prefill / prompt processing (pp, tokens/s): the performance gap differs from that in generation
| Card | Quant | ctx 512 | ctx 2048 | ctx 4096 | ctx 8192 |
|---|---|---|---|---|---|
| P40 | Q4_K_M | 462.3 | 563.4 | 540.6 | 506.7 |
| V100-16 | Q4_K_M | 803.5 | 1638.2 | 1816.3 | 2132.9 |
| V100-32 | Q4_K_M | 781.1 | 1722.7 | 1770.7 | 2138.4 |
Here the V100's advantage grows with the context: from about 1.7× at 512 tokens to more than 4× at 8192. A hypothesis: prefill is limited by compute power to a greater extent than decode (matrix multiplications over the whole context at once), so the difference in compute (Volta has Tensor Cores, Pascal does not) would play a larger part here than in generation. That is consistent with the gap growing with the context, but our test does not isolate it. For RAG the practical conclusion is independent of the cause: the longer the context you inject (more fragments (chunks) from retrieval), the more the P40 loses to the V100 at the prefill stage, i.e. processing the context before the model starts answering. Prefill time is the main component of the time to first token, but not the full TTFT: we did not measure that end to end.
V100-16 vs V100-32: does the extra memory cost performance?
Short answer: in our measurement series not much, but the difference is visible. At Q4_K_M/ctx 512 the V100-16 gives 79.84 tok/s, the V100-32 73.82 tok/s; the difference shrinks with the context (at 8192 it is 72.53 vs 68.63). Our V100-32 unit was in this series usually a few per cent (0 to about 8%, typically about 5%) slower than our V100-16 unit on the same model and quant. This is an observation on our pair of units; the cause is undetermined. NVIDIA's public specification gives the same nominal memory bandwidth for both V100 variants, so the result may stem from, among other things, differences between units, the clocks reached under load, temperature, the slot and platform, or measurement variance. The pattern is consistent in our measurement matrix, but without repeats and multiple units we do not settle whether it is a feature of the 32 GB variant, of this particular pair of cards or of the bench conditions. The purchasing conclusion remains the same: you buy the V100-32 for capacity, not for speed: if the model and context fit in 16 GB, the extra memory gives you nothing in generation speed, and our V100-16 unit generated text faster in this measurement series (in prefill the two were within about 5% of each other).
VRAM: where 24 GB makes a difference
| Card/quant | VRAM peak ctx512 | VRAM peak ctx8192 |
|---|---|---|
| P40 Q4_K_M | 6.7 GB | 8.4 GB |
| P40 Q8_0 | 11.6 GB | 13.2 GB |
| V100-16 Q8_0 | 11.9 GB | 13.4 GB |
| V100-32 Q8_0 | 11.9 GB | 13.5 GB |
Bielik-11B in Q8_0 at an 8k context takes about 13.2–13.5 GB of VRAM on all three cards, so the V100-16 fits it with a margin of just under 3 GiB (16,384 − 13,404 MiB nominal; a little less is usable in practice). That matters if you are thinking of running Q8_0 together with a separate card/process for embeddings, or of a context longer than 8k: the V100-16 starts to get tight, while the P40 24 GB and the V100-32 have much more headroom.
Quality: is it worth paying in speed/VRAM for a higher quant?
We also measured answer quality (not just speed) on the V100-32, the scifact-pl corpus, 60 rated answers per quant: Q4_K_M ≈ 3.10, Q5_K_M ≈ 3.05, Q8_0 ≈ 3.00 (the test's correctness scale, higher = better). The ratings were given by an LLM judge, without a statistical test: with a sample like this we treat differences of this order as undetected, not as proof of equality. An additional measurement on the P40 (Q8_0, 20/20 answers, 2026-09-11) did not change this picture. In other words: in this particular test Q4_K_M gave the best speed/VRAM trade-off with no detected loss of quality: higher quants cost speed and VRAM, and we did not observe an improvement in quality on this task. A caveat: this is one model (Bielik-11B) and one corpus (scifact-pl, Polish biomedical abstracts); on another type of content (e.g. legal texts where exact wording carries a lot of weight) the quality difference between quants may look different; do not extrapolate this conclusion without checking it yourself on your target data.
Energy and temperature: the hidden cost of the P40
Peak power at Q4_K_M: P40 about 229 W, V100-16 about 247 W, V100-32 about 250 W. This is the maximum from sampled nvidia-smi telemetry (1 Hz) over the whole run, not the TDP: a single telemetry reading is not the same as the declared power limit, and we did not diagnose the causes of any discrepancies. For energy calculations it serves only as a rough proxy for comparing the cards, not as an energy measurement. The difference in peak draw between the cards is small, and tg on the P40 is about 2.6–3.1× lower, hence in our simple peak_pwr × time estimate the P40 comes out worse in energy per generated answer, not better. On top of that comes idle: our measurement (2026-09-11) shows the P40 with the highest idle power draw of all the cards tested (about 39–49 W), while the V100-16 is about 34–37 W and the V100-32 about 25–36 W. We did not record whether the model was held in VRAM during this idle measurement. Ollama keeps models in memory for five minutes without requests by default; this can be changed with OLLAMA_KEEP_ALIVE or the request's keep_alive parameter. With no model loaded, the P40 can enter a low-power state: forum users report about 8–10 W, but we did not measure it. If the RAG server spends long periods idle, idle power can contribute substantially to its electricity bill. Comparing the cards' running costs requires actual power draw and time spent both idle and serving requests; peak-power differences alone do not settle the comparison.
What about embeddings?
This is a separate stage of the RAG pipeline (indexing documents, not answering questions), but it is worth mentioning here, because the GPU for the LLM usually computes the embeddings too. Our earlier test (bge-m3, batch 64) showed the V100-16 about 3× faster than the P40 (30.7 vs 10.0 fragments/s); for e5-large at the same batch the ratio was similar (40.7 vs 14.6). With the more demanding bge-m3, the V100-16 GB runs out of memory at batch 128, while the P40 with 24 GB of VRAM still got through batch 128 (we did not run larger bge-m3 batches on the P40). If you plan large, infrequent re-indexing of an extensive corpus, the extra capacity of the P40/V100-32 matters in practice here; with small, incremental updates the difference is less noticeable.
When the P40 is enough, and when to reach for the V100
The P40 24 GB makes sense when:
- Your traffic is low (single queries, not a queue of users waiting for answers): 25–28 tok/s at Q4 is still a readable pace for live generation.
- You are building a large index and want spare VRAM for embeddings/a longer context without compromises.
- Budget is the priority, not the number of simultaneous users.
The V100 (16 or 32 GB) makes sense when:
- The response time of a single query matters: a faster decoding rate reduces the time taken to generate each answer proportionally (we did not measure serving many simultaneous users in this test).
- The context injected into RAG is long (many fragments from retrieval): that is where the V100 reaches more than a 4× advantage over the P40 in prefill alone.
- The model and context fit in 16 GB: in our measurement series the V100-16 unit generated text faster than the V100-32 (prefill was on a par), so paying extra for capacity you will not use does not buy speed.
What follows from this
A difference of about 2.6–3.1× in generation alone sounds unambiguously in favour of the V100, and where the bottleneck is generation speed it really is decisive. But it is not the only number that matters: in prefill the V100's advantage grows with the context, in VRAM it is the P40 and the V100-32 that give headroom the V100-16 does not have, and the P40 had the highest idle draw in our measurement, although we did not record whether the model was loaded, so this result does not establish the ranking after models are unloaded. Choose the card according to where the bottleneck of your RAG really lies, not according to which card has the higher number in its specification.
Sources
The benchmark results come from our own measurements, carried out on our own test bench. Nominal memory bandwidth figures come from the NVIDIA datasheets linked below; the unloaded P40 idle-power estimate is attributed to forum reports in the Energy section.
- LLM inference measurements (Bielik-11B-v2.3, Ollama, Q4_K_M/Q5_K_M/Q8_0 quants, context 512–8192 tokens) on the P40, V100-16 GB and V100-32 GB cards: full power limit, a uniform measurement methodology for all three cards, one run per data point, one unit of each card; peak power is the maximum from sampled telemetry over the whole run. Measurement date: 2026-09-06.
- Answer quality measurements (Q4 vs Q5 vs Q8, scifact-pl corpus, rated on a 1–5 scale) and idle power draw per card. Measurement date: 2026-09-11.
- Embedding throughput measurements (e5-large, bge-m3) on the P40 and V100. Measurement date: 2026-09-01.
- For the nominal memory bandwidth quoted in the text: NVIDIA Tesla V100 datasheet (900 GB/s for both V100 variants): https://images.nvidia.com/content/technologies/volta/pdf/tesla-volta-v100-datasheet-letter-fnl-web.pdf and NVIDIA Tesla P40 datasheet (346 GB/s GDDR5): https://images.nvidia.com/content/pdf/tesla/184427-Tesla-P40-Datasheet-NV-Final-Letter-Web.pdf.
Full documentation of the methodology and the raw measurement data are available on request: write to us.
FAQ
- How much faster is the V100 than the P40 at text generation?
- In our measurements (Bielik-11B, Ollama) about 2.6–3.1× depending on the quant and the V100 variant: e.g. Q4_K_M at a 512 context: 79.8 tok/s on the V100-16 against 27.9 tok/s on the P40. The advantage holds across the whole context range up to 8192.
- Is the difference the same in prompt processing (prefill)?
- No: there the V100's advantage grows with the context: from about 1.7× at 512 tokens to more than 4× at 8192. If your RAG injects a long context from retrieval, in our benchmark that is where the difference grows the most.
- Will Bielik-11B in Q8_0 fit on a V100 16 GB?
- Yes: at an 8k context it takes about 13.2–13.5 GB, so just under 3 GiB of headroom remains on the V100-16. The P40 24 GB and V100-32 leave a much bigger margin for a longer context or an additional process; whether a specific embedder fits alongside has to be worked out separately.
- Is the V100 32 GB faster than the V100 16 GB?
- Not in our series: the V100-32 unit was usually about 5% slower at generation than the V100-16 (e.g. 73.8 vs 79.8 tok/s at Q4/ctx 512). In our test the advantage of the 32 GB variant was capacity, not generation speed.
- When is a P40 enough for local RAG?
- When traffic is low (single queries), the index is large and the budget matters more than response time: 25–28 tok/s at Q4 for a single user can be acceptable interactively (that is an assessment, not a measurement), and 24 GB gives a VRAM margin the V100-16 does not have.