Not at token generation: there it is surprisingly good for its price and 24 GB of memory. It is slow at processing a long prompt. These are two different phases, and confusing them leads to disappointment.

Can Tensor Cores or Flash Attention be added through the driver?

Tensor Cores are a hardware feature the GP102 chip lacks: no driver can add them. Flash Attention is something else: it is a software optimisation, available in llama.cpp through -fa on|off|auto (current builds default to auto) and in Ollama through OLLAMAFLASHATTENTION=1, but its real gain on the P40 depends on the version, the model and the kernels, and we did not measure it in this campaign. In practice the prefill problem is worked around with better settings and a shorter context, not with a flag.

How long does prefill really take on a P40?

In our tests Bielik-11B Q4KM on the P40 does prompt eval at ~594 tok/s with 512 tokens and ~551 with 2048. At pp2048 ≈551 tok/s, prompt evaluation of 2048 tokens alone corresponds roughly to ~3.7 s of work measured by llama-bench: this is not the application's full TTFT (tokenisation, scheduling, transfers, server overhead and the first decode step come on top). Between 512 and 2048 tokens the throughput changed little, so in this range prefill time grows mainly with the number of tokens; with much longer contexts the relationship may change. Your figure depends on the model, the quantisation, the context and the batch.

I have RAG with long documents: should I take a P40 or a V100?

If the context is large and response time matters to you, a V100 32GB. If the budget is tight and the documents can be trimmed aggressively, a P40 may be enough. Write to us and I will describe both options realistically.

Do Ollama and llama.cpp support these batch settings on the P40?

Partly. llama.cpp exposes both --batch-size and --ubatch-size; Ollama, which builds on llama.cpp's GGML code, exposes only numbatch (default 512) as a request option, with no separate ubatch setting, and turns Flash Attention on with OLLAMAFLASHATTENTION=1. For tuning prefill, llama.cpp gives you more control. I have described a practical P40 setup for a local LLM in the post on the P40 and Ollama.

← Blog

Why the prompt is slow on a P40 - and how to really improve it

Agile Store · 21 October 2026 · 12 min read

topic:troubleshooting
use:llm
arch:pascal
model:p40-24
model:v100-32

Short answer: the Tesla P40 generates tokens decently, but processing a long prompt (prefill) is slow on it, because the P40 (the GP102 chip) has no Tensor Cores and does not offer the high FP16 throughput known from later accelerators. If your use case is short questions and long answers, the P40 can be a very good choice in terms of cost/VRAM. If it is agents and RAG with a large context, the prefill will hurt and it is worth considering a V100 32GB.

I sell used P40s and I see how people trip up on this. Someone buys the card, starts a model, asks a short question: it works instantly. Then they paste half the documentation into the context and suddenly the card "hangs" for a dozen or so seconds before it even starts answering. Nothing has broken. That is simply how Pascal works. I would rather tell you now, before you buy, than read later that I cheated you.

Before we get into the details, a map of the symptom, the cause and what to do about it:

Symptom	Cause	What to do
Long time before the first token	The cost of prefill (prompt processing)	Shorten the actual input context, choose the quantisation, measure your own time
Generation is OK after the start	Decode has a different profile (memory-bandwidth limit)	Do not diagnose the card as faulty
Still too slow despite tuning	A limitation of the architecture / the nature of the workload	Consider a V100 32GB

The essentials to start with

Token generation (decode) on the P40 is OK: that is the most common reason people buy it and are happy with it.
Prompt processing (prefill) is slow: no Tensor Cores, no high FP16 throughput on GP102. The longer the context, the longer you wait for the first token.
It hurts mainly with agents, RAG and work on long documents, where the context can be thousands of tokens with every query.
It can be mitigated: the choice of quantisation, the batch/ubatch settings in llama.cpp, a shorter actual input context and, when the workload has a common prefix, prompt/prefix cache mechanisms.
There are no miracles: if fast prefill is critical, the V100 32GB is a much better choice than the P40 in this class of used cards.
The specific figures depend on the model and settings; below I give our own measurements on the P40 (Bielik, PLLuM, Llama), not other people's charts.

What actually happens: prefill versus decode

LLM inference has two phases. The first is prefill: the model has to read the whole prompt (the context) and compute its representation before it produces the first token. This is a heavily matrix-based operation, very well parallelised, and here newer GPUs with suitable matrix units achieve much higher throughput. The second phase is decode: generating the next tokens one by one; at a small batch size, decode of typical models is often limited above all by memory bandwidth rather than by raw FP16 compute power.

The P40 (the GP102 chip) has no Tensor Cores and does not offer the high FP16 throughput known from later accelerators. This is a feature of this particular chip, not of the whole Pascal family: the P100/GP100 is an important exception here (I covered the architectural details in the post on what CUDA 13 means for Pascal and Volta). On the other hand, the P40 has 24 GB of memory and decent bandwidth, so it does surprisingly well at decode. The result: a short question → a fast answer. A long prompt → the card first "thinks" through the prefill, and only then does generation start.

This is not a fault of the unit. It is the performance profile of the whole architecture.

When it really hurts (and when it does not)

I will say it plainly, because this is the most important section of this post:

It does not hurt when you ask short questions and want longer answers. Chat, a writing assistant, translating short passages, generating code from a short description: here the P40 is a very sensible price/memory choice.
It hurts when every query carries a large context with it: agents repeatedly including their tool-use and reasoning history, RAG adding several document fragments, analysis of long files, "read this PDF and answer". Here prefill dominates the response time, and Pascal is at its weakest in this phase.

If your use case is in the second group, read on to the sections on mitigation, but keep in mind that you are up against a limitation of the architecture. I have written more on choosing a GPU for a specific type of inference in the guide to the best GPU for LLM inference.

How to really mitigate slow prefill on the P40

There is no single setting that removes the limitation of the architecture, but a few parameters really help.

1. Choice of quantisation

Quantisation makes the model smaller and reduces the amount of data to churn through. On Pascal it is worth sticking to formats that this architecture handles efficiently in integers, instead of counting on a speed-up from FP16. In practice this means experimenting with the Q4/Q5/Q6 variants in llama.cpp and checking which gives an acceptable prefill time at a quality that is good enough for you.

Our measurements on the P40 (GGUF, llama-bench, -ngl 99, full 250 W power limit, driver 580.173.02, llama.cpp build 6ad4ab0, -r 3): prompt eval / generation in tok/s:

Model / quant	Prompt eval (pp512)	Prompt eval (pp2048)	Generation (tg128)
Bielik-11B Q4_K_M	593.6	550.8	28.5
Bielik-11B Q5_K_M	562.3	n/a	24.3
PLLuM-12B Q4_K_M	612.2	578.0	27.1
Llama-3.1-8B Q4_K_M	933.1	n/a	41.1

With Bielik-11B on our P40, Q4_K_M was faster than Q5_K_M: ~5.6% in pp512 and ~17% in tg128. A smaller quant reduces the amount of data to churn through, but the real gain also depends on the kernel implementation and the cost of dequantisation, so this is one model and one pair of quants: we do not carry this difference over automatically to other models or backends. You can also see that a smaller model (8B) clearly speeds up both phases. Which variant to choose depends on the model and on the quality you find acceptable.

2. Batch and ubatch settings in llama.cpp

This is the most underrated set of knobs. The --batch-size (n_batch) and --ubatch-size (n_ubatch) parameters control the size of the prompt chunks processed at once during prefill. Values that are too small waste the card's parallelism; values that are too large can hit memory. In the version we used, the starting point was -ub 512, but llama.cpp's default values change over time, so check your version's --help instead of relying on a fixed claim about the default. The optimal setting remains specific to the card and model; we did not measure a separate -ub sweep in this campaign, so we do not give figures for it here: it is worth measuring the time to first token on your own prompt.

What we did measure directly is how the cost of prefill grows with the length of the context. Bielik-11B Q4_K_M on the P40, a separate run with a growing prompt (-p 512,2048,4096,8192); its 512 and 2048 results differ from the table above by about 0.3%, which is within run-to-run variation:

Prompt length (tokens)	Prompt eval (tok/s)
512	595.6
2048	551.8
4096	502.7
8192	429.7

The peak VRAM usage across the whole sweep (up to an 8192-token context) was ~8.3 GB, far from the 24 GB limit, so on the P40 it is speed, not capacity, that is the bottleneck here. The prefill rate falls with length (from 595.6 at 512 to 429.7 at 8192 tokens), which means that prefill time grows somewhat faster than the number of tokens: about 0.9 s for 512 tokens, but about 19 s for 8192, 22 times longer for 16 times the tokens. Every input token costs, and the later ones cost more. That leads straight to the cheapest speed-up.

3. A shorter actual input context

The cheapest prefill speed-up is simply fewer input tokens. In RAG that means more aggressive trimming, fewer and better-chosen fragments instead of "let us throw everything in just in case". In agents: compacting the history, summarising instead of appending full logs. Every token you do not send is a token the P40 does not have to churn through.

4. Prompt cache and "warm-up"

It is worth distinguishing two different mechanisms. The KV cache speeds up further decode within a single, active context: that is not the same as reusing a common prefix (prompt/prefix cache) between successive queries. If a large part of the context does not change (e.g. a fixed system prompt, the same documentation), it is the prompt cache that lets you avoid computing the prefill of the same fragment over and over, but how it works depends on the server, the version and whether the prefix matches. Do not assume that every query will automatically recover the previous prefill; this is especially important in RAG, where the fragments often change. Similarly with "warm-up": the first run of a model is often slower than subsequent ones, so a one-off start-up cost should not distort your assessment of performance. Check in your configuration to what extent a given server (llama.cpp, Ollama) caches the prefill.

5. Flash Attention: check it, do not count on it

Flash Attention is an optimisation of the attention mechanism that speeds up work with a long context. llama.cpp has had FP32 Flash Attention kernels for cards without fast FP16, such as the P40, since 2024, and current builds use -fa auto by default, so it may already be on in your setup. We do not assume, however, that it will solve the prefill problem on the P40: the effect depends on the llama.cpp version, the model, the KV cache type and the kernels available. In this campaign we did not measure -fa on against -fa off separately on the P40, so we do not give a percentage gain here. Unlike Tensor Cores, Flash Attention is software, not a missing hardware unit. If you decide to try it yourself, measure both variants on your prompt and compare.

When to stop fighting and take a V100

I will be honest to the end: the settings above mitigate the problem, they do not remove it. If you are building an agent that pushes several thousand tokens of context through at every step, or you work on long documents and the time to first token is critical for you, no knob in llama.cpp will turn Pascal into an architecture with Tensor Cores.

Then, in this class of used cards, a much better choice than the P40 is the V100 32GB. Volta has Tensor Cores and genuinely more efficient FP16, so it handles prefill of a long context clearly better than the P40. The scale of the advantage can be seen on the same model and quant: Bielik-11B Q4_K_M, the same benchmark settings, llama-bench -ngl 99, full power limit (P40 figures from the context-length run):

Metric	P40	V100-PCIE-16GB
Prompt eval pp512 (tok/s)	595.6	1868.6
Prompt eval pp2048 (tok/s)	551.8	1719.9
Generation tg128 (tok/s)	28.5	83.0

In the matched benchmark we had a V100-PCIE-16GB; the 32 GB version recommended above has the same class of compute architecture (Volta, Tensor Cores), but we do not present the figures above as a separate measurement of the V100 32 GB. In this test the V100 was ~3.1× faster than the P40 at prefill (and ~2.9× at generation): the result is consistent with Volta's advantages, although the benchmark does not isolate the effect of the Tensor Cores alone from HBM2 and other differences in architecture. The V100 32GB is a more expensive card; if your model and context fit in 16 GB, the V100 16GB we measured above is an option at a price close to the P40's. I say this knowing that some of you will still choose the P40 for its 24 GB, and rightly so if your prompts are short. The point is for you to make an informed choice, rather than discover the performance profile only after buying the card.

Sources

NVIDIA Tesla P40: datasheet (12 TFLOPS FP32, 47 TOPS INT8, 346 GB/s; no FP16 or Tensor Core figures): https://images.nvidia.com/content/pdf/tesla/184427-Tesla-P40-Datasheet-NV-Final-Letter-Web.pdf; FP16 at 1/64 of FP32: the compute capability 6.1 instruction-throughput table in the CUDA documentation https://docs.nvidia.com/cuda/archive/12.9.1/cuda-c-programming-guide/index.html#arithmetic-instructions, together with NVIDIA's identification of the P40 as CC 6.1 https://developer.nvidia.com/cuda/gpus/legacy.
NVIDIA Volta: architecture whitepaper (Tensor Cores, which Pascal does not have): https://images.nvidia.com/content/volta-architecture/pdf/volta-architecture-whitepaper.pdf
llama.cpp: the --batch-size/--ubatch-size parameters are best checked in llama-cli --help / llama-bench --help for the specific commit (our measurements: build 6ad4ab0); the tool's repository: https://github.com/ggml-org/llama.cpp

Disclaimer: We sell USED cards, from the secondary server market. The P40 is often chosen for its 24 GB of VRAM at a relatively low price on the secondary market, but it has a real limitation in the form of slow prefill: if someone tells you it is fast "at everything", do not quote me on it. Before shipping, we put every card through a multi-stage qualification that includes, among other things, a VRAM check, a burn-in with a watchdog, ECC, PCIe and computational correctness; this of course does not change the performance profile of the architecture: we test that the unit is sound, we do not promise that Pascal will suddenly get Tensor Cores. Unsure whether the P40 fits your use case? write to us: I would rather talk you out of a purchase than accept a return.

FAQ

Is the P40 "slow"?: Not at token generation: there it is surprisingly good for its price and 24 GB of memory. It is slow at processing a long prompt. These are two different phases, and confusing them leads to disappointment.
Can Tensor Cores or Flash Attention be added through the driver?: Tensor Cores are a hardware feature the GP102 chip lacks: no driver can add them. Flash Attention is something else: it is a software optimisation, available in llama.cpp through -fa on|off|auto (current builds default to auto) and in Ollama through OLLAMA_FLASH_ATTENTION=1, but its real gain on the P40 depends on the version, the model and the kernels, and we did not measure it in this campaign. In practice the prefill problem is worked around with better settings and a shorter context, not with a flag.
How long does prefill really take on a P40?: In our tests Bielik-11B Q4_K_M on the P40 does prompt eval at ~594 tok/s with 512 tokens and ~551 with 2048. At pp2048 ≈551 tok/s, prompt evaluation of 2048 tokens alone corresponds roughly to ~3.7 s of work measured by llama-bench: this is not the application's full TTFT (tokenisation, scheduling, transfers, server overhead and the first decode step come on top). Between 512 and 2048 tokens the throughput changed little, so in this range prefill time grows mainly with the number of tokens; with much longer contexts the relationship may change. Your figure depends on the model, the quantisation, the context and the batch.
I have RAG with long documents: should I take a P40 or a V100?: If the context is large and response time matters to you, a V100 32GB. If the budget is tight and the documents can be trimmed aggressively, a P40 may be enough. Write to us and I will describe both options realistically.
Do Ollama and llama.cpp support these batch settings on the P40?: Partly. llama.cpp exposes both --batch-size and --ubatch-size; Ollama, which builds on llama.cpp's GGML code, exposes only num_batch (default 512) as a request option, with no separate ubatch setting, and turns Flash Attention on with OLLAMA_FLASH_ATTENTION=1. For tuning prefill, llama.cpp gives you more control. I have described a practical P40 setup for a local LLM in the post on the P40 and Ollama.