Tesla P40 24GB for Ollama and local LLMs - how to set it up and what it can really do

The short answer: yes, the P40 is very well suited to local LLMs, provided you stick to quantised models (GGUF) in llama.cpp or Ollama and do not try to run something that expects native FP16. Ignore this caveat and you will be convinced you have bought a faulty card. Take it into account and you get 24 GB of VRAM for working with 13B and 34B models for a fraction of the price of new hardware.

I write this from the perspective of someone who has this card in a home server, not from the perspective of someone who wants to sell it (although, as it happens, I do sell it; see the disclaimer at the end). Below is what is worth knowing before you put a P40 into a chassis and start wondering why your favourite framework will not start.

Why the P40 in particular

The Tesla P40 is a card from 2016: Pascal generation, GP102 GPU, 24 GB of GDDR5 memory and 346 GB/s of bandwidth. On paper it is nothing special by today's standards, but for local LLMs one thing matters: the amount of VRAM for a reasonable price. 24 GB is the threshold at which interesting things start to happen: 13B models fit with room to spare, and 34B can be squeezed in with suitable quantisation.

That is why the P40 is so popular in the r/LocalLLaMA community: not because it is fast (it is not), but because it offers a memory capacity for which you would pay several times more in newer cards. It is a server card, so it is passively cooled (250W), it is powered through an 8-pin EPS connector (not PCIe, more on that shortly), and it has no video output. It is not a card for gaming or for connecting a monitor, but a pure compute accelerator.

Quantisation instead of FP16: the most important thing to understand

Here is the catch that most often spoils people's first impression. The P40's GP102 chip runs native FP16 at only 1/64 of its FP32 rate (the P100, also Pascal, is the exception, with fast FP16). If you try to run something that computes in FP16 (ExLlama and ExLlamaV2, or transformers pipelines that load the weights in FP16), the card will crawl, not because it is weak, but because that particular number format does not suit it. Some engines will not start at all: current vLLM requires compute capability 7.5 or higher, so it does not support the P40.

The solution is simple and proven by the whole llama.cpp community: GGUF quantisation. Models in Q4_K_M, Q5_K_M or Q8_0 format compute mainly in INT/FP32, which the P40 does well: ~11.8 TFLOPS of FP32 and ~47 TOPS of INT8 is real power for a card that costs a fraction of the price of new hardware. Ollama and llama.cpp work with GGUF by default, so if you stay within this ecosystem, the FP16 issue does not affect you at all.

A practical rule: if the framework you want to use requires native FP16 to operate, it is not a suitable match for the P40. If it works with quantised weights, it is.

What fits in 24 GB

Approximately, without quoting made-up tokens-per-second figures (because they depend on the model, the context, the quantisation and whatever else is running on the card):

• 7B–8B: comfortably, with plenty of headroom for the context.
• 13B: fits easily; this is probably the most convenient starting point on a single P40.
• 34B: fits only at around 4-bit quantisation: a 34B model in Q4_K_M is about 20 GB, which leaves roughly 3-4 GB for the context, while Q8_0 (about 36 GB) does not fit on a single card at all.
• 70B: no chance on a single card with sensible quantisation. Two P40s in one host make running 70B models a realistic possibility; this is one of the main reasons people buy them in pairs.

If you need something more than the P40 offers (faster generation, native FP16 without workarounds, more headroom on a single card for 34B), it is worth considering a step up, for example the V100 32GB. Note: a 70B model will not fit on a single 32 GB card anyway (in Q4 it realistically needs ~40–48 GB), so for 70B you are aiming for a pair of cards either way: 2× V100 32GB or 2× P40. But that is a different budget. The P40 is a good starting point, but not the only option: if you need something newer or with more VRAM, we also source other accelerators to order and quote for them individually on request.

Subjectively: 13B in Q4/Q5 on a single P40 runs smoothly, at a pace suitable for normal interactive work. 34B is noticeably slower, but still usable. For a sense of scale: at 346 GB/s the theoretical single-stream ceiling is about 44 tokens/s for a 13B model in Q4_K_M (7.9 GB) and about 17 tokens/s for a 34B one (20 GB); real figures are lower. The performance estimates above are theoretical ceilings, not benchmark results. My impressions of usability come from everyday use, and everyone should verify performance with their own model and use case.

Cooling and power in practice

Three things you need to think about before the card will even start in a desktop chassis:

Cooling. The P40 is passive: in a server it is cooled by the airflow forced through the chassis. An ordinary PC has no such airflow. You need a shroud (a cover that directs the air) and an additional blower fan, typically a 40mm blower mounted on the rear edge of the card. Without it the card will overheat, start throttling or, in the worst case, shut down under load. This is not an optional extra; it is a condition for it to work at all.

Power. The P40 needs 8-pin EPS power (the same connector that powers the CPU on the motherboard), not the standard 8-pin PCIe connector used by graphics cards. They look similar, but they are not the same. Before you order a cable, check what your power supply actually has; sometimes you need an EPS cable for a modular PSU or an adapter, such as NVIDIA's own one that takes two PCIe 8-pin cables and outputs a CPU 8-pin plug. Never force a PCIe 8-pin plug into the card: the 12 V and ground pins are on opposite sides.

BIOS. In compute mode the P40 asks for a 32 GB BAR1 address window. Enable "Above 4G Decoding" in the motherboard BIOS before you install the card; without it the card may not initialise (on Windows it shows up with error Code 12).

Step by step: how to get it running

Without inventing commands that do not exist, this is the general outline:

1. Driver. On Linux, install an NVIDIA driver from the 580 branch with proprietary kernel modules and pin that branch (Ollama needs 570 or newer for this card). NVIDIA's open kernel modules do not support Pascal. The 580 branch is the last one that supports Pascal: 590 and later no longer support the P40, so upgrading to the newest branch can make the card unavailable to the NVIDIA driver. On Linux the same driver with proprietary kernel modules supports both Tesla and GeForce cards; on Windows use NVIDIA's R580-family data-centre driver for Tesla cards. Linux is clearly the simpler route here than Windows.
2. Checking the card. Before you put anything into production, it is worth checking the condition of the hardware, especially if it is a second-hand card. A test such as gpu-burn is a good first safeguard: it shows whether the card can sustain the load without computational errors and without overheating. Also read the ECC error and retired-page counters (nvidia-smi -q -d ECC,PAGE_RETIREMENT), which record the memory's history; ECC is enabled by default on the P40.
3. Installing Ollama. A standard installation of Ollama on Linux: the tool detects the available NVIDIA GPU by itself, provided the driver is correctly installed.
4. Downloading a model. Download a model in GGUF format through Ollama (or directly through llama.cpp, if you prefer lower-level control). It is best to start with something in the 7B-13B range to check that everything works before moving on to larger models.
5. Testing. Run a query and watch VRAM usage and the temperature under load: this is the moment you find out whether the cooling is really sufficient.

Pitfalls to watch out for

• FP16 as the first instinct. Described above, but worth repeating, because it is the most common cause of disappointment: trying to use a framework that assumes native FP16.
• Drivers. The P40 needs a driver from the 580 branch or older (570 or newer for Ollama), and Linux requires NVIDIA's proprietary kernel modules. The open kernel modules do not support Pascal. Installing "the latest" NVIDIA driver without checking can give you a 590+ driver that no longer supports Pascal, making the card unavailable to the NVIDIA driver. On supported distributions, Ollama's installer attempts to install the unpinned cuda-drivers package if it detects NVIDIA hardware but finds no working driver. Install the 580 branch with proprietary kernel modules and pin that branch before installing Ollama.
• No video output. Obvious, but people still sometimes try to connect a monitor to it. The card is purely for compute: the system boots from another graphics card or from integrated graphics, and the P40 only adds computing power.
• Pascal is the oldest generation in our range. It works very well today, but the end of software support has already been announced: CUDA 13.0 dropped Pascal (compute capability 6.1), CUDA 12.9 is the last toolkit that builds for it, the R580 driver branch (supported until mid-2028) is the last one for it, and PyTorch 2.14 is the last release with a ready-made wheel. Ollama and llama.cpp still support the P40 today, but when planning several years ahead, count on a frozen software stack rather than on new releases.

Disclaimer

I sell the P40 and the other accelerators linked in this article through Agile Store, so weigh my recommendations with that in mind. Everything above comes from my own use of the card in a home server; verify it against your own model and workload before you buy.