2× GPU for LLMs: how llama.cpp splits a model between P40 and V100

Short answer: yes, two cards will let you run a model that does not fit on one, but not because you "add the VRAM up into one big pool" (that does not happen). VRAM does not form one shared pool, but a model can be spread across the memories of several GPUs. llama.cpp and Ollama split the model into layers and distribute them between the cards, so 2× Tesla P40 24GB give you 48 GB of total physical VRAM divided between two devices: room for one large model, at the cost of communication over PCIe. If, however, your model fits on a single larger card, for a workload similar to our test it is almost always better to take a single V100 32GB: fewer bottlenecks, simpler, faster. Below: how this split really works, what you control, where the bottlenecks are (PCIe, P2P topology) and when two cards make sense and when they are an unnecessary complication.

I am writing this because "memory pooling" is the most common myth I hear about multi-card setups. People imagine that two 24 GB cards magically become one 48 GB card and the model behaves as it would on a single GPU. The reality is more prosaic, and much more useful once you understand it. To be honest: yes, we sell these cards, in pairs too (full disclaimer at the end).

How splitting a model really works

A large language model is a stack of layers. Current llama.cpp controls the split between GPUs with the --split-mode option, which accepts, among others, three modes:

• --split-mode layer (layer split): layers 0–N land on GPU 0, layers N+1–M on GPU 1, and so on. During generation the data "passes" from card to card, layer by layer. This is the default mode, the simplest one, and the one we used to measure all the figures in this post.
• --split-mode row (row split): individual operations are cut "across" the cards, which compute them in parallel. It potentially makes better use of both cards, but puts a heavier load on communication between the GPUs. It synchronises the cards after every operation. llama.cpp removed it from the CUDA backend in July 2026 in favour of --split-mode tensor, so on current builds it no longer loads on NVIDIA cards; older builds still have it (details below).
• --split-mode tensor: newer, experimental tensor parallelism with separate requirements (including for Flash Attention/KV). We did not test it in this round of testing, so I will not comment on it.

A note on terminology, because this is a frequent source of mistakes: --split-mode chooses the algorithm (layer/row/tensor), and the separate option --tensor-split (-ts) only sets the proportions in which the model is allocated to the cards (e.g. -ts 1,1 = equally). --tensor-split does not turn on "tensor split": despite its name, it is purely a weighting of the split.

The key point: in all these modes each card stores the parts of the model assigned to it in its own, physically separate memory, and in layer mode the associated KV cache and buffers are distributed along with the layers. There is no shared pool, no "48 GB as one area". There is one large model spread across several smaller memories, and that is entirely sufficient to run a model that would not fit on one card.

That is why I say "48 GB in total, divided between two GPUs" and not "a 48 GB pool": the total memory limits the size of the model you can fit, but the space actually available for the weights alone is smaller: the KV cache, the context and per-device buffers eat into it. This nuance determines what to expect from performance.

What you control: CUDA_VISIBLE_DEVICES, --split-mode and --tensor-split

The levers worth knowing:

• CUDA_VISIBLE_DEVICES: you choose which cards the process sees. CUDA_VISIBLE_DEVICES=0,1 uses the first two; =1 runs only on the second. Useful when you want to separate tasks or keep one card for something else.
• --split-mode (llama.cpp): how to split the model: layer (default), row, tensor. This is the option that decides the algorithm.
• --tensor-split (-ts) (llama.cpp): in what proportions to distribute the model between the cards. With two identical P40s, usually equally (-ts 1,1). If your cards have different amounts of VRAM (e.g. 24 GB + 16 GB), you set the proportions by hand so as not to overfill the smaller one. It is a split weighting, not a choice of mode.
• -ngl / layer offload: how many layers go to the GPU. In llama-cli and llama-server, use -ngl all for full offload; the default auto allows offload to be fitted to available memory. Numeric values must also include the output layer: at least 65 for the 64-layer Qwen2.5-32B and 81 for the 80-layer Llama 3.3 70B, so 99 covers both. In this build, llama-bench takes a numeric value, such as -ngl 99. The rest lands on the CPU, which slows things down dramatically: aim for full offload to the GPU.

In Ollama this happens automatically: a model that fits on one GPU is loaded on that GPU alone, and only a model that does not fit is spread across all visible GPUs. When you want control over the split, llama.cpp gives it to you directly.

Bottlenecks: PCIe, topology and NVLink

Since data travels between the cards, the link between them becomes a bottleneck. And here is a difference you need to know before buying:

• The P40 communicates over PCIe. It has no NVLink. With a layer split this is usually acceptable (the transfer between layers is moderate), but with parallel modes and heavier loads PCIe can become a limit. Make sure each card gets real PCIe lanes, not x1 from a cheap riser.
• The V100 in its SXM2 version has NVLink, a very fast direct link between GPUs. But the Tesla V100-PCIE-16GB/32GB used in our test (the version you put into an ordinary motherboard) does not use NVLink: it communicates over PCIe like the P40. Do not buy SXM2 "for the NVLink" if you do not have an SXM2 server; the card is useless without the right baseboard.

Do not guess from the number of lanes: measure the topology. Before you decide that PCIe is the bottleneck, check the real generation and width of the link and the physical/logical path between the cards:

nvidia-smi topo -m

The matrix shows connections via NVLink (NV#), one or more PCIe switches without traversing the host bridge (PIX/PXB), a PCIe host bridge, typically the CPU (PHB), connections between PCIe host bridges within one NUMA node (NODE), or an interconnect between NUMA nodes (SYS, for example QPI/UPI). Topology, link generation and width, and working P2P access affect communication cost; the "x8 vs x16" label alone does not determine it. Note: PIX/PXB describe the topology; they are not a guarantee that CUDA peer access (P2P) actually works: if you need to know whether P2P actually works, check it separately with a suitable CUDA/P2P test for your platform. You can see the current and maximum link generation and width with nvidia-smi --query-gpu=index,pcie.link.gen.current,pcie.link.gen.max,pcie.link.width.current,pcie.link.width.max --format=csv; check it under load, because an idle card can drop to a lower PCIe generation to save power.

The practical conclusion: with two PCIe cards (2× P40, 2× V100 PCIe) do not count on a speed-up proportional to the number of cards. You gain capacity (a larger model), not a linear increase in speed. Sometimes two cards are even slightly slower on a small model than one, because communication overhead has been added.

Identical or different cards, and what about mixing

It is simplest to work with two identical cards (2× P40 or 2× V100 of the same version): an even split, predictable behaviour, no surprises with proportions. Then you usually leave --tensor-split equal and that is the end of it.

Mixing cards is possible, but needs care. llama.cpp will cope with cards with different amounts of VRAM, as long as you set the split proportions by hand so as not to overfill the smaller one. Mixing architectures is worse: combining, say, Pascal with Ampere can be a source of strange behaviour (different compute paths, different optimal quantisations). With a layer split the effect depends on the allocation proportions, the characteristics of each card and the communication: the slower card can become the bottleneck if it is given too large a share of the work, but it does not always simply set a global limit. If you are building a setup from scratch, stick to one generation and preferably identical cards: fewer variables, fewer evenings spent debugging.

And one more thing about scaling: going from one card to two is a jump in capacity, but each additional card adds communication overhead. Three and four cards only make sense with really large models; otherwise you add complexity (power, cooling, PCIe, POST: we write separately about POST problems with multi-card rigs) without a proportional gain.

Power and cooling: do not skip this

With two cards, allow for a power limit of up to about 250 W per card plus short-lived power-draw spikes (transients) under sudden load. Do not assume a constant 500 W up front without measuring: that is an upper limit, not continuous draw. That means:

• A strong, good power supply with headroom and a solid 12 V rail, not "exactly as much as the sum of the TDPs". Transients can trip a weak power supply despite an apparently sufficient nominal rating.
• The correct power cable/adaptation for each card. The P40 and the V100 PCIe take a CPU 8-pin (EPS) plug, not a PCIe 8-pin: use a CPU 8-pin cable from the power supply or NVIDIA's adapter fed from two separate PCIe 8-pin cables, and never force a PCIe plug into the card. A cable with the wrong pinout can damage the card. (More in Tesla not detected or no picture: step-by-step diagnosis.)
• Cooling. These are passive cards; two side by side in a desktop can heat each other up and throttle. Without forced airflow a multi-card setup will not last long. I have described how to do it sensibly in the guide to cooling passive server cards.

When 2× P40, and when a single V100 32GB

This is a real purchasing decision, so let us set it out directly.

Criterion	2× P40 (48 GB in total, divided)	1× V100 32GB
Max. model size	largest (two memories together)	up to ~32 GB
Speed on a model that fits in 32 GB	about 3× lower (slower Pascal cards; the split itself costs almost nothing)	higher, simpler
Link between GPUs	PCIe (no NVLink)	not applicable (one card)
Tensor Cores / FP16	no Tensor Cores, FP16 at 1/64 of FP32 (Pascal)	Tensor Cores, fast FP16 (Volta)
Configuration complexity	greater (split, tuning)	minimal
Power/cooling	2× 250 W, harder	one card, easier
When to choose	the model does not fit on one card	the model fits on one card

• Take 2× P40 when you really need a large model that will not fit in the 24/32 GB of a single card, and you accept the PCIe overhead and more tuning. It is the cheapest way to get "a lot of total VRAM for one model".
• Take a single V100 32GB when your model fits in 32 GB: you get Tensor Cores, HBM2, no inter-card overhead and a much simpler setup. For most home and business uses it is the less troublesome choice.

Our measurements (GGUF Q4_K_M, llama-bench build 6ad4ab0, driver 580.173.02, --split-mode layer, -ngl 99, -r 3, full 250 W power limit, cards on PCIe Gen3 without NVLink):

Configuration	Model	Prompt eval (pp512)	Generation (tg128)	Note
1× P40	Qwen2.5-32B Q4	225.3	10.1	reference point
2× P40 layer	Qwen2.5-32B Q4	224.5	10.1	≈ 1 card: the gain is capacity, not speed
2× P40 layer	Llama-3.3-70B Q4	101.3	4.8	70B fits only on 2 cards (~21 GB of weights per card)
1× V100-32	Qwen2.5-32B Q4	643.4	30.3	reference point
2× V100 layer	Qwen2.5-32B Q4	630.2	30.7	≈ 1 card
2× V100 layer	Llama-3.3-70B Q4	317.3	16.0	~3× faster than 2× P40 in this test

With 70B Q4_K_M (42.5 GB) on 2× P40 little room is left for context: with ECC enabled each P40 offers about 22,900 MiB (24.0 GB), so after the weights and the compute buffers roughly 1-2 GB per card remain for the KV cache, enough for about 8K tokens at FP16. For a longer context quantise the cache (-ctk q8_0 -ctv q8_0, with Flash Attention) or choose a smaller quant.

This supports a narrow conclusion about our test: in this backend, for these models and quants, --split-mode layer primarily increases capacity: when the model fitted on one card, the second brought no large throughput gain (the differences were up to about 2.1%). Without standard deviations, we do not assess their statistical significance. With this build's default 512-token microbatch size, pp512 fits in one microbatch and does not show the benefit of pipelining multiple microbatches for longer prompts. I do not extend this to "a layer split always scales only capacity": with a different backend, batching/concurrency or model the behaviour may differ. The point of two cards is to run a model that does not fit on one (like 70B).

The ~3× difference between 2× V100 and 2× P40 (317.3 vs 101.3 pp512 on 70B) is consistent with the advantages of the Volta architecture (Tensor Cores, HBM2, newer kernels), but our benchmark does not isolate the effect of the Tensor Cores alone from the other differences. Treat this figure as a difference between two configurations, not as proof that "the Tensor Cores give 3×". For generation (16.0 vs 4.8 t/s), memory bandwidth is a plausible factor: NVIDIA specifies 900 GB/s for V100 and 346 GB/s for P40, about 2.6×. In this llama.cpp build, quantized weight matrix-vector operations during single-token generation on these GPUs use MMVQ kernels with dp4a instructions rather than Tensor Cores. Other generation operations use different kernels.

What about --split-mode row: in our llama.cpp build (6ad4ab0) --split-mode row ends in a model-loading error (device does not support split buffers). This has nothing to do with our topology or the lack of NVLink: llama.cpp removed row split from its CUDA backend in July 2026 (PR #24216), because the experimental --split-mode tensor replaces it, so a current build shows the same error on every NVIDIA setup. Older builds still offer row split, and on P40s it was the mode people used to speed up generation. The mode used for the benchmark was layer. If you want the cards to compute in parallel, test --split-mode tensor on your build. It requires Flash Attention. Build 6ad4ab0 does not support automatic memory fitting for this mode, so setting the context size manually is advisable; otherwise the model's full training context may be used. Check supported KV cache types too: PR #19378 lists quantized cache as unsupported. Support and requirements change between commits, so check your llama.cpp version and topology (nvidia-smi topo -m). Your figures will depend on the model, the quantisation, PCIe and the context length.

Sources

• llama.cpp: the --split-mode {none,layer,row,tensor} and --tensor-split options: llama-cli --help / llama-bench --help for the commit used (build 6ad4ab0); repository: https://github.com/ggml-org/llama.cpp
• llama.cpp PR #24216 "CUDA: remove -sm row, refactor cuBLAS": https://github.com/ggml-org/llama.cpp/pull/24216
• llama.cpp PR #19378 (experimental tensor parallelism, -sm tensor): https://github.com/ggml-org/llama.cpp/pull/19378
• llama.cpp PR #6017 (pipeline parallelism with layer split): https://github.com/ggml-org/llama.cpp/pull/6017
• nvidia-smi documentation (topology legend, link fields): https://docs.nvidia.com/deploy/nvidia-smi/index.html
• Ollama FAQ (multi-GPU loading): https://docs.ollama.com/faq
• NVIDIA Tesla V100: technical data (32 GB HBM2, PCIe/SXM2 variants, NVLink): https://images.nvidia.com/content/technologies/volta/pdf/tesla-volta-v100-datasheet-letter-fnl-web.pdf

---

A disclaimer at the end, as always: yes, we sell these cards, in pairs too, so read this with that in mind. But I do not want to sell you two cards if a single V100 32GB does the job more simply and faster: that is honestly cheaper to run and less troublesome. We put every unit through a multi-stage qualification (technical fingerprint, VRAM check, burn-in with a watchdog, ECC, PCIe, a check of computational correctness) and include a report/certificate from this process, and with two-card setups we will help you choose the power supply and settings. Wondering whether your model needs two cards or will fit on one? Write to us with the model name and quantisation: we will work out the VRAM and advise on the configuration.

(See also: Cooling passive server cards)