Is PLLuM free for commercial use?

Usually yes, subject to the specific weights' licence terms: 12B is released under Apache 2.0 and the Llama-based variants under the Llama 3.1 licence. The 4B model card declares Apache 2.0, but the Gemma Terms of Use for model derivatives must also be considered. But it depends on the specific version: the family also includes older -nc (non-commercial) weights. Before deploying anything commercially, check the licence for exactly those weights on Hugging Face.

The smallest card for PLLuM?

It depends on the model size. The 4B variant fits on a Tesla P4 8GB even in Q80 (about 4.1 GB), and the 8B in Q4KM (about 4.9 GB) or Q5KM (about 5.7 GB), but on 8 GB keep the context moderate, because the KV cache needs room too. For larger models, choose memory capacity according to the specific quantisation and context length: PLLuM-12B can also run on a P100 16GB, while a P40 24GB or V100 32GB provides more memory headroom.

Does PLLuM work on 2× P40?

Yes: llama.cpp and Ollama can split a large model across two cards. 2× P40 24GB gives ~48 GB of total VRAM and may allow 70B-class models in Q4, depending on the specific GGUF, the context length, the split settings and the headroom for the KV cache. Remember: two cards are not one 48 GB card: with the default layer split they work one after the other, so a 70B model in Q4 generates much more slowly than a model that fits on a single card.

GDPR: is a local model enough for compliance?

Local processing helps a great deal (the data does not leave your infrastructure), but compliance also means procedures, access control and documentation. The model is one piece of the puzzle, not the whole: consult your own data protection officer (DPO) or lawyer on compliance decisions.

Why does the prompt "think" for a moment before the answer on the P40?

That is prefill: on the P40 the older Pascal architecture and the lack of Tensor Cores translate into clearly lower performance in processing a long prompt than on the V100. Shorten the context or consider a V100.

← Blog

PLLuM locally on a used Tesla - how to run it and how much VRAM

Agile Store · 20 August 2026 · 12 min read

model:v100-32
model:v100-16
model:p40-24
model:p100-16
arch:pascal
arch:volta

The short answer: you can run PLLuM (the Polish Large Language Model) locally on a used Tesla just like any other model in GGUF format, through Ollama or llama.cpp. The key question is not "whether", but "which model size" and "how much VRAM your card has". The Tesla P40 24GB is the budget choice with plenty of memory headroom; the V100 16GB or V100 32GB gives more power with a longer context. The data does not have to leave your infrastructure, which is the main reason Polish public administration and businesses want this locally.

I write this from the perspective of someone who gets questions like this every day: "I have sensitive customer data / official documents, I cannot send them to OpenAI's cloud: can I run a Polish model on my own premises?". You can. And you do not need a card costing well over PLN 10,000 to do it. Below: the specifics, without beating about the bush, with an honest indication of what I am not sure about.

What PLLuM is and why keep it local

PLLuM is the Polish Large Language Model: a family of language models developed by a consortium of Polish institutions led by NASK National Research Institute. The original consortium included Wrocław University of Science and Technology, the Institute of Computer Science of the Polish Academy of Sciences, the National Information Processing Institute (OPI PIB), the University of Łódź and the Institute of Slavic Studies of the Polish Academy of Sciences; in the follow-up HIVE AI project it was joined by the Central Informatics Centre (COI) and AGH University of Krakow (computing power from ACK CYFRONET AGH), and COI is deploying the models in the mObywatel app (the Polish government's citizen app). The idea is simple: a model trained with the Polish language and the Polish cultural and administrative context in mind, with weights published openly on Hugging Face (the CYFRAGOVPL organisation) in 4B, 8B, 12B, 8x7B (mixture of experts) and ~70B sizes, depending on the release, and, depending on the model, in base, instruct and chat variants.

One important technical note, because a shorthand is circulating online: the official PLLuM repositories publish the weights in safetensors format, not GGUF. The GGUF files (the ones used by Ollama and llama.cpp) are community quantisations, made and published on Hugging Face by independent authors (for example mradermacher, who publishes every size of the current release, and tensorblock). It is still PLLuM, just repackaged in a format convenient for llama.cpp: you simply download the GGUF from a community repository, not from the CYFRAGOVPL account.

Why locally rather than in the cloud?

GDPR and data sovereignty. Personal data, official documents, medical or legal data: with local processing you do not have to send them to an external API provider. This makes control over the data much easier, but running the model locally does not in itself mean automatic GDPR compliance (more on that in the FAQ). For public offices and many businesses, not sending data outside is a necessary condition, not a convenience.
No per-token fee. You do not pay for every request to an external API: the cost shifts to your own hardware, energy, cooling and maintenance. At high volumes this adds up.
Offline and in control. The model works without the internet, you freeze the version, and nobody "updates" it without your say-so.
The Polish language. PLLuM (like Bielik) is developed with an emphasis on Polish and the Polish context, so it can be particularly interesting for tasks in Polish. Whether it will do better in your case than a particular general-purpose model depends on the application, so verify it on your own tasks rather than trusting general rankings.

How much VRAM you need

There is no magic, just arithmetic. The VRAM requirement depends on two things: the number of model parameters and the number of bits per parameter (quantisation).

The rule of thumb: VRAM ≈ number of parameters × bits per parameter / 8, plus overhead for the context (KV cache) and the runtime itself. Quantisation is a way of "compressing" the weights:

Q8_0 (~8.5 bits/param): close to the original, the most VRAM.
Q5_K_M (~5.7 bits): good quality, a reasonable size.
Q4_K_M (~4.9 bits): the most common compromise; the model is clearly smaller, and the quality is still usable.

Sizes of the community GGUF files for PLLuM (mradermacher, as listed on Hugging Face; the weights alone, add headroom for the context/KV cache). The table shows above all whether the weights alone will fit on a given card, not guaranteed comfort when working with a long context:

Model	Q4_K_M	Q5_K_M	Q8_0	Card (fits the weights)
PLLuM-4B (2512)	2.5 GB	2.8 GB	4.1 GB	any card, including the P4 8GB
Llama-PLLuM-8B (2512)	4.9 GB	5.7 GB	8.5 GB	P4 8GB (Q4/Q5), P40 / P100 / V100
PLLuM-12B (2512)	7.5 GB	8.7 GB	13.0 GB	P40 24GB, P100 / V100 16GB
PLLuM-8x7B (2412, MoE)	28.5 GB	33.2 GB	49.6 GB	V100 32GB (Q4) or 2× P40 24GB
Llama-PLLuM-70B (2512)	42.5 GB	50.0 GB	75.0 GB	2× P40 24GB (Q4 only)*

* For 70B what matters is the total VRAM, not the number of cards: 2× P40 24GB = 48 GB is enough for the weights in Q4 (about 42.5 GB), but 2× V100 16GB = 32 GB is not enough. If you are building this with V100s, aim for a configuration with a sufficiently large total memory (for example, 2× V100 32GB).

Other quantisers' files differ slightly: check the exact size of the GGUF you download (Hugging Face shows it next to each file). For a concrete example: Llama-PLLuM-8B-instruct in community quantisations weighs ~4.9 GB in Q4_K_M, ~5.7 GB in Q5_K_M and ~8.5 GB in Q8_0 (as of August 2026), so the 8-billion-parameter PLLuM fits with room to spare on any of the cards described. Add the KV cache on top: the longer the context, the more memory it consumes. I have the full breakdown of "how much VRAM for which model" here: how much VRAM for LLaMA 7B/13B/70B.

Which card for PLLuM

Tesla P40 24GB: budget and capacity. 24 GB of GDDR5 is the most VRAM for the least money. 13B in Q4 fits with plenty of headroom for the context; ~30B in Q4 will also fit, but then the margin for a long context is much smaller. A drawback I repeat honestly: the P40 has no Tensor Cores, and the architectural differences from Pascal to Volta translate in practice into much weaker prefill (processing a long prompt). For working with shorter prompts and a single user, it is a very sensible choice.
V100 16GB / V100 32GB: more compute power. HBM2 (high bandwidth) plus first-generation Tensor Cores. The V100 gives higher prefill performance and copes better with serving many requests; the 32 GB version additionally allows larger models and a longer context without combining cards. Note: 16 GB is less memory than the P40's 24 GB: on the context margin alone, the V100 16GB does not beat the P40. You pay more and get a shorter "time to first token".
P100 16GB: a price compromise. HBM2 gives it twice the P40's memory bandwidth, but in our measurements it was slower than the P40 in prefill and no faster in generation: llama.cpp's fast quantised kernels need the dp4a instruction, which the P40 (compute capability 6.1) has and the P100 (6.0) does not; in addition, 16 GB limits larger models.

To be honest about the technicalities: the Tesla V100 and P100 do have NVENC (a video encoder), according to NVIDIA's official matrix, but for this compute task the video encoder does not matter anyway; the card without NVENC is the A100. I mention this so that nobody buys these cards "for transcoding on the side": for LLMs the encoder makes no difference. And one more thing people forget: these cards use an 8-pin CPU/EPS power connector (not PCIe): before connecting, check the required cable for the specific variant of the card and server; do not use a random cable just because it fits mechanically. That is a separate topic, but a real cost.

How to run it: Ollama or llama.cpp

Before anything else, two things that are specific to these cards. Install an NVIDIA driver from the 580 branch and pin it: it is the last branch that supports Pascal and Volta, and Ollama's Linux installer installs the newest driver (590+, without Pascal) if it finds none. And if you use llama.cpp's prebuilt binaries, take the CUDA 12 build: the CUDA 13 builds do not run on the P40, P100 or V100. Step by step for the P40: Tesla P40 with Ollama.

The simplest way is through Ollama, if you have a PLLuM GGUF file. You create a Modelfile:

FROM ./pllum-<size>-Q4_K_M.gguf

# optional: prompt template and parameters
PARAMETER num_ctx 4096
PARAMETER temperature 0.6

Then:

ollama create pllum -f Modelfile
ollama run pllum

Note: choose num_ctx according to how much VRAM you have left after loading the weights: too large a context can exhaust the VRAM. Depending on the runtime and settings, the result is different: Ollama keeps fewer layers on the GPU and runs the rest on the CPU, which can reduce performance dramatically; llama.cpp with an explicit -ngl 999 and -c will usually stop with an out-of-memory error at load time, and only when you leave those values unset can its automatic fitting lower the context and the number of GPU layers for you. For the original instruct and chat weights in release 2512, the chat template is in the repository's chat_template.jinja file; for PLLuM-8x7B release 2412, it is in the chat_template field of tokenizer_config.json. These variants, including the Llama-based ones, use [INST] ... [/INST] formatting rather than the Llama 3.1 header format. Good GGUF files carry the template in their metadata. llama.cpp can read it, while Ollama attempts to recognise it during import; inspect the result with ollama show pllum --template and compare it with the specific model's template. If the GGUF lacks a template or Ollama does not recognise it correctly, write a TEMPLATE in Ollama's Go-template syntax that reproduces the format in the model's chat_template.jinja or the chat_template field of its tokenizer_config.json.

Alternatively, llama.cpp directly:

./llama-server -m pllum-<size>-Q4_K_M.gguf -ngl 999 -c 4096

-ngl 999 puts all layers on the GPU (make sure the model fits), and -c is the context length. You get an OpenAI-compatible endpoint to which you can connect your own application. Slots configured with --parallel serve several users concurrently. In current llama-server versions, automatic mode creates four slots with a shared KV cache pool; with the shared pool disabled, the -c context is divided among slots. Size the total context and VRAM headroom for the number of concurrent requests. The current NVIDIA backend in vLLM requires compute capability 7.5 or newer, so it does not support the cards discussed here.

The whole procedure is identical to Bielik: if you have already set that up, PLLuM will take you five minutes. I show it step by step, with measured speeds, here: Bielik and PLLuM benchmarks on the P40/P100/V100; on the P40, PLLuM-12B in Q4_K_M processed about 612 prompt tokens/s and generated about 27 tokens/s.

Bielik vs PLLuM: which to use

Both families are Polish, and both are worth having locally. There is no single right answer: it depends on the task.

Bielik: mature, popular in the community, proven in everyday tasks in Polish.
PLLuM: built by a consortium that includes public institutions, with an emphasis on the Polish administrative context (it has found its way into the mObywatel app, among others). The licence depends on the specific weights: the 12B (Mistral-Nemo base) and ordinary 8x7B variants without -nc are released under Apache 2.0. The Gemma 3-based 4B model card declares Apache 2.0, but the Gemma Terms of Use for model derivatives must also be considered, including use restrictions and distribution obligations. The Llama-based 8B and 70B variants use the Llama 3.1 licence, which permits commercial use subject to conditions including attribution and naming. Section 2 requires a separate licence from Meta in specified cases involving more than 700 million monthly active users in the month preceding the Llama 3.1 release date. Note: the family also includes older variants with the suffix -nc (non-commercial), which cannot be used commercially, so check the licence for the specific weights you download.

My honest advice: download both in GGUF, run them on the same card and test them on YOUR OWN prompts. General benchmarks will not tell you which one is better at summarising your contracts or answering in your tone. It takes two evenings, and you will make the decision on data, not on someone else's opinion.

Sources

PLLuM details verified on 2026-08-11 against official sources (confidence: confirmed):

PLLuM: project website (composition of the consortium, NASK as leader): https://pllum.org.pl/projects
NASK: "New language models are now available. The PLLuM family is growing" (development of the family, partners): https://www.nask.pl/aktualnosci/nowe-modele-jezykowe-sa-juz-dostepne-rodzina-pllum-sie-powieksza
PLLuM: CYFRAGOVPL (weights and variants on Hugging Face: 4B/8B/12B/~70B, base/instruct/chat, safetensors format): https://huggingface.co/CYFRAGOVPL
CYFRAGOVPL/Llama-PLLuM-8B-instruct-2512 (Llama-3.1-8B base, Llama 3.1 licence, [INST] chat template in chat_template.jinja): https://huggingface.co/CYFRAGOVPL/Llama-PLLuM-8B-instruct-2512
CYFRAGOVPL/PLLuM-12B-instruct-2512 (Mistral-Nemo-Base-2407 base, Apache-2.0 licence): https://huggingface.co/CYFRAGOVPL/PLLuM-12B-instruct-2512
mradermacher/Llama-PLLuM-8B-instruct-2512-GGUF (community GGUF of the 2512 release; Q4_K_M 4.92 GB, Q5_K_M 5.73 GB, Q8_0 8.54 GB): https://huggingface.co/mradermacher/Llama-PLLuM-8B-instruct-2512-GGUF
NVIDIA Video Encode/Decode GPU Support Matrix (the P100 and V100 have NVENC; A100 = 0): https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new
NVIDIA Tesla V100 datasheet (HBM2, Tensor Cores): https://images.nvidia.com/content/technologies/volta/pdf/tesla-volta-v100-datasheet-letter-fnl-web.pdf

Yes, we sell these cards (P40 24GB, V100 16GB, V100 32GB, P100 16GB, Tesla P4 8GB), so I have an interest in you buying one. That is why I prefer to speak plainly, including about the drawbacks (the P40's slow prefill, EPS power) and the fact that the NVENC video encoder does not matter for LLMs, rather than sell you a disappointment. Every unit goes through a burn-in test and receives a certificate. Not sure which PLLuM size and which card suit your case? Write to us and we will help you choose.

FAQ

Is PLLuM free for commercial use?: Usually yes, subject to the specific weights' licence terms: 12B is released under Apache 2.0 and the Llama-based variants under the Llama 3.1 licence. The 4B model card declares Apache 2.0, but the Gemma Terms of Use for model derivatives must also be considered. But it depends on the specific version: the family also includes older -nc (non-commercial) weights. Before deploying anything commercially, check the licence for exactly those weights on Hugging Face.
The smallest card for PLLuM?: It depends on the model size. The 4B variant fits on a Tesla P4 8GB even in Q8_0 (about 4.1 GB), and the 8B in Q4_K_M (about 4.9 GB) or Q5_K_M (about 5.7 GB), but on 8 GB keep the context moderate, because the KV cache needs room too. For larger models, choose memory capacity according to the specific quantisation and context length: PLLuM-12B can also run on a P100 16GB, while a P40 24GB or V100 32GB provides more memory headroom.
Does PLLuM work on 2× P40?: Yes: llama.cpp and Ollama can split a large model across two cards. 2× P40 24GB gives ~48 GB of total VRAM and may allow 70B-class models in Q4, depending on the specific GGUF, the context length, the split settings and the headroom for the KV cache. Remember: two cards are not one 48 GB card: with the default layer split they work one after the other, so a 70B model in Q4 generates much more slowly than a model that fits on a single card.
GDPR: is a local model enough for compliance?: Local processing helps a great deal (the data does not leave your infrastructure), but compliance also means procedures, access control and documentation. The model is one piece of the puzzle, not the whole: consult your own data protection officer (DPO) or lawyer on compliance decisions.
Why does the prompt "think" for a moment before the answer on the P40?: That is prefill: on the P40 the older Pascal architecture and the lack of Tensor Cores translate into clearly lower performance in processing a long prompt than on the V100. Shorten the context or consider a V100.