RAG on company documents offline - what hardware and how to start (GDPR)

Short answer: RAG is a way to make a language model answer questions on the basis of your documents, and to do it locally when the documents contain data that the company does not want to, or may not, release to external services. (GDPR is sometimes one of the reasons for such a decision, but using the cloud is not automatically non-compliant with GDPR; I will come back to that.) For the embeddings alone a light card is enough: a Tesla P4 8GB or an NVIDIA A2 16GB. The LLM that generates the answers does the heavy lifting: here you want a lot of VRAM, so a V100 32GB or a Tesla P40 24GB. You can start with a single 24 GB card and Ollama, and expand later. And yes, we sell these cards; full disclaimer at the end.

I am writing this mainly for companies, law firms and offices that have a real problem: hundreds or thousands of documents in which nobody can find anything quickly any more, and not all of which the company wants to, or may, pass to an external API. This is not an "AI will change your business" guide: it explains in practical terms how to build a search tool that answers questions using your own files, so that the data does not leave the server room.

What RAG is (without the marketing)

RAG stands for Retrieval-Augmented Generation. In practice it works like this: you ask a question, the system first finds the fragments of your documents that match the question, and then puts them into the LLM together with the instruction "answer on the basis of these fragments". The result: the model answers with your knowledge (the content of contracts, procedures, minutes, correspondence), not with what it "saw somewhere on the internet".

Why is this better than just questioning the model? Because an LLM on its own makes things up (hallucinates), especially about things specific to your company that are not in its knowledge. RAG gives the model specific fragments as context and can enforce showing the source ("this is in contract X, clause 4"), which for a lawyer or an accountant is the difference between a toy and a working tool. The mechanism itself does not, however, guarantee that the model always stays within the bounds of the fragments it was given (more on that in the FAQ).

Why offline: GDPR, confidentiality and control over data

This is not always a "nice to have". For some companies local processing is an important requirement for legal, security or confidentiality reasons:

• Personal data: including, in some cases, special categories of data (e.g. medical records) and material covered by professional secrecy (advocates', legal advisers'). Sending such data to an external API means entrusting its processing to an external provider and requires that relationship to be properly regulated. If the data leaves the EEA, the requirements for transfers to a third country also apply.
• Trade secrets: contracts, quotes, strategies. You do not want them on someone else's servers, even "only for processing".
• Sovereignty and predictability: it works without the internet, without depending on someone else's price list and service availability.

Offline RAG means: the model, the vector database and the documents live on your hardware. In a correctly configured local environment the content of the documents is not passed to an external API, which removes the need to entrust this particular operation to an external provider. A caveat: "local" does not mean "automatically GDPR-compliant": the application itself may send telemetry, download models or back up externally, so the environment has to be deliberately cut off and supplemented with access control, logging and backups (more on that below).

Architecture: what it is made of

The heart of RAG is three building blocks:

1. Embeddings: a model turns every fragment of a document (and your question) into a vector of numbers that represents its "meaning". Texts with similar meanings have similar vectors. This is a light, fast step.
2. A vector database: it stores these vectors and instantly finds the ones most similar to the question. Popular, free and runnable locally: Qdrant, Chroma, Milvus. Chroma is well suited to rapid prototyping, and Qdrant is a frequent choice for server deployments that need persistent storage, filtering and more operational control.
3. The LLM: it receives the fragments found + the question and formulates the answer in Polish. It is usually the most computationally demanding element and the one that most determines the VRAM requirements, but what counts is not only the size of the model, but also its quantisation, the context length, the KV cache and the number of parallel requests.

The flow in one sentence: document → embedding → vector database → (on a question) retrieval of fragments → LLM → answer with citations.

What hardware, and why we split the tasks

The key observation that saves money: embeddings and the LLM have completely different appetites.

Embeddings are light. Embedding models (e.g. from the multilingual/E5 or BGE families) have hundreds of millions of parameters, not billions. They compute quickly and fit in a few GB. For this a Tesla P4 8GB is entirely sufficient (8 GB, up to 75 W, low-profile: good as an inexpensive, energy-efficient "indexer", provided there is adequate airflow through the card's heatsink; it is a passive card), or an NVIDIA A2 16GB if you want a newer architecture and headroom. Do not overpay for a lot of VRAM here: this is not the computationally demanding part.

The LLM does the heavy lifting. Do not choose the model solely by its number of parameters: start with a modern model that handles Polish well (of the order of a few to a dozen or so billion parameters) and test it on your own documents; a larger one is sometimes better, but it increases the VRAM requirements. A longer context also matters, because RAG puts the fragments found into the prompt (that can be several thousand tokens). The context eats VRAM on top of the model's weights. That is why here you aim for maximum memory:

• V100 32GB: if the priority is plenty of spare VRAM on a reasonable budget and you have a suitable server platform, this is one of the strongest options in our range for the LLM in RAG. Volta with Tensor Cores + 32 GB of HBM2 = headroom for a larger model, a long context and fast batch processing of fragments. A caveat: an older architecture, it draws a lot of power and needs a platform and cooling suited to a server card.
• Tesla P40 24GB: a budget way to get 24 GB of VRAM. In Ollama/llama.cpp you will use it for inference of quantised GGUF models; slower FP16 (it is Pascal), so stick to quantisation: with this generation the Q4/Q5 variants make particular sense.

In practice a common and sensible layout is one 24/32 GB card for the LLM + possibly an inexpensive P4/A2 for the embeddings (or the embeddings on the same card, if memory is left over). How much VRAM exactly for which model: I have set that out in how much VRAM for Llama 7B/13B/70B. If a Polish model matters to you, see also Bielik locally on a used Tesla: it is one of the models worth testing for Polish-language RAG.

A note on the driver: Ollama currently supports the P4, P40 and V100, but needs NVIDIA driver 570 or newer for Pascal cards (P4/P40, compute capability 6.1) and 550 or newer for the V100 (Volta, 7.0). Do not go above the 580 branch: R580 is the last driver branch for Pascal and Volta, and 590+ no longer supports these cards. Install and pin the 580 branch with proprietary kernel modules before installing Ollama. On supported Linux distributions, Ollama's installer may install the latest driver branch if it cannot find the nvidia-smi command; the presence of that command alone does not guarantee a working driver. If you have an older installation (Ubuntu/Proxmox/Debian with an older data-centre driver), the system may detect the card correctly, but the current version of Ollama may require a newer driver: check the current requirements before building the server, because they change from time to time.

What our measurements showed

To avoid staying with generalities, we tested the individual elements of a local RAG pipeline based on Ollama + Chroma + LlamaIndex: indexing, retrieval, LLM generation, VRAM usage and behaviour under parallel requests. We ran the tests on one Polish corpus, scifact-pl (a Polish translation of the SciFact benchmark: about 5,200 biomedical abstracts; 100 queries with relevance judgements), with the Bielik 11B model and e5-large/bge-m3 embeddings, on cards set to the full power limit and with ECC; I give the detailed scope of each measurement with the results. The corpus contains scientific text, not contracts or procedures. This is one corpus and a limited set of models: treat the figures as a reference point, not an oracle for every document collection.

Measurement	P4	A2	P40	V100
Indexing ~5,000 fragments	~14 min	~11 min	~6 min	~2 min
Bielik 11B Q4 generation (output tokens/s)	n/a	n/a	~28	~74–80
Full answer (e2e), 1 user	n/a	n/a	~10 s	~3.5 s

n/a: no result in this table. In our tests the P4 and A2 served as embedding cards; the P4 can run Bielik 11B Q4 on its own, but not together with the embedder (see below).

The GPU affected the compute time, not retrieval recall: search quality depends on the embedding model. On the scifact-pl corpus:

Retrieval (scifact-pl)	Recall@10
e5-large (native chunking)	0.83
bge-m3 (native chunking)	0.75
bge-m3, ~256-token chunks (separate experiment)	0.82

Embeddings really are light. Indexing ~5,000 fragments took ~2 min on the V100 and ~6 min on the P40; the A2 ~11 min, and the P4 ~14 min. Writing the vectors to the database took only ~7 s regardless of the card, while the indexing time clearly depended on the card (from ~2 min on the V100 to ~14 min on the P4), so the indexing time was driven mainly by the embedding computation on the GPU. A stronger card clearly shortens indexing, but at this volume even the P4/A2 finish the job in eleven to fourteen minutes, and since this is usually a periodic batch job in the background, it need not justify buying a powerful GPU just for embeddings.

The LLM is a different world. Answer generation (Bielik 11B Q4, 512-token context) reached ~28 output tokens/s on the P40 and ~74–80 on the V100: in LLM generation alone, the V100 delivered about 2.6–2.9× higher throughput than the P40 in these tests (depending on the unit). In practice a full answer (retrieval + generation) took ~3.5 s on the V100 against ~10 s on the P40 for a single user; the retrieval itself took ~30 ms, so in this configuration the LLM generated the vast majority of the latency. With prompts of several thousand tokens, prompt processing adds noticeably more time on the P40 than on the V100. You can also see why it is worth leaving spare VRAM beyond the model weights alone: increasing the context to 8,000 tokens raised Q4 usage from ~6.7–7 to ~8.7 GB, and Q8, i.e. less aggressive quantisation, already needed ~12–13.5 GB.

8 GB was not enough for our full pipeline on a single card. In our configuration the Tesla P4 8GB computed the embeddings and, separately, ran the LLM alone, but Bielik 11B Q4 and the embedder together did not fit in 8 GB of VRAM (with ECC on, as in our tests, about 7.5 GiB remains after ECC overhead, before other reservations). This is not a defect of the card, but the capacity limit of this particular configuration: a smaller model, CPU offload or embeddings computed on the CPU may change the result. For what we tested, 16 GB was already a much safer starting point, and the P4 remains an excellent inexpensive indexer.

Search quality depends on the embedding model, not on the card. On our corpus, with each model's default chunking in our configuration, e5-large did better than bge-m3: recall@10 of 0.83 vs 0.75 (in simplified terms: how often the right fragment made it into the search engine's top ten results). In a separate experiment with bge-m3, reducing the fragment size to ~256 tokens raised recall@10 from 0.75 to 0.82, at the cost of more units to index. This was practically a tie with e5-large, so the chunking mattered as much as the model. This is domain-dependent: on your documents the order may come out differently. What our result does show is that, for retrieval quality, it is worth tuning above all the embedding model and the way documents are chunked; a stronger card shortens the computation, but by itself does not improve the recall of the same pipeline.

Concurrency: in our setup the queue grew, not the throughput. For the configuration tested (Bielik 11B Q4, a single instance), adding parallel requests did not increase throughput: what grew above all was the queue, and the p95 time lengthened with the number of requests. This is a result for a specific model and backend configuration, not a universal property of LLM inference: by default Ollama processes one request per model at a time (OLLAMA_NUM_PARALLEL defaults to 1) and queues the rest. Raising OLLAMA_NUM_PARALLEL lets requests share the card, at the cost of VRAM (the context is multiplied by the number of parallel requests); servers built for batching, such as vLLM, behave differently, but the current CUDA build of vLLM needs compute capability 7.5+, which excludes the P4, P40 and V100. If you need predictable latency for many users, test concurrency on the target stack; one way to scale is to run additional instances or cards.

How to start: the simplest route

You do not need an ML team to set up your first working RAG. The minimal stack:

1. Ollama as the LLM server: ollama run and you have a local model with an API. For Polish, use Bielik or another multilingual model.
2. A vector database: Chroma (the simplest, pip install chromadb) or Qdrant in Docker, if you are thinking about production.
3. A framework to tie it together: LangChain or LlamaIndex. Both have ready-made building blocks: load PDFs/DOCX files, chunk them, compute embeddings, put them into the database, and on a question, retrieve and query the LLM. You can build a first proof of concept in a few dozen lines of code; a production deployment is a bigger project: permissions, index updates, OCR, monitoring and quality control of the answers come on top.

The starting scheme: load the documents → chunk → index (embeddings into the database) → question loop (retrieve fragments → pass to Ollama → show the answer). Start with a small collection (one department, one type of document), check the quality of the answers, and only then scale.

Cost: your own hardware vs the cloud

Honestly, without stretching things. The cloud wins when you use RAG occasionally, you do not process confidential documents and you do not want to administer anything: you pay per query and that is it. Your own hardware is worth considering especially when at least one of these conditions is met: security policy, confidentiality or the organisation's requirements impose local processing, the load is constant (an assistant used every day by the team), or you want a predictable cost without a token meter.

But honestly: the break-even point depends on what you are comparing with, and it does not always come out in favour of your own hardware.

• Compared with renting a dedicated card in the cloud 24/7 (e.g. an RTX 4090 from ~$0.31–0.35/hour on marketplace/community platforms: dedicated instances cost more; ≈ ~PLN 1,000/month at 24/7 operation, as of August 2026), a used P40-based build for ~PLN 2,900–3,900 corresponds to roughly 3–4 months of renting such a card in the cloud (a V100 32GB build costs more; and the rented RTX 4090 is considerably faster than a P40, so this compares cost, not equal performance). Here "months, not years" really is true: just remember that the full cost of a local server also includes electricity, the hardware beyond the GPU and administration.
• Compared with an API billed per token, things look different: the cheapest modern models today cost fractions of a dollar per million input tokens, but more advanced models cost many times more, and output tokens are usually much more expensive than input tokens (as of August 2026). At moderate volume (hundreds of queries a day), the API alone is often cheaper in pure token terms than amortising your own card. In that case on-premises is justified by requirements for confidentiality, compliance and control over data, and by predictability, not by price alone: a purely cost-based threshold against the API only appears with heavy, constant traffic.

In other words: if legal, contractual or internal requirements rule out passing documents to an external provider, the comparison with a public API stops being purely a cost decision, and that is often a stronger argument than the cost of tokens alone. Work out your volume and compare it with the current price list, instead of believing the slogan "self-hosting is always cheaper".

Add to the bill what the cloud price list does not show: independence. An API price list can change, a model can be withdrawn or a service can be unavailable at a key moment. Your own infrastructure gives you more control over the environment and does not depend directly on changes to an API provider's prices, limits or decisions; in exchange, the company is responsible for its maintenance, updates and failures. For a company that builds a process around a document assistant, this predictability can be worth more than the cost of tokens alone.

Common mistakes at the start

Before you order hardware, it is worth knowing the typical pitfalls that spoil RAG quality, and they usually have nothing to do with the card:

• Poor document chunking: fragments that are too large dilute relevance, ones that are too small lose context. In our experience it is one of the most common causes of weak answers, more often than a model that is too small.
• No metadata and sources: if you do not record which document a fragment comes from, you lose RAG's biggest asset: citation. Take care of it from the start.
• A poorly chosen LLM: do not choose it solely by size or by how much VRAM is left on the card. Check the quality on representative questions from your own documents; a larger model (and 24/32 GB for it) only makes sense when it gives measurably better answers.
• The wrong embedding model for Polish: use a multilingual model that handles Polish well, otherwise retrieval will miss relevant results even with a good LLM.
• No access control in the retrieval layer: local RAG does not mean every user should see all the documents. Permissions have to work before an answer is given: the retriever should not even return fragments of documents the person asking has no access to, otherwise someone from sales will pull HR files out with a simple question.
• Assuming every PDF contains text: many company archives are scans. If a PDF is only an image, you need OCR before indexing; without it there is no text to chunk, embed and search.

Sources

• multilingual-e5-large (multilingual embedding model): https://huggingface.co/intfloat/multilingual-e5-large
• BAAI/bge-m3 (multilingual embedding model): https://huggingface.co/BAAI/bge-m3
• Qdrant (vector database): https://github.com/qdrant/qdrant
• Chroma (vector database): https://github.com/chroma-core/chroma
• Milvus (vector database): https://github.com/milvus-io/milvus
• Ollama (local LLM server): https://github.com/ollama/ollama
• LangChain (RAG framework): https://python.langchain.com/
• LlamaIndex (RAG framework): https://docs.llamaindex.ai/
• Ollama: supported hardware (compute capability and required NVIDIA drivers): https://docs.ollama.com/gpu
• LLM model API pricing (per 1M tokens; input floor of the order of $0.20, advanced models many times more expensive, output usually several times more expensive than input; as of August 2026): CloudZero, "LLM API Pricing Comparison 2026": https://www.cloudzero.com/blog/llm-api-pricing-comparison/
• Cloud GPU rental pricing (RTX 4090 ~$0.31–0.35/hour marketplace/community, dedicated more expensive; as of August 2026): Spheron, "GPU Cloud Pricing Comparison 2026": https://www.spheron.network/blog/gpu-cloud-pricing-comparison-2026/
• Tesla P40 24GB / V100: our range (basis for the build cost): https://store.agilesoftware.pl/en/products/p40-24
• Our own measurements (Agile Store, September 2026): the Ollama + Chroma + LlamaIndex stack; the scifact-pl corpus (Polish): a fixed sample of the first 100 queries from the test qrels by ascending id; e5-large and bge-m3 embeddings; the Bielik 11B model in Q4_K_M / Q5_K_M / Q8_0 variants; cards at the full power limit, ECC on. The figures in the "What our measurements showed" section come from this benchmark (raw data internal).

---

An honest disclaimer: we sell used data-centre cards, so I have an interest in this, but offline RAG makes sense regardless of where you buy the hardware; if your volume is small and the documents are not confidential, sometimes the cloud is simply the better choice, and I will tell you that too. This is used hardware: before sale every unit goes through our six-stage qualification procedure (technical fingerprint, VRAM test, burn-in with a watchdog, ECC, PCIe and a check of computational correctness), and we document the test results of the specific unit in a certificate: ask for the certificate of the specific card. Want to talk through an architecture for your documents and budget? write to us: we will help you choose a card for the LLM and a separate one for the embeddings, or advise against buying if it does not add up in your case.