The short answer before I go into detail: if you are planning real fine-tuning, even with LoRA, and compute throughput on large batches matters, go for the V100, ideally the 32GB version. If you are building an economical server on which fine-tuning is an addition to inference, and low power consumption and a modern stack (BF16, TF32) are the priority, the A2 will be the better choice. Below I explain why, and where this rule breaks down.
I write this from hands-on experience with both cards, not from a manufacturer's marketing brochure; the numbers below are NVIDIA's specifications, so read them as ceilings, not measurements. So let me say straight away: this is not an article in which one card wins at everything. Both have sensible uses, just different ones.
Why compare Volta with Ampere at all
The V100 and A2 are two architectures apart (Volta, then Turing, then Ampere), and in practice they belong to two different worlds of application. The V100 (GV100) is a card designed from the ground up for training: full HBM2, considerable power draw, first-generation Tensor Cores. The A2 (GA107) is an inference/edge card: low profile, a few dozen watts, more modern third-generation Tensor Cores. The fact that both now reach the secondary market at similar prices does not mean they are interchangeable. I am leaving out Pascal-architecture cards here: they are in a completely different league and are a separate topic.
What really matters for fine-tuning
Before I compare the specifications, it is worth establishing what "a good card for fine-tuning a model" actually means. Four things matter:
VRAM. This is the first barrier to entry: you have to fit the model weights, the gradients, the optimiser state (Adam keeps two extra values per parameter, running averages of the gradient and of its square, usually in FP32) and the activations. With full fine-tuning of even a medium-sized model, VRAM disappears in no time.
Memory bandwidth. Training includes large matrix multiplications, which can be limited by Tensor Core throughput at suitable sizes, and memory-heavy operations such as normalisation, activations and optimiser updates, whose speed can depend strongly on bandwidth. Bandwidth decides how fast a training step runs, not how large a batch or context fits: that is set by VRAM.
Support for number formats. FP16, BF16, TF32: which formats training can use natively affects both the speed and the stability of learning.
Tensor Cores. Successive generations add number formats (BF16 and TF32 arrived with Ampere) and features such as structured sparsity; raw throughput still depends on how big the chip is.
And this is where it gets interesting, because the V100 and A2 win in different categories of these four.
V100: power and memory, but no BF16
The V100 is a Volta GV100 with compute capability 7.0, with 16 or 32GB of HBM2 and a bandwidth of around 900GB/s. It is realistically the closest thing to a "proper training card" that you will find on the secondary market today at a reasonable price. In the PCIe version, the first-generation Tensor Cores deliver about 112 TFLOPS in FP16 mixed precision, while pure FP32 is about 14 TFLOPS and FP64 7 TFLOPS. Power consumption is 250W, the card is passive (it requires forced airflow in the chassis or server) and is powered through an EPS connector, and there is of course no video output: it is a compute card, not a gaming card.
The V100's greatest strength is the combination of high memory bandwidth with a solid amount of VRAM. The 32GB version genuinely widens your room for manoeuvre: larger batches, a longer context, the ability to fine-tune models that simply will not fit in 16GB together with the optimiser. With full fine-tuning (not LoRA) this is often make or break. For scale, a conventional mixed-precision AdamW setup can use about 18 bytes per parameter before activations. Roughly 1–1.5B parameters is a planning example for 32GB, rather than a fixed limit. A 7B model will not fit fully in one card's VRAM with that setup; alternatives include LoRA/QLoRA, sharded multi-GPU training, or CPU/NVMe offload.
There is, however, one "but" that needs to be stated plainly, because it is easy to overlook when looking only at TFLOPS: the V100 has no native BF16. Volta supports FP16, not BF16. This difference matters in training in practice, not only in theory: FP16 has a narrower exponent range than BF16, so small gradients can underflow to zero and large values can overflow. Mixed-precision FP16 training commonly uses loss scaling to reduce gradient underflow. PyTorch's torch.amp.GradScaler("cuda"), when enabled, adjusts the scale and skips parameter updates on non-finite gradients. Frequent skipped steps or NaN losses call for checking numerical stability; scaling does not guarantee that a model will train successfully. It is possible to work with this (the whole PyTorch ecosystem has tools for it), but it is an extra variable that you simply do not have to worry about with newer cards.
For larger training configurations and multi-GPU setups, see also the Dell R740 AI servers with the Tesla V100.
A2: modern and economical, but with a memory bottleneck
The A2 is already Ampere: GA107, compute capability 8.6, with 16GB of GDDR6 memory and a bandwidth of about 200GB/s. Its third-generation Tensor Cores bring native support for BF16 and TF32, plus 2:4 structured sparsity, which doubles Tensor Core throughput only for a model pruned to that pattern and is used mainly for inference, not in ordinary LoRA fine-tuning. The A2 has a configurable 40–60W power limit and a low-profile, single-slot form factor. It is passively cooled and requires system airflow. Its lower power limit makes heat removal easier than for the 250W V100 PCIe, but noise depends on the chassis, fans and operating conditions.
BF16 is an advantage for training stability: Hugging Face recommends it over FP16 because it needs no loss scaling, although it must be enabled in the training configuration. TF32 is something else, a faster mode for FP32 matrix multiplications that PyTorch leaves off for matmuls by default, so it has to be enabled explicitly. Software support is the other difference. CUDA 13.0 removed offline compilation and CUDA library support for Volta; CUDA 12.9 is the last toolkit that compiles for the V100. R580 is its last driver branch, supported until June 2028, and existing applications built with older CUDA can still run on compatible drivers. PyTorch 2.14 is the last release line with prebuilt CUDA 12.6 wheels for the V100; later releases require a source build with a compatible CUDA 12.x. The main CUDA FlashAttention-2 implementation and current vLLM do not support the V100. The A2 has none of these limits today, which makes it the more future-proof purchase.
The problem is the size of the gap. NVIDIA specifies 18 TFLOPS of dense FP16/BF16 Tensor Core throughput for the A2 (36 with 2:4 structured sparsity), compared with 112 FP16 Tensor TFLOPS for the standard V100 PCIe, and memory bandwidth of 200GB/s versus 900GB/s. The 16GB of VRAM limits the batch size and context that fit in a given configuration; compute and bandwidth affect training-step time, but the actual difference must be measured for the specific model and setup. The A2 excels at inference and light edge tasks, but heavy fine-tuning is not its strong suit.
Comparison table
| Parameter | V100 | A2 |
|---|---|---|
| Architecture | Volta (GV100), CC 7.0 | Ampere (GA107), CC 8.6 |
| VRAM | 16GB or 32GB HBM2 | 16GB GDDR6 |
| Memory bandwidth | ~900 GB/s | ~200 GB/s |
| Tensor Cores | 1st generation | 3rd generation |
| Formats | FP16 (no native BF16) | BF16, TF32 |
| FP16 mixed (Tensor) | ~112 TFLOPS | ~18 TFLOPS dense (36 with 2:4 sparsity) |
| FP32 / FP64 | ~14 / ~7 TFLOPS | ~4.5 TFLOPS FP32 (FP64 not listed by NVIDIA) |
| Power consumption | 250W | ~40–60W |
| Cooling | passive, forced airflow | passive/low profile |
| Sparsity | none | 2:4 structured sparsity (pruned models only) |
| Strength | raw power, memory, VRAM | low power draw, BF16/TF32, length of support |
The role of LoRA and QLoRA in this equation
This whole discussion of VRAM and bandwidth changes dramatically when you use LoRA or QLoRA instead of full fine-tuning. Instead of further training all the model's weights, you train small adapter matrices, and the base model (with QLoRA additionally quantised to 4-bit) stays frozen. This drastically reduces the VRAM requirement: a model that would need tens of GB for full training can fit into 16GB in its QLoRA version.
This is exactly where the line between the V100 and the A2 blurs. The A2 will handle light QLoRA fine-tuning of a smaller model (in the range of a few billion parameters) without difficulty, and it will also benefit from native BF16, which can be more convenient than juggling loss scaling in FP16. But when a larger model, a longer context, a larger batch or full fine-tuning without LoRA come into play, we are back to VRAM capacity and raw throughput, and there the V100, especially the 32GB version, has no equal in this comparison.
Verdict: which to choose when
Choose the V100 for serious training, such as full fine-tuning of small models, with roughly 1–1.5B parameters as a planning example on the 32GB version, or LoRA/QLoRA of larger models, when power consumption and fan noise are not priorities. For 7B–13B models, quantisation and memory-saving techniques may be needed; check the batch size and context that fit your configuration. The 32GB version is clearly the better choice here than the 16GB one if the budget allows: in practice, more VRAM means fewer compromises when choosing the model and the batch size.
Choose the A2 if you are building an economical server in which the card is mainly used for inference, with fine-tuning as an additional task carried out with LoRA/QLoRA on smaller models. You value native BF16/TF32, low power consumption and a server that is easier to cool quietly, and you want hardware that current CUDA, PyTorch and FlashAttention-2 still support.
This is not, however, a choice limited to these two cards: if your case is better suited to another accelerator, for example a newer Ampere or Ada card with more VRAM, we can source it to order and quote for it individually on request.
If you are not sure how much VRAM you actually need for a particular model, it is worth working it out in advance; I write more about this in the article how much VRAM you need for LLaMA 7B, 13B and 70B. And if you already know that the priority is inference rather than training, also have a look at which GPU for LLM inference: the calculation looks somewhat different there.
A practical note to finish
Both cards on the secondary market are passive designs: without forced airflow in the chassis or server they will simply overheat, whichever one you choose. Linux is a far friendlier environment here than Windows if you are serious about working with PyTorch and the Hugging Face tools. When buying either used accelerator, run a load test before your first training job. For example, build gpu-burn with make COMPUTE=70 using a CUDA 12.x toolkit for the V100, or make COMPUTE=86 for the A2, then run ./gpu_burn 1200 for 20 minutes. Check the program's computation-error report and compare available counters before and after the test with nvidia-smi -q -d ECC,PAGE_RETIREMENT,ROW_REMAPPER; unsupported fields may show N/A. This can reveal problems but does not guarantee future reliability.
FAQ
- Is the A2 suitable for training at all, or only for inference?
- It is suitable for light fine-tuning with LoRA/QLoRA on smaller models, and native BF16 even makes this easier. For full training of larger models it is limited by its 16GB of VRAM and by throughput: 18 dense FP16/BF16 TFLOPS and 200GB/s, versus 112 FP16 Tensor TFLOPS and 900GB/s for the standard V100 PCIe.
- Does the lack of BF16 on the V100 really get in the way in practice?
- It does not rule out training: FP16 with suitable loss scaling works and has been widely tested in the community. But it is an extra variable to deal with, which simply does not exist with Ampere.
- Is the V100 16GB worth it, or should I aim straight for 32GB?
- If you are planning anything more serious than small models and a short context, 32GB gives you real breathing room and fewer compromises when choosing the batch size; the difference can be felt sooner than you might expect.
- Does the A2 make sense as the only card in a home server for LLM experiments?
- Yes, provided the project is mainly inference plus occasional light fine-tuning. The low power limit is an advantage at home, but the card is passive and requires forced airflow; noise depends on how the entire server is cooled.
- Does the difference in Tensor Cores (1st gen vs 3rd gen) make up for the A2's lower raw performance?
- No. On NVIDIA's figures the A2 reaches 18 TFLOPS of dense FP16/BF16 Tensor Core throughput against the V100's 112. The newer generation adds BF16, TF32 and 2:4 structured sparsity, but sparsity only helps a model pruned to that pattern and is mainly used for inference. What the A2 offers instead is far lower power draw and longer software support.