For self-hosted AI, the question is rarely "which card is fastest". It is "which card fits the model I want to run, at a cost per hour I can justify". Memory decides what fits, bandwidth decides how fast tokens come out, and the price decides whether the project makes sense at all.
Memory comes first
If a model does not fit in video memory, speed is irrelevant. A rough rule: a model needs about its parameter count times the bytes per weight, plus room for the context cache.
- A 7B to 8B model in 4-bit needs roughly 5 to 6 GB.
- A 13B to 14B model in 4-bit needs about 9 to 10 GB.
- A 32B model in 4-bit needs about 20 to 22 GB, which fits on a 24 GB card with a short context.
- A 70B model in 4-bit needs about 40 GB or more, so it needs a 48 GB or 80 GB card or several smaller ones.
Long contexts grow the cache quickly, so leave headroom.
Bandwidth sets token speed
For a single user, generating each token means reading the model weights once. That makes memory bandwidth the limit. Datacenter cards with HBM memory have several times the bandwidth of consumer GDDR cards, which is why an H100 produces tokens faster on the same model.
Where a consumer card wins
- Models up to about 30B in 4-bit.
- Image generation such as Stable Diffusion and Flux.
- Development, testing and low-volume private use.
- Cost: the hourly price is a fraction of a datacenter card.
Where a datacenter card wins
- 70B-class models without heavy compression.
- Many users at once, where batching uses the extra compute well.
- Fast links between several GPUs for larger models.
- Long context and high-precision workloads.
Work out the cost per job, not per hour
Measure your own tokens per second with the model you will really run. Then divide the hourly price by tokens per hour. A cheap card that is slow can cost more per million tokens than an expensive fast one, and the other way round when the cheap card is not kept busy. Idle time is the hidden cost, so shut servers down when they are not in use.
A simple way to choose
- Pick the model and quantisation you want.
- Check that it fits with context to spare.
- Rent the cheapest card that fits and benchmark it for an hour.
- Move up only if speed or concurrency fails your target.
Ready to try? Run Ollama and Open WebUI or set up Stable Diffusion. Prices and cards change, so confirm current options on the GPU server page.