Best GPU cloud for open-source LLM inference in 2026
How to choose a GPU cloud for serving Llama, Qwen and other open-weight LLMs, which GPU each model size needs, and what it costs on Lium at live prices.
Lium team7 min read
llm-inference
vllm
llama
h200
gpu-pricing
Short answer: for serving an open-weight LLM yourself, pick a GPU cloud that bills per second with no minimum, publishes live prices you can read from a script, lets you run any Docker image (so you can run vLLM or SGLang as-is), and auto-terminates forgotten pods. Lium does all four. Then size the GPU to the model, not the other way round: a 7-8B model fits on one 24 GB card such as an RTX 4090, a 70B model in FP8 fits on one H200, and a 405B model needs a whole 8-GPU node.
This guide is for teams deciding where to run their own inference endpoint instead of paying per token for a hosted API. Every Lium price below was read from lium.io/pricing.json and the public node list at lium.io/api/public/v1/nodes on 15 October 2026 at 15:05 UTC. Prices on Lium are set by each GPU provider and move during the day, so treat them as a dated snapshot and check /pricing before you rent.
What to look for in a GPU cloud for inference
Inference has a different cost shape from training. Traffic is uneven, you often test several models before committing to one, and a forgotten endpoint keeps billing all weekend. Five things decide the bill more than the headline hourly price:
- Billing granularity. Per-second billing with no minimum means a 20-minute test costs 20 minutes. Lium bills per second at the node's hourly price with no term commitment; renting needs a balance that covers 15 minutes (lium.io/llms.txt, read 15 Oct 2026).
- The right VRAM tier. The cheapest GPU that holds your weights plus KV cache wins. Paying for an H200 to serve an 8B model wastes most of the card.
- Live, machine-readable prices. If an agent or a script picks the GPU, it needs prices it can read without a sales call. Lium's
pricing.jsongives, for each GPU model, a reference price (the marketplace baseline per GPU-hour), the current lowest and highest ask, and how many GPUs are free now. No account needed. - Your own serving stack. Open models change weekly. You want to run a current vLLM or SGLang image, not wait for a managed catalog to add the model.
- Guard rails against idle spend. A time-to-live on every pod is the cheapest insurance there is.
How much GPU memory each model needs
The weights alone take about 2 bytes per parameter in BF16 and about 1 byte in FP8. Parameter counts are from the models' Hugging Face pages (read 15 Oct 2026); the KV cache for your context length and batch size comes on top, so leave headroom.
| Model | Parameters | BF16 weights | FP8 weights |
|---|---|---|---|
| Llama 3.1 8B Instruct | 8.0 B | about 16 GB | about 8 GB |
| Llama 3.3 70B Instruct | 70.6 B | about 141 GB | about 71 GB |
| Llama 3.1 405B Instruct | 405.9 B | about 812 GB | about 487 GB (Meta's FP8 release keeps some layers in BF16) |
The Llama repositories are gated: accept Meta's license on Hugging Face and pass your token to the pod. Qwen2.5 models of the same sizes (7.6 B and 72.7 B parameters) are not gated and land in the same GPU tiers.
Which Lium GPU to rent for each model size
Reference prices are per GPU-hour from
pricing.json at 15:05 UTC on 15 October 2026. The daily figure is the reference price times GPU count times 24 hours, for an endpoint that stays up all day.| Workload | GPUs on Lium | Total VRAM | Reference price | Per hour | Per 24 h |
|---|---|---|---|---|---|
| 7-8B, BF16 | 1x RTX 4090 | 24 GB | $0.30 | $0.30 | $7.20 |
| 7-8B, BF16, longer context | 1x RTX 5090 | 32 GB | $0.40 | $0.40 | $9.60 |
| 70B, FP8 | 1x H200 | 141 GB | $3.65 | $3.65 | $87.60 |
| 70B, BF16 | 4x A100 SXM4 80GB | 320 GB | $0.70 | $2.80 | $67.20 |
| 70B, BF16 | 2x H200 | 282 GB | $3.65 | $7.30 | $175.20 |
| 405B, FP8 | 8x H100 | 640 GB | $1.39 | $11.12 | $266.88 |
Three things the reference price does not tell you, all visible in the same feeds:
- Live asks can sit above the reference. At read time H200 asks ranged from $3.40 to $5.72 per GPU-hour across 10 free GPUs, and the free A100 SXM4 80GB GPUs were on 8-GPU nodes at $1.68 per GPU-hour that rent only as a whole node (
min_rentable_gpu_countis 8 in the node list). - Popular cards can be fully rented. Lium listed 1,670 GPUs at read time, 1,571 of them rented and 73 free. All 102 listed RTX 4090s and all 235 listed H100 80GB HBM3s were taken at that moment. Keep a second GPU type in mind: an 8B model also fits a 24 GB RTX 3090, which had 4 free GPUs at $0.22.
- Spot versus secure. Every node has a
tierofsecureorspot, and spot capacity can be reclaimed. The $3.40 H200 was a single spot GPU. For an endpoint users depend on, filter tosecure; for batch jobs that can restart, spot is fine.
A worked example: Llama 3.3 70B on one H200
Check the price first, without an account:
bash
curl -s https://lium.io/pricing.json | jq '.models[] | select(.slug=="h200") | {reference_price_usd_per_gpu_hour, min_price_usd_per_gpu_hour, max_price_usd_per_gpu_hour, available_gpus}'Install the CLI and sign in once;
lium init opens the browser and saves your API key and SSH key:bash
curl -fsSL https://lium.io/install.sh | bash # or: pip install lium.io
lium initPick the cheapest secure-tier H200 node.
--ttl 6h removes the pod after six hours even if you forget it, and --verify-gpus --strict-gpus checks the container sees as many GPUs as you are billed for and removes the pod if not:bash
ID=$(lium ls --gpu H200 --format json | jq -r 'map(select(.tier=="secure")) | sort_by(.price_per_gpu_hour) | .[0].id')Serving works as in our DeepSeek V4.1 Flash guide: the pod starts from the
vllm/vllm-openai image rather than a template, vLLM binds to 127.0.0.1 and requires an API key, and you reach it through an SSH tunnel. Llama 3.3 is a gated model, so first accept its license on Hugging Face and create a read token there. Then, on your laptop:bash
export HF_TOKEN=hf_... # your Hugging Face read token
export VLLM_API_KEY=$(openssl rand -hex 16) # keep this terminal open; you need the key to call the serverRent one GPU on that node. For the 70B model, vLLM's online FP8 quantization (vLLM docs, read 15 Oct 2026) converts the BF16 weights at load time;
HF_HOME puts the download on the pod volume. lium up then streams the server's log; it is ready when it prints Application startup complete:bash
lium up "$ID" -c 1 --name llama70b --ttl 6h --verify-gpus --strict-gpus -y \
--image vllm/vllm-openai:v0.31.0 \
-e HF_TOKEN="$HF_TOKEN" \
-e VLLM_API_KEY="$VLLM_API_KEY" \
-e HF_HOME=/workspace/hf \
--entrypoint vllm \
--cmd 'serve meta-llama/Llama-3.3-70B-Instruct --quantization fp8_per_tensor --max-model-len 32768 --host 127.0.0.1 --port 8000'The SSH tunnel and the request are step 3 of the DeepSeek guide, with
llama70b as the pod name and meta-llama/Llama-3.3-70B-Instruct as the model.We have not published a benchmark of this exact setup yet, so the sizing above is memory math, not a measured throughput figure. Our measured end-to-end run is the DeepSeek guide: an 8x H100 node serving a 510 GB model, ready about 10 minutes after the pod was created.
When you are done, remove the pod so billing stops:
bash
lium rm llama70b -yWhen Lium is the right choice, and when it isn't
Pick Lium when you want to choose the exact model, version and serving engine yourself; your usage is bursty or experimental, so per-second billing matters; an agent or CI job has to rent GPUs from code; or you want to compare the price of a 4090, an A100 and an H200 for the same job in one feed before deciding.
Look elsewhere when you only need a popular model behind a per-token API and never want to manage a server; a hosted inference API is less work. The same goes when you need a specific GPU guaranteed at a fixed time: Lium is a live marketplace, and as the snapshot above shows, a popular card can be fully rented at a given moment.
Keeping these numbers honest
Every price on this page is dated 15 October 2026, 15:05 UTC. To re-check any row, read the matching
/gpu/<model> page, /pricing, or pricing.json, which updates continuously. If you are comparing clouds, run the same model, context length and traffic pattern on each, and compare cost per million output tokens rather than the hourly price.Run it on Lium
See H200 pricing and availability before you start.
