Fine-tune an open LLM with LoRA on Lium
Fine-tune Qwen2.5-0.5B-Instruct with LoRA on a Lium GPU using Hugging Face transformers and PEFT, then copy the adapter back. Five minutes of pod time, about three cents.
Lium team6 min read
lora
fine-tuning
llm
qwen
peft
rtx-a6000
LoRA lets you teach an open model new behaviour by training a few million extra parameters instead of the whole network. The result is a small adapter file you can keep, share and load on top of the base model. All it needs is a GPU for a few minutes, which is exactly what Lium rents.
In this guide we fine-tune
Qwen/Qwen2.5-0.5B-Instruct on 2,000 examples from the public Alpaca-cleaned instruction dataset with Hugging Face transformers and PEFT. Neither needs a Hugging Face token. Every command and number below comes from our test run on 15 October 2026: a 1x RTX A6000 at $0.42 per hour, ready 11 seconds after we asked for it, 152 seconds of training, five minutes of pod time in total.1. Rent a GPU
Install the CLI and sign in once.
lium init opens the browser and saves your API key and SSH key.bash
pip install lium.io
lium initA 0.5B model with LoRA fits easily on a 16 GB card. List single-GPU nodes with at least that much memory, cheapest first:
bash
lium ls --min-vram 16 --count 1On our run the cheapest were RTX A6000 (48 GB) nodes at $0.42 per GPU-hour. Check the current A6000 price before you start. Rent one;
--ttl 2h removes the pod two hours later even if you forget it:bash
lium up --gpu A6000 --name lora --ttl 2h -y --no-sshThe pod starts from Lium's default PyTorch template (PyTorch 2.12, CUDA 13.0) with root access and its own Docker daemon. Its home directory,
/root, is the pod volume, which the default PyTorch templates encrypt at rest, so we keep the model, data and adapter there.2. The training script
Save this as
lora.py. It adds rank-16 LoRA adapters to every attention and MLP projection, formats each Alpaca example with the model's own chat template, and computes the loss on the answer only: prompt tokens get the label -100, which the loss ignores. It holds out 200 examples and reports their loss before and after training.python
import torch
from datasets import load_dataset
from peft import LoraConfig, get_peft_model
from transformers import (AutoModelForCausalLM, AutoTokenizer, DataCollatorForSeq2Seq,
Trainer, TrainingArguments)
MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.bfloat16)
model = get_peft_model(model, LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]))
model.print_trainable_parameters()
# 2,000 instruction/response pairs for training, 200 held out for evaluation
data = load_dataset("yahma/alpaca-cleaned", split="train").shuffle(seed=0).select(range(2200))
data = data.train_test_split(test_size=200, seed=0)
def tokenize(row):
question = row["instruction"] + ("\n\n" + row["input"] if row["input"] else "")
prompt = tok.apply_chat_template([{"role": "user", "content": question}],
tokenize=False, add_generation_prompt=True)
prompt_ids = tok(prompt, add_special_tokens=False)["input_ids"]
answer_ids = tok(row["output"] + tok.eos_token, add_special_tokens=False)["input_ids"]
input_ids = (prompt_ids + answer_ids)[:1024]
# learn only the answer: prompt tokens get label -100
labels = ([-100] * len(prompt_ids) + answer_ids)[:1024]
return {"input_ids": input_ids, "attention_mask": [1] * len(input_ids), "labels": labels}
data = data.map(tokenize, remove_columns=data["train"].column_names)
trainer = Trainer(
model=model,
args=TrainingArguments(
output_dir="out", num_train_epochs=2, per_device_train_batch_size=8,
gradient_accumulation_steps=2, learning_rate=2e-4, lr_scheduler_type="cosine",
warmup_steps=10, bf16=True, logging_steps=25, eval_strategy="epoch",
save_strategy="no", report_to="none"),
train_dataset=data["train"], eval_dataset=data["test"],
data_collator=DataCollatorForSeq2Seq(tok, padding=True),
)
print("eval loss before:", round(trainer.evaluate()["eval_loss"], 4), flush=True)
trainer.train()
print("eval loss after:", round(trainer.evaluate()["eval_loss"], 4), flush=True)
model.save_pretrained("qwen2.5-0.5b-alpaca-lora")
tok.save_pretrained("qwen2.5-0.5b-alpaca-lora")
print("saved adapter to qwen2.5-0.5b-alpaca-lora", flush=True)Swap in your own dataset by changing the
load_dataset line and the field names in tokenize.3. Upload, install and start
lium scp without a destination puts the file in the pod's home directory. Then install the Hugging Face libraries; on our pod that took 25 seconds and installed transformers 5.19.0, peft 0.21.2, datasets 5.1.0 and accelerate 1.15.0.bash
lium scp lora ./lora.py
lium exec lora "pip install -q transformers peft datasets accelerate"Start the run in the background with
lium exec --detach. It returns immediately and the job keeps running on the pod even if your laptop disconnects.bash
lium exec lora -d --log /root/lora.log "python lora.py"4. Follow the log
bash
lium exec lora "tail -n 20 /root/lora.log"The adapter trains 8.8 million of the model's 503 million parameters (1.75%). Two epochs of 125 steps each took 152 seconds on the A6000, about 1.6 steps per second. The numbers from our run:
| Held-out loss | |
|---|---|
| Before training | 1.362 |
| After epoch 1 | 1.315 |
| After epoch 2 | 1.323 |
Training loss fell from 1.33 to 1.16, while the held-out loss was lowest after the first epoch and rose slightly in the second, a sign the model was starting to memorise the 2,000 examples. With this little data, one epoch (
num_train_epochs=1) is enough; with more data, more epochs pay off.5. Try the adapter
To check the adapter on the pod, save this as
chat.py. It loads the base model, applies the adapter and answers one prompt.python
import sys, torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("qwen2.5-0.5b-alpaca-lora")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", dtype=torch.bfloat16, device_map="cuda")
model = PeftModel.from_pretrained(base, "qwen2.5-0.5b-alpaca-lora")
prompt = tok.apply_chat_template([{"role": "user", "content": sys.argv[1]}], tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=120, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))bash
lium scp lora ./chat.py
lium exec lora "python chat.py 'Give three tips for writing a clear bug report.'"Our adapter answered in the numbered, Alpaca-style format it was trained on (cut off at the 120-token limit):
6. Copy the adapter back
lium scp copies single files, so pack the adapter folder first, then download and unpack it. The folder is 45 MB: a 35 MB adapter_model.safetensors, its config and the tokenizer.bash
lium exec lora "tar czf qwen2.5-0.5b-alpaca-lora.tar.gz qwen2.5-0.5b-alpaca-lora"
lium scp lora /root/qwen2.5-0.5b-alpaca-lora.tar.gz . -d
tar xzf qwen2.5-0.5b-alpaca-lora.tar.gzLoad it anywhere with
PeftModel.from_pretrained(base, "qwen2.5-0.5b-alpaca-lora"), as in chat.py, or merge it into the base weights with model.merge_and_unload() to serve it as a plain model.Clean up
Remove the pod once the adapter is on your machine:
bash
lium rm lora -y
lium psOur pod was up five minutes from rent to removal, and the billing ledger charged $0.03.
Why Lium for this
Lium bills by the second with no minimum, and only while the pod exists, so a five-minute fine-tune costs a few cents.
--ttl schedules the pod's removal as soon as it's created, a safety net if a run finishes while you're away. You get root and your own Docker daemon in the pod, closer to a VM than a plain container, so pip install or a containerised training stack works the way it does on your own machine. And the pod volume on the default PyTorch templates is encrypted at rest, a sensible place for your data and adapters.Run it on Lium
See RTX-A6000 pricing and availability before you start.