Teaching a 9B model to direct motion graphics

How we fine-tuned Qwen3.5-9B to write motion-graphics scenes, and six files that repeat it: data checks, a 230-line LoRA trainer, the merge, and serving with llama.cpp and vLLM.

Prakash Kumar Oct 21, 2026 AI 39 min read

We fine-tuned Qwen3.5-9B to write the JSON scenes our video renderer plays. This post tells how we did it, what broke, and why each choice is there. It also gives you six files that repeat it: one script on a rented GPU trains a LoRA adapter. Run it on our toy data first, then on yours.

Our run: 20 minutes on one H100, about $1.30 of GPU time at secure-cloud prices. Validation loss fell from 0.433 to 0.193.

What decides the result

Most of a fine-tune is decided before the first step. Six things matter, most important first:

1. The eval, not the loss. Loss is a proxy. Our 27B run reached a lower validation loss and passed fewer checks than the 9B. Decide what “good” means before you train. (Why Qwen3.5-9B)
2. Data you would ship. The model copies its answers, mistakes included. Cover every task, balance the categories, keep near-duplicates on one side of the split. (The data)
3. A base that can almost do it. A LoRA teaches format and taste on top of what the base already knows. If the base fails with a good prompt, a few hundred rows will not fix it.
4. Training and serving match. Same chat template, same tokenizer revision, same thinking switch. Our two worst bugs lived here; training was correct both times. (Merge, Serverless)
5. Loss on the right tokens. Only the answer carries loss. A wrong mask teaches the model to write the prompt. (Grade only the answer)
6. The learning rate. The knob that matters most. Too high: spikes and forgetting. Too low: nothing moves. Rank and batch size come second. (Optimizer and schedule)

How to read the loss:

• Compare against step 0, not against a fixed number. e^−loss is the geometric-mean probability the model gives the right token: ours went from 0.65 to 0.82.
• Validation rises while training falls: overfitting. Keep the best checkpoint; ours was step 80 of 133.
• Flat from step 0: wrong mask, adapter not attached, or learning rate too low.
• Validation below training: leakage between the splits, or an easier validation set.
• Sudden spikes: learning rate too high, or one bad row.

0. What you need

Every code block starts with where it runs:

Step	Runs on
2. Check the data	your machine
3. Rent the GPU	your machine
4. Train	the pod
7. Merge, copy the adapter, stop the pod	the pod, then your machine
8–9. Serve	your Mac, or RunPod Serverless

• A Hugging Face account. Qwen3.5-9B is open, so you need no token for it.
• A RunPod account with credit, and runpodctl set up with runpodctl doctor. Add your SSH key once with runpodctl ssh add-key --key-file ~/.ssh/id_ed25519.pub.
• Python 3.10+ and jq on your machine.
• Optional: a Mac with 16 GB or more, or any machine with llama.cpp, to test the result locally.

Get the files:

# your machine
mkdir qwen-lora && cd qwen-lora
BASE=https://thecandor.in/downloads/qwen-lora
for f in prepare_data.py example.jsonl setup.sh run.sh train_lora.py merge.py; do
  curl -fsSO "$BASE/$f"
done

File	Runs on	Job
prepare_data.py	your machine, and again on the pod	checks rows, drops long ones, splits by group
example.jsonl		40 toy rows: headline → one-node scene, plus edits
setup.sh	pod	venv, pinned packages, kernel check, base weights
run.sh	pod	setup → data → 3-step smoke test → full run
train_lora.py	pod	the trainer, section 5
merge.py	pod	adapter + base → one bf16 model

1. Why Qwen3.5-9B

We ran the same recipe on Qwen3.5-27B, Gemma 4 E4B and Gemma 4 12B. The 27B reached a lower validation loss but passed 3 of 4 structured checks against the 9B’s 4 of 4 (durations within 10 s). Two traits made the 9B a good base:

• Long context is cheap. 24 of its 32 layers are Gated DeltaNet linear attention, with a fixed-size state instead of a growing KV cache. Our prompts reach 17k tokens. Here is what that does to serving memory.
• Thinking switches off. We want JSON, not a reasoning trace. enable_thinking=False goes into the chat template at training time and at serving time.

Pin the model revision. Upstream repositories get re-uploaded. A changed tokenizer silently breaks an adapter. Every file uses commit c202236235762e1c871ad0ccb60c8ee5ba337b9a.

2. The data

Format

One conversation per line of JSONL. The last message is the answer the model learns to write; everything before it is context. One row, pretty-printed:

{
  "id": "film-0412-build-3",
  "category": "animated_text",  // loss is reported per category
  "group": "family-07d92aa5",   // never split across train and validation
  "messages": [
    { "role": "system",    "content": "Return only the JSON. ..." },
    { "role": "user",      "content": "<task + approved plan>" },
    { "role": "assistant", "content": "<scene JSON>" }
  ]                             // ↑ the answer: the only tokens with loss
}

Both content values are JSON encoded as strings, so the real row escapes every quote: "{\"task\":\"build\",...}".

• category is the axis you measure quality on. Ours are motion graphics, animated text, visual styling and media storytelling.
• group ties together rows that must stay on one side of the split. The trainer refuses to start if a group is in both files.

How we built ours

The model copies its training answers, so every answer must be one you would ship. We kept only films that passed these gates:

finished videos + studio films      creative fields only, no account data
        │
        ▼
render every film to MP4            we judge the video
        │
        ▼
review frames at 2·20·45·70·95 %    fail: text < 36 px, clip, overlap,
        │                                 missed timing
        │                           pass: score ≥ 8, no score < 7
        ├─ fail ─► 1 repair ─► review again ─► fail ─► drop
        ▼
lock the approval                   SHA-256 of JSON + render;
        │                           1 changed byte voids it
        ▼
split into rows                     plan, build, revise
        │                           (table below)
        ▼
rewrite the plan                    must describe the accepted scenes
        │
        ▼
group near-duplicates               same creator, theme, or
        │                           5-gram Jaccard ≥ 0.65
        ▼
drop impossible rows                6 asked 40 s of a 36.9 s track
        │
        ▼
389 rows

The rows map to the model’s three stages:

Stage	Input	Output	Rows per film
plan	brief	script plan	1
build	plan + scene index	that scene’s node tree	1 per scene
revise	scene + change request	edited scene	1 per real repair

Two gates need a reason:

• Rewrite the plan. A plan that promises a scene the renderer never showed teaches the model to promise what it does not build.
• Group. Near-duplicates on both sides of the split make validation loss look good and mean nothing.

389 conversations, about 2 million tokens, split 354 / 35. Median 4,352 tokens, longest 17,827. Answers run from 360 to 7,435 tokens.

Check it

We check the data before we pay for a GPU:

# your machine
pip install transformers jinja2
python prepare_data.py example.jsonl      # or your own file

rows=40 dropped=0 median=386 max=405 total=12366
train: 36 rows {'build': 18, 'revise': 18}
validation: 4 rows {'build': 2, 'revise': 2}

It stops on a row without id, category, group or messages. It drops rows over --max-tokens (default 24,576) and names them. It writes train.jsonl and validation.jsonl with about 10% of the groups held out, and warns if a category has no validation rows.

Watch max: peak training memory follows the longest conversation. Our 17,827 tokens peaked at 29.3 GB. The toy rows are short, so they prove the pipeline, not quality.

3. Rent the GPU

We keep everything on a network volume: the venv, the Hugging Face cache, the runs and the merged model. A stopped pod loses nothing, and the serverless endpoint later mounts the same volume.

• One data center for both. A network volume lives in one data center. Pods and serverless workers can mount it only there. Pick a region that has both H100s for training and 48 GB cards for serving.
• 160 GB holds the venv (~10 GB), the base weights (19.3 GB), the merged model (19.3 GB) and run artifacts, with room for a second run. It costs $11.20 a month.
• The volume mounts at /workspace on a pod and at /runpod-volume on a serverless worker. Same files, two paths.
• Why an H100: peak memory is 29 GB, so a 48 GB card also fits. Training is compute-bound, and an H100 has roughly 6× the bf16 compute of an RTX A6000 at about 6.8× the hourly price. Cost per run is similar; the H100 finishes in 20 minutes instead of about two hours.

# your machine
VOL=$(runpodctl network-volume create --name lora-vol --size 160 \
  --data-center-id CA-MTL-1 | jq -r .id)

POD=$(runpodctl pod create --name lora-train \
  --image runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2404 \
  --gpu-id "NVIDIA H100 80GB HBM3" --cloud-type SECURE \
  --data-center-ids CA-MTL-1 --network-volume-id "$VOL" \
  --container-disk-in-gb 100 --ports 22/tcp --wait | jq -r .id)

runpodctl ssh info "$POD"      # prints the ssh command for this pod
cd .. && runpodctl send qwen-lora   # prints a one-time code

Open a second terminal, SSH into the pod with the printed command, and receive the folder onto the volume:

# on the pod
cd /workspace && runpodctl receive <code>

4. Train

# on the pod
cd /workspace/qwen-lora
bash run.sh example.jsonl      # or your own file
tail -f runs/run-1/log.jsonl

run.sh does four things:

1. setup.sh creates the venv, installs pinned packages, checks the kernels and downloads the base weights. The venv and HF_HOME live on the volume, so only the first pod pays for the install and the 19 GB download.
2. prepare_data.py checks and splits the data again, on the pod.
3. A 3-step smoke test on the longest conversations first, so peak memory shows up in the first minute instead of at step 90. It ends with smoke test passed. Read max_mem_gb in its lines. If it is near your card’s limit, rerun with a lower --max-tokens in prepare_data.py.
4. The full run, detached with nohup so an SSH drop does not kill it. It writes runs/run-1/log.jsonl, keeps runs/run-1/adapters/best and resumes if you run the same command again.

setup.sh checks that Transformers found the flash-linear-attention kernels. Without them, Transformers falls back to a pure-PyTorch loop for the DeltaNet layers. It runs more than 10× slower and prints no warning. A slow run looks like a working run until the budget is gone.

Our log:

Step	Validation loss (macro)
0 (base)	0.433
16	0.286
48	0.201
80 (kept)	0.193
133 (final)	0.196

133 steps: 3 epochs over 1.8 M unique tokens, 5.4 M processed. Most of the gain came in the first epoch. After step 80 the loss rose slightly: the start of overfitting, which selection caught. Media storytelling improved most (0.484 → 0.139). Animated text stays hardest at 0.282, because its answers carry the most exact per-word timing.

On 4 held-out briefs, the model returned valid JSON with the right stage fields, scene count and media bindings 4 out of 4 times. Durations matched within 10 s; a strict duration check failed 1 of 4. Renders still show mistakes JSON checks cannot see: two words that overlap, a final keyframe 152 px short.

Watch only the validation lines:

# on the pod
tail -f runs/run-1/log.jsonl | jq -c 'select(.kind=="validation") | {step, val_loss}'

5. The trainer

This is train_lora.py, the same file you downloaded. It is a plain PyTorch loop, not a framework Trainer: at 17k tokens per sequence you need to see where every gigabyte goes.

"""LoRA fine-tune of Qwen3.5-9B on chat-format JSONL. Adapted from
MaximeRivest/sciencemadereadable training/train.py (Apache-2.0).

    python train_lora.py --name smoke --max-steps 3 --longest-first --epochs 1
    python train_lora.py --name run-1
"""
import os

os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")

import argparse
import inspect
import json
import math
import random
import shutil
import time
from pathlib import Path

import torch
import torch.nn.functional as F
from torch.utils.checkpoint import checkpoint

MODEL = "Qwen/Qwen3.5-9B"
REVISION = "c202236235762e1c871ad0ccb60c8ee5ba337b9a"


def encode(tokenizer, messages):
    """Token ids of the conversation and the index where the assistant answer starts."""
    prompt, kw = messages[:-1], {"enable_thinking": False}
    head = list(tokenizer.apply_chat_template(
        prompt, add_generation_prompt=True, tokenize=True, return_dict=False, **kw))
    full = list(tokenizer.apply_chat_template(messages, tokenize=True, return_dict=False, **kw))
    if full[:len(head)] != head:
        answer = tokenizer(messages[-1]["content"], add_special_tokens=False)["input_ids"]
        full = head + answer + [tokenizer.eos_token_id]
    return full, len(head)


def load_split(tokenizer, path, max_tokens):
    rows = [json.loads(line) for line in Path(path).read_text().splitlines() if line.strip()]
    out = []
    for row in rows:
        ids, start = encode(tokenizer, row["messages"])
        if not 0 < start < len(ids) <= max_tokens:
            raise ValueError(f"{row['id']}: {len(ids)} tokens, answer starts at {start}")
        out.append(dict(ids=torch.tensor(ids), start=start,
                        category=row["category"], group=row["group"]))
    return out


def load_model(rank, device):
    from peft import LoraConfig, get_peft_model
    from transformers import AutoModelForCausalLM, AutoTokenizer
    import transformers.models.qwen3_5.modeling_qwen3_5 as qwen
    kernel = inspect.getclosurevars(qwen.torch_chunk_gated_delta_rule)
    if not kernel.nonlocals.get("is_new_implementation"):
        raise SystemExit("flash-linear-attention kernels missing: refusing the slow path")
    tokenizer = AutoTokenizer.from_pretrained(MODEL, revision=REVISION)
    model = AutoModelForCausalLM.from_pretrained(
        MODEL, revision=REVISION, dtype=torch.bfloat16, attn_implementation="sdpa")
    model.config.use_cache = False
    model = get_peft_model(model, LoraConfig(
        r=rank, lora_alpha=2 * rank, lora_dropout=0.0, bias="none",
        target_modules="all-linear", task_type="CAUSAL_LM"))
    for p in model.parameters():
        if p.requires_grad:
            p.data = p.data.float()
    model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False})
    model.enable_input_require_grads()
    return model.to(device), tokenizer


def _chunk_loss(weight, hidden, target):
    return F.cross_entropy(F.linear(hidden, weight).float(), target, reduction="sum")


def answer_loss_sum(model, ids, start, chunk=2048):
    """Summed cross-entropy over answer tokens. Logits are built 2,048 positions at a time
    and recomputed in backward, so the full [tokens x 248k] matrix never exists."""
    base = model.get_base_model()
    with torch.autocast("cuda", dtype=torch.bfloat16):
        hidden = base.model(input_ids=ids[None]).last_hidden_state[0]
    hidden, target = hidden[start - 1:-1], ids[start:].long()
    weight = base.get_output_embeddings().weight
    total = hidden.new_zeros((), dtype=torch.float32)
    for a in range(0, len(target), chunk):
        total = total + checkpoint(_chunk_loss, weight, hidden[a:a + chunk],
                                   target[a:a + chunk], use_reentrant=False)
    return total, len(target)


@torch.no_grad()
def validate(model, val, device):
    """Macro average: mean answer-token loss per category, then the mean of those."""
    model.eval()
    losses = {}
    for category in sorted({r["category"] for r in val}):
        loss, n = 0.0, 0
        for r in (r for r in val if r["category"] == category):
            l, k = answer_loss_sum(model, r["ids"].to(device), r["start"])
            loss, n = loss + l.item(), n + k
        losses[category] = loss / n
    model.train()
    return sum(losses.values()) / len(losses), losses


def learning_rate(step, total, peak, warmup, decay_share=0.15):
    """Warmup-stable-decay: linear ramp, flat, then linear decay to 10% over the last 15%."""
    if step < warmup:
        return peak * (step + 1) / warmup
    decay_start = int(total * (1 - decay_share))
    if step < decay_start:
        return peak
    return peak * (1 - 0.9 * (step - decay_start) / max(1, total - decay_start))


def save_last(model, optimizer, step, run):
    """Write last.tmp, then swap it in with renames. A kill at any point leaves
    either last/ or last.old/ complete."""
    tmp, last, old = run / "last.tmp", run / "last", run / "last.old"
    shutil.rmtree(tmp, ignore_errors=True)
    model.save_pretrained(tmp / "adapter")
    torch.save({"optimizer": optimizer.state_dict(), "step": step}, tmp / "trainer.pt")
    shutil.rmtree(old, ignore_errors=True)
    if last.exists():
        last.rename(old)
    tmp.rename(last)
    shutil.rmtree(old, ignore_errors=True)


def resume(model, optimizer, run):
    from peft import load_peft_weights, set_peft_model_state_dict
    saved_dir = next((d for d in (run / "last", run / "last.old")
                      if (d / "trainer.pt").exists()), None)
    if saved_dir is None:
        return 0
    set_peft_model_state_dict(model, load_peft_weights(str(saved_dir / "adapter")))
    saved = torch.load(saved_dir / "trainer.pt", weights_only=False)
    optimizer.load_state_dict(saved["optimizer"])
    return saved["step"]


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--name", required=True)
    ap.add_argument("--train-file", default="train.jsonl")
    ap.add_argument("--validation-file", default="validation.jsonl")
    ap.add_argument("--epochs", type=int, default=3)
    ap.add_argument("--lr", type=float, default=1e-4)
    ap.add_argument("--rank", type=int, default=32)
    ap.add_argument("--per-step", type=int, default=8)
    ap.add_argument("--max-tokens", type=int, default=24576)
    ap.add_argument("--evals", type=int, default=8)
    ap.add_argument("--max-steps", type=int, default=0)
    ap.add_argument("--longest-first", action="store_true")
    ap.add_argument("--checkpoint-minutes", type=float, default=10)
    ap.add_argument("--seed", type=int, default=0)
    args = ap.parse_args()

    torch.manual_seed(args.seed)
    device = "cuda"
    run = Path("runs") / args.name
    run.mkdir(parents=True, exist_ok=True)

    def log(**row):
        print(json.dumps(row), flush=True)
        with open(run / "log.jsonl", "a") as f:
            f.write(json.dumps(row) + "\n")

    model, tokenizer = load_model(args.rank, device)
    train = load_split(tokenizer, args.train_file, args.max_tokens)
    val = load_split(tokenizer, args.validation_file, args.max_tokens)
    if {r["group"] for r in train} & {r["group"] for r in val}:
        raise SystemExit("A group appears in both splits: validation would leak")

    order = []
    for epoch in range(args.epochs):
        epoch_order = list(range(len(train)))
        random.Random(args.seed + epoch).shuffle(epoch_order)
        order += epoch_order
    if args.longest_first:
        order.sort(key=lambda i: -len(train[i]["ids"]))
    total_steps = math.ceil(len(order) / args.per_step)
    if args.max_steps:
        total_steps = min(total_steps, args.max_steps)
    eval_every = max(1, total_steps // args.evals)
    warmup = max(1, min(30, total_steps // 20))

    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.AdamW(params, lr=args.lr, betas=(0.9, 0.99), weight_decay=0.0)
    start = resume(model, optimizer, run)
    log(kind="start", trainable_M=round(sum(p.numel() for p in params) / 1e6, 1),
        train=len(train), val=len(val), total_steps=total_steps, resumed_at=start)

    if not args.max_steps and start == 0:
        loss, by_category = validate(model, val, device)
        log(kind="validation", step=0, val_loss=round(loss, 4), categories=by_category)

    best_file = run / "best-checkpoint.json"
    best = json.loads(best_file.read_text())["val_loss"] if best_file.exists() else float("inf")
    last_save = time.time()
    model.train()
    for step in range(start, total_steps):
        lr = learning_rate(step, total_steps, args.lr, warmup)
        for g in optimizer.param_groups:
            g["lr"] = lr
        batch = [train[i] for i in order[step * args.per_step:(step + 1) * args.per_step]]
        n_answer = sum(len(r["ids"]) - r["start"] for r in batch)
        t, loss_sum = time.time(), 0.0
        for r in batch:
            loss, _ = answer_loss_sum(model, r["ids"].to(device), r["start"])
            if not torch.isfinite(loss):
                raise RuntimeError("non-finite loss")
            (loss / n_answer).backward()
            loss_sum += loss.item()
        grad_norm = torch.nn.utils.clip_grad_norm_(params, 1.0).item()
        optimizer.step()
        optimizer.zero_grad(set_to_none=True)
        log(kind="train", step=step + 1, loss=round(loss_sum / n_answer, 4), lr=lr,
            grad_norm=round(grad_norm, 3), secs=round(time.time() - t, 1),
            max_mem_gb=round(torch.cuda.max_memory_allocated() / 2**30, 1))

        done = step + 1 == total_steps
        if not args.max_steps and ((step + 1) % eval_every == 0 or done):
            loss, by_category = validate(model, val, device)
            log(kind="validation", step=step + 1, val_loss=round(loss, 4), categories=by_category)
            model.save_pretrained(run / "adapters" / f"step-{step + 1:04d}")
            if loss < best:
                best = loss
                model.save_pretrained(run / "adapters" / "best")
                best_file.write_text(json.dumps(
                    {"step": step + 1, "val_loss": loss, "categories": by_category}))
        if not args.max_steps and (done or time.time() - last_save > args.checkpoint_minutes * 60):
            save_last(model, optimizer, step + 1, run)
            last_save = time.time()

    print("smoke test passed" if args.max_steps else "finished", flush=True)


if __name__ == "__main__":
    main()

6. Why each piece is there

LoRA, rank 32, every linear layer

The base weights stay frozen. Each linear layer W gets a trainable update W' = W + (α/r)·B·A, where A is r×d_in and B is d_out×r. B starts at zero, so step 0 is exactly the base model.

• r = 32, α = 64. α/r = 2 keeps the update scale fixed if you change the rank later.
• all-linear adapts attention, DeltaNet and MLP projections. A schema is a pattern in what the model writes, and the MLPs hold as much of that as attention does.
• Dropout 0. Three epochs over 354 rows is short. Dropout slows learning more than it prevents overfitting here, and checkpoint selection handles overfitting.

Result: 86.6 M trainable parameters, 0.96% of the model. The adapter file is 346 MB, because the trainer keeps LoRA weights in fp32 (173 MB if you cast to bf16).

Grade only the answer

Prompts carry thousands of tokens of plan and asset JSON. Loss on the prompt teaches the model to predict inputs it will always receive. encode() renders the conversation with the model’s own chat template and returns the index where the answer starts. The loss starts there and includes the end-of-turn token, so the model also learns when to stop.

There are no -100 label masks. The trainer slices the hidden states to the answer positions before the vocabulary projection, which also skips 248k-wide logits for every prompt token.

The fallback branch in encode() handles templates that render the prompt prefix differently once the answer is appended. Train with the exact template you serve with, enable_thinking=False included. A mismatch costs quality and raises no error.

Never build full logits

The vocabulary has 248,320 entries. Full fp32 logits for a 17,827-token conversation are 17,827 × 248,320 × 4 bytes = 17.7 GB, and backward needs the same again. answer_loss_sum() runs the transformer body once, then computes logits and cross-entropy 2,048 positions at a time. Each chunk runs under checkpoint, so its logits are freed after the forward pass and recomputed in backward. Peak logits memory: 2,048 × 248,320 × 4 bytes ≈ 2 GB.

One conversation per forward pass

Our lengths run from 1.5k to 17.8k tokens. Padding a batch to its longest member wastes up to 90% of the compute. Packing several conversations into one sequence needs cross-document masks, and the DeltaNet kernels carry state across the whole sequence. So each conversation runs alone, and gradients accumulate across --per-step 8 conversations before one optimizer step.

Weight every token the same

(loss / n_answer).backward() divides each conversation’s summed loss by the total answer tokens in the step. The step’s gradient is then the mean over all answer tokens in it. A per-example mean would give a 360-token answer the same weight as a 7,000-token one.

Memory: trade compute for space

• Gradient checkpointing (use_reentrant=False) stores only layer inputs and recomputes activations in backward. About 30% more compute, and 17k-token sequences fit. enable_input_require_grads() makes the frozen embedding output require grad, so every checkpointed block has a differentiable input.
• bf16 base, fp32 adapter. bf16 halves the frozen weights. The adapter is 87 M parameters, so fp32 costs little. In bf16, small LoRA updates round to zero against large weights.
• expandable_segments:True. Sequence length changes every forward pass. Without it the CUDA allocator fragments and fails with memory still free.
• use_cache=False. Training never needs a KV cache.

Peak: 29.3 GB on the longest conversation.

Optimizer and schedule

• AdamW, β = (0.9, 0.99), weight decay 0. β₂ = 0.99 adapts faster than the default 0.999 to the uneven gradient scale of variable-length data in a short run. Weight decay pulls B toward zero, which pulls the adapter toward doing nothing.
• Peak LR 1e-4, warmup–stable–decay. Warmup is max(1, min(30, total//20)) steps: 6 for our 133. Adam’s second-moment estimate is noise for the first steps, and small steps keep that noise harmless. The flat phase learns fastest. A linear decay to 10% over the last 15% settles the weights. Unlike cosine, the shape does not depend on the total length, so a longer run reuses it.
• Gradient clipping at norm 1.0. One 7k-token answer can give a large gradient. Clipping limits the damage to one step.
• 3 epochs, reshuffled each epoch with seed + epoch. A rerun sees the same order.

Select on categories

Validation runs at step 0, which gives the base-model baseline, and then 8 more times. The score is a macro average: the mean loss per category, averaged. A plain token mean lets the largest category hide a regression in a small one. The trainer keeps adapters/best, not the last step.

Resume safely

Every 10 minutes the trainer writes adapter, optimizer state and step to last.tmp/. Then it renames last/ to last.old/ and last.tmp/ to last/. Each rename is atomic, so a pod killed at any point leaves one complete checkpoint, and resume() takes whichever exists. Optimizer state is saved too: a restart without Adam’s moments jolts the weights. Run the same command again to resume.

7. Merge, copy, stop

vLLM can serve LoRA adapters at runtime, but a merged model has no adapter overhead per token and no --enable-lora limits.

Our first merge saved the unchanged base model. Training loaded the text-only Qwen3_5ForCausalLM, so adapter keys read model.layers.N. The full model nests the same layers under model.language_model.layers.N. PEFT matched none of the 496 adapter tensors, printed only a warning, and merge_and_unload() saved the base. merge.py now remaps the keys and turns that warning into a crash.

We merge on the training pod after the run finishes, copy the adapter to the Mac for llama.cpp, then stop the pod:

# on the pod
cd /workspace/qwen-lora
/workspace/venv/bin/python merge.py runs/run-1/adapters/best \
  /workspace/models/my-model-merged
runpodctl send runs/run-1/adapters/best      # prints a one-time code

# your machine, in qwen-lora/
runpodctl receive <code>                     # creates best/
runpodctl pod stop "$POD"

Stop the pod. An idle H100 costs $2.69–3.99 an hour. The volume keeps the merged model for serving.

"""Merge a LoRA adapter into Qwen3.5-9B and save a standalone bf16 model.

    python merge.py runs/run-1/adapters/best /workspace/models/my-model-merged
"""
import sys
import warnings

import torch
from peft import PeftModel
from transformers import (AutoProcessor, AutoTokenizer,
                          Qwen3_5ForConditionalGeneration)

MODEL = "Qwen/Qwen3.5-9B"
REVISION = "c202236235762e1c871ad0ccb60c8ee5ba337b9a"
# Training loaded the text-only model; the full model nests its layers.
KEY_MAPPING = {r"^model\.layers\.": "model.language_model.layers."}

adapter_dir, out = sys.argv[1], sys.argv[2]
warnings.filterwarnings("error", message="Found missing adapter keys")

base = Qwen3_5ForConditionalGeneration.from_pretrained(
    MODEL, revision=REVISION, dtype=torch.bfloat16,
    device_map="cuda:0", attn_implementation="sdpa")
adapter = PeftModel.from_pretrained(base, adapter_dir, key_mapping=KEY_MAPPING)
merged = adapter.merge_and_unload().to(torch.bfloat16)
merged.save_pretrained(out, max_shard_size="4GB")
AutoTokenizer.from_pretrained(MODEL, revision=REVISION).save_pretrained(out)
AutoProcessor.from_pretrained(MODEL, revision=REVISION).save_pretrained(out)
print("merged into", out)

• Merge into the class named in config.json (Qwen3_5ForConditionalGeneration). The saved checkpoint then has the same layout and weight names as the upstream model, which is what vLLM expects for this architecture.
• merge_and_unload() computes W + (α/r)·B·A once per layer and removes the LoRA modules. The result is an ordinary bf16 checkpoint, 19.3 GB.
• Save the tokenizer and processor files with the weights. The directory must load on its own, with no reference to the Hub.

8. Serve locally with llama.cpp

For development, we run the adapter on a quantized base on a Mac. Homebrew ships the Metal build of the binaries; the clone supplies the Python conversion scripts. On Linux, build the binaries with cmake -B build -DGGML_CUDA=ON && cmake --build build -j and call them from build/bin/.

Convert the base and the adapter to GGUF and start the server. The base download is 19 GB:

# your machine, in qwen-lora/
brew install llama.cpp                      # llama-server, llama-quantize
git clone --depth 1 https://github.com/ggml-org/llama.cpp
python3 -m venv .venv-gguf && . .venv-gguf/bin/activate
pip install -r llama.cpp/requirements.txt
SNAP=$(python -c "from huggingface_hub import snapshot_download as s
print(s('Qwen/Qwen3.5-9B',
        revision='c202236235762e1c871ad0ccb60c8ee5ba337b9a'))")

python llama.cpp/convert_hf_to_gguf.py "$SNAP" --outtype bf16 \
  --outfile qwen3.5-9b-bf16.gguf
llama-quantize qwen3.5-9b-bf16.gguf qwen3.5-9b-Q6_K.gguf Q6_K
python llama.cpp/convert_lora_to_gguf.py --outtype f16 --base "$SNAP" \
  --outfile my-lora-f16.gguf best

llama-server -m qwen3.5-9b-Q6_K.gguf --lora my-lora-f16.gguf \
  -ngl 99 -fa on -c 24576 -np 1 -kvu --host 127.0.0.1 --port 8100 \
  --alias my-model --jinja --chat-template-kwargs '{"enable_thinking": false}'

• Q6_K is about 6.6 bits per weight: ~7.5 GB for the text model, close to bf16 quality, and it fits a 16 GB Mac with a 24k context.
• -c 24576 -np 1: one slot with the full context. -kvu uses one unified KV buffer.
• --jinja --chat-template-kwargs applies the model’s own template with thinking off: the same template as training.

Ask it for a toy scene from a second terminal:

# your machine
curl -s http://127.0.0.1:8100/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-model",
    "messages": [
      {"role": "system", "content": "Build one scene. Return only the scene JSON."},
      {"role": "user", "content": "{\"task\":\"build\",\"headline\":\"Hello world\",\"mood\":\"calm\",\"duration\":4}"}
    ],
    "max_tokens": 1024,
    "temperature": 0
  }' | jq -r '.choices[0].message.content'

The server speaks the OpenAI API at http://127.0.0.1:8100/v1, so any OpenAI client works. A trained adapter answers with one scene in the toy format: a text node, the calm palette and a four-step timeline. The base model alone writes a different structure.

9. Serve on RunPod Serverless with vLLM

Create a template that runs RunPod’s vLLM worker against the merged model on the volume, then an endpoint:

# your machine
TPL=$(runpodctl template create --name my-model-vllm --serverless \
  --image runpod/worker-v1-vllm:v2.28.0 --container-disk-in-gb 20 \
  --env '{
    "MODEL_NAME": "/runpod-volume/models/my-model-merged",
    "OPENAI_SERVED_MODEL_NAME_OVERRIDE": "my-model",
    "MAX_MODEL_LEN": "49152",
    "GPU_MEMORY_UTILIZATION": "0.90",
    "MAX_CONCURRENCY": "8"
  }' | jq -r .id)

EP=$(runpodctl serverless create --name my-model --template-id "$TPL" \
  --gpu-id "NVIDIA A40" --gpu-count 1 --network-volume-id "$VOL" \
  --data-center-ids CA-MTL-1 --workers-min 0 --workers-max 1 \
  --idle-timeout 300 --execution-timeout 900 | jq -r .id)

• MODEL_NAME is the mount path. The worker reads the weights from the volume, so a cold start downloads nothing.
• MAX_MODEL_LEN=49152 is a memory reservation, not a feature. How to size it.
• MAX_CONCURRENCY=8 lets one worker take 8 jobs at once, which vLLM batches. At 1, every request waits for the one before it.
• Do not set ENFORCE_EAGER=true. It shortens startup by skipping CUDA graph capture and makes every decode step slower.
• Add more GPU types in the dashboard (RTX A6000, L40, L40S, RTX 6000 Ada). A volume pins workers to one data center, and one GPU type there can run out.

Call it through the async route. A cold start takes minutes, and a synchronous call times out:

# your machine
JOB=$(curl -s https://api.runpod.ai/v2/$EP/run \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": {
      "route": "/v1/chat/completions",
      "body": {
        "model": "my-model",
        "messages": [
          {"role": "system", "content": "..."},
          {"role": "user", "content": "..."}
        ],
        "max_tokens": 8192,
        "temperature": 0,
        "chat_template_kwargs": {"enable_thinking": false}
      }
    }
  }' | jq -r .id)

curl -s https://api.runpod.ai/v2/$EP/status/$JOB \
  -H "Authorization: Bearer $RUNPOD_API_KEY" | jq .status

Poll /status until it returns COMPLETED. Use the route + body form: the worker passes body to vLLM unchanged. The shorthand form (messages + sampling_params at the top of input) copies only sampling_params, so a top-level chat_template_kwargs is dropped.

The worker’s DEFAULT_CHAT_TEMPLATE_KWARGS setting had no effect on our version, and the model started emitting thinking text. Send chat_template_kwargs on every request, and test the raw output once after every worker upgrade.

Measured on our endpoint:

Metric	Value
Cold start (container, 19.3 GB load, graph capture)	3 min 28 s
Time to first token, 22-token prompt	0.8 s
Time to first token, 8k-token prompt	2.3 s
Decode, one stream	31.5–31.9 tokens/s

A 4,000-token scene at 32 tokens/s takes about two minutes. What a token price means on your own GPU turns these numbers into dollars per video.

10. What we do next

1. Score renders, not tokens. An automatic check on rendered frames for overlaps, off-canvas elements and keyframes that miss their target. Use it to compare checkpoints and decoding settings.
2. Constrain decoding. vLLM’s structured outputs and llama.cpp grammars make malformed JSON impossible without retraining.
3. Train on preferences. Sample several scenes per brief, score the renders, train on better-versus-worse pairs with DPO. Cross-entropy imitates the reference answer; a preference pair penalises the overlap itself. Render checks still gate the output.

Link copied