What a token price means on your own GPU
How to turn $/million-token API prices into GPU-hours: prefill vs decode, the roofline math, KV-cache sizing, batching, serverless cold starts, and what an API gives you that a rented GPU does not. Worked with real numbers from our Qwen3.5-9B deployment.
When does a rented GPU beat an API? This post does the math for one case: a fine-tuned Qwen3.5-9B serving our video pipeline on RunPod.
Every number below is either measured on our endpoint, read from a live price sheet on 28 October 2026, or derived with the formula next to it. Estimates are labelled as estimates.
Build your intuition
Qwen3.5-9B runs on one RTX A6000 with 768 GB/s memory bandwidth. Serverless rent is $1.22/h.
| Decode | Numbers |
|---|---|
| Reading weights sets the step time | Read 18 GB per step: 23.4 ms, rounded to 24 ms; math per token: ~0.1 ms. |
| Batching shares the weight read | 32 requests share one 18 GB read per step. Each request also reads its own KV cache. |
| Each step adds one token per request | 32 requests produce 32 tokens per step. After 100 steps, each answer has 100 new tokens: 3,200 total. |
| 1 request | 31.5 tok/s measured; ~43 tok/s ceiling. Output costs $10.76/MTok. |
| 32 requests, each at 8k context | KV reads add 12 ms: 24 + 12 = 36 ms per step. At 74% efficiency, estimate ~660 tok/s total and ~$0.50/MTok, matching Haiku output. |
| KV memory limits the batch | 22 GiB fits ~80 requests at 8k context or ~14 at 49k. |
1. Read the price sheet
API prices are per million tokens (MTok), split into input and output:
| Model | Input | Cache hit | Output | Batch API (input / output) |
|---|---|---|---|---|
| Claude Opus 5.5 | $4 | $0.20 | $20 | $2 / $10 |
| Claude Sonnet 5.5 | $2 | $0.10 | $10 | $1 / $5 |
| Claude Haiku 5.5 (prompt ≤ 100k) | $0.10 | $0.01 | $0.50 | $0.05 / $0.25 |
A cache write costs 1.25× input for a 5-minute TTL and 2× for a 1-hour TTL. The Batch API halves both directions and returns results asynchronously.
Output costs 5× input because the hardware does two different jobs.
- Prefill (input) processes all prompt tokens in one parallel pass. It is matrix-matrix work. The GPU is compute-bound, and thousands of tokens go through per second.
- Decode (output) produces one token per forward pass. Every pass reads all the weights from GPU memory to make one token per sequence. It is matrix-vector work. The GPU is memory-bandwidth-bound, and its tensor cores sit mostly idle.
Decode is where the GPU time goes.
2. Price your own GPU in the same unit
One formula converts a GPU-hour into a token price:
$/MTok = ($/hour) / (tokens per second × 3600) × 1,000,000
The price is fixed by the provider. Tokens per second is the only number you control. Prices on RunPod, read from their API and pricing page on 28 October 2026:
| GPU (VRAM) | Bandwidth | Pod, secure | Pod, community | Serverless (flex) |
|---|---|---|---|---|
| RTX A6000 / A40 (48 GB) | 768 / 696 GB/s | $0.59/h | $0.33–0.35/h | $1.22/h |
| L40S / RTX 6000 Ada (48 GB) | 864 / 960 GB/s | $1.09 / $0.99/h | $0.79 / $0.74/h | $1.75/h |
| A100 80 GB | ~2.0 TB/s | $1.79/h | $1.19–1.39/h | $2.72/h |
| H100 SXM (80 GB) | 3.35 TB/s | $3.99/h | $2.69/h | $4.79/h |
Serverless costs about 2× a secure pod per hour. You pay that premium for scale-to-zero.
3. Measure decode speed, then check it against the roofline
For one stream, decode speed has a hard ceiling: memory bandwidth ÷ bytes read per token. For a dense or hybrid model in bf16, that is roughly the weight size.
Qwen3.5-9B has 9.65 B parameters, 19.3 GB in bf16. The language-model path is about 18 GB of that; the vision encoder is not read during text decode.
ceiling = 768 GB/s / 18 GB ≈ 43 tokens/s (RTX A6000)
ceiling = 696 GB/s / 18 GB ≈ 39 tokens/s (A40)
We measured 31.5–31.9 tokens/s single-stream on our endpoint. That is 74–82% of the ceiling, which is normal for a well-configured server. If you measure 40% of the ceiling, something is wrong: eager mode, a slow kernel path or a CPU bottleneck.
Two consequences:
- An H100 is not a cheaper way to serve one stream. Its ceiling is ~186 tokens/s at 6.8× the price of an A6000 pod. At the same 74% efficiency, one stream costs about $8/MTok on an H100 and $5.20/MTok on an A6000 pod.
- Quantization speeds decode almost linearly. FP8 weights halve the bytes per token, so the ceiling roughly doubles.
Prefill comes from time-to-first-token. Our endpoint returned the first token in 0.8 s for a 22-token prompt and 2.3 s for an 8k-token prompt. The marginal rate is (8,192 − 22) / (2.3 − 0.8) ≈ 5,400 prompt tokens/s.
Put both into the formula on a $1.22/h serverless worker:
input: 1.22 / (5,400 × 3600) × 1e6 ≈ $0.06 / MTok
output: 1.22 / (31.5 × 3600) × 1e6 ≈ $10.76 / MTok
On our own GPU, output costs 170× input, not 5×. Large decode batches close that gap. That is the next lever.
4. Batching is the whole game
A decode step reads the weights once, no matter how many sequences are in the batch. With B sequences, the step reads the weights once plus each sequence’s KV cache:
step_time ≈ (weight_bytes + B × per_sequence_cache_bytes) / bandwidth
throughput ≈ B / step_time
For Qwen3.5-9B on an RTX A6000, with an 8k-token average context (per-sequence cache from section 5, efficiency fixed at the measured 74%):
| Streams | Read per step | Total tok/s | Serverless $/MTok | Secure pod $/MTok |
|---|---|---|---|---|
| 1 | 18.3 GB | 31.5 (measured) | $10.76 | $5.20 |
| 8 | 20.4 GB | ~220 | ~$1.50 | ~$0.75 |
| 32 | 27.4 GB | ~660 | ~$0.50 | ~$0.25 |
At 32 streams the step still needs only ~3.8 ms of compute (32 × 2 × 9 GFLOP at ~150 TFLOPS) against ~36 ms of memory traffic. Decode stays memory-bound well past any concurrency a small team will see.
These rows are estimates. Measure your own before you price anything:
vllm bench serve --model /models/my-model-merged \
--dataset-name random --random-input-len 3400 --random-output-len 2000 \
--max-concurrency 8 --num-prompts 64
Run it at concurrency 1, 8 and 32. Read Output token throughput and plug it into the formula.
The catch: batching needs traffic. A batch of 32 needs 32 requests in flight at the same time. An API provider gets that from millions of users. You get it only if your own workload is parallel. Ours is: a video’s scenes build independently, so one video can fill a batch of 5–10.
5. The KV cache sets your maximum batch
Every token in every live sequence keeps its keys and values in GPU memory. For a standard attention layer:
KV bytes per token = 2 (K and V) × layers × kv_heads × head_dim × bytes_per_value
Qwen3.5-9B is a hybrid. Only 8 of its 32 layers are full attention (4 KV heads × 256 dims). The other 24 are Gated DeltaNet linear-attention layers, which keep a fixed-size state per sequence instead of a growing cache.
full attention: 2 × 8 × 4 × 256 × 2 bytes = 32 KiB per token
linear state: 24 layers × 32 heads × 128 × 128 × 2 bytes ≈ 25 MB per sequence (+ ~1 MB conv state)
A dense 8B model with 32 full-attention layers and 8 KV heads × 128 dims needs 128 KiB per token: four times more.
The budget on a 48 GB card with GPU_MEMORY_UTILIZATION=0.90:
43.2 GiB usable (0.90 × 48 GiB)
− 18.0 GiB weights (19.3 GB)
− ~3 GiB activations, CUDA graphs, allocator slack
≈ 22 GiB KV cache → ~700,000 tokens of full-attention cache
| Context per sequence | Cache per sequence | Sequences that fit |
|---|---|---|
| 8,192 | 256 MiB + 26 MB | ~80 |
49,152 (our MAX_MODEL_LEN) |
1.5 GiB + 26 MB | ~14 |
Three rules follow:
MAX_MODEL_LENis a memory reservation, not a feature. vLLM refuses to start if one sequence of that length does not fit. Set it to your longest real prompt plus your longest real answer, not to the model’s 262k maximum.- Long context costs concurrency first, then throughput. A 49k-token conversation holds the memory of six 8k-token conversations for its whole lifetime. Each decode step also reads its whole cache in the 8 attention layers, so per-token speed drops as context grows.
- Architecture changes the economics. A hybrid model fits 4× more concurrent context than a dense model of the same size. Check
num_key_value_heads,head_dimand the layer types inconfig.jsonbefore you choose a base model.
6. A worked example: one video
One Clipwise video is one plan call and one build call per scene. The token counts below are medians and means from our training data, for a 5-scene video:
| Call | Input tokens | Output tokens |
|---|---|---|
| Plan × 1 | 340 | 2,042 |
| Build × 5 | 5 × 3,380 | 5 × 1,981 |
| Total | ~17,200 | ~11,900 |
| Option | Cost per video | How |
|---|---|---|
| Opus 5.5 | $0.31 | 17.2k × $4 + 11.9k × $20 per MTok |
| Opus 5.5, Batch API | $0.15 | half price, asynchronous |
| Sonnet 5.5 | $0.15 | 17.2k × $2 + 11.9k × $10 |
| Haiku 5.5 | $0.008 | 17.2k × $0.10 + 11.9k × $0.50 |
| Our 9B, serverless, warm, sequential | $0.13 | ~387 s × $1.22/h |
| Our 9B, serverless, cold, then idle | $0.30 | + 208 s cold start + 300 s idle timeout |
| Our 9B, serverless, warm, scenes in parallel | ~$0.05 (estimate) | ~140 s of GPU time |
Two caveats change this table:
- A general model needs the grammar in its prompt. Our fine-tuned model learned it. If a general model needs, say, a 10k-token schema prompt on all six calls, Opus 5.5 adds $0.24 per video uncached, or about $0.06 with caching (one 5-minute write at $5/MTok, five reads at $0.20/MTok). Caching turns that from a blocker into noise.
- The cold-start row is the honest one for low traffic. A single video on a scaled-to-zero endpoint costs the same as Opus 5.5, because you pay for the cold start and the idle tail too.
Break-even for a 24/7 pod (RTX A6000 secure, $0.59/h × 730 h = $431/month):
| Against | Videos per month to break even |
|---|---|
| Opus 5.5 | ~1,400 |
| Sonnet 5.5 | ~2,800 |
| Haiku 5.5 | ~56,000 |
To beat Haiku, the pod must sustain ~258 output tokens/s, 24 hours a day (55,953 videos × 11,947 tokens ÷ 2.59 M s in a 30-day month):
| Streams | Total tok/s | Videos/month, max | vs Haiku |
|---|---|---|---|
| 8 | ~220 | ~48,000 | loses |
| 32 | ~660 | ~143,000 | wins, if 32 requests run all day |
Real traffic has nights and gaps. Against Haiku, we self-host for control: a model trained on our format, data on our hardware, no vendor change overnight. Savings need sustained parallel traffic.
7. What the API gives you that your GPU does not
The per-token price includes work you now own:
- Global batching. The provider fills large batches from all customers. That is how decode reaches $20/MTok on a frontier model and $0.50 on a small one. You batch only your own traffic.
- Fleet-wide prompt caching. A cache hit costs 5% of input on Opus 5.5. vLLM has prefix caching too, but only per worker and only while the worker is alive. Scale-to-zero discards it.
- No cold start, and elastic capacity. The API answers the first request in seconds and absorbs a spike up to your rate limit. Your endpoint answers in minutes from zero and queues behind
workersMax. - Uptime and incidents. GPU shortages in your data center, driver regressions, a worker image update that breaks a flag: all yours now.
- Model upgrades. A new API model is a string change. A new open model is a new training run, a new eval and a new deployment.
- Features beyond tokens. Structured outputs, tool use, 1M-token context, safety classifiers and data-residency options ship with the API. On your server you add each one yourself, or go without.
8. Optimizations for self-hosting, ordered by payoff
- Concurrency first. Send independent requests in parallel and let vLLM batch them. Raise
MAX_CONCURRENCYon the RunPod worker (jobs per worker) and--max-num-seqsin vLLM. Nothing else comes close. - Keep CUDA graphs on.
ENFORCE_EAGER=truemakes startup faster and decode slower, because every step launches each kernel from Python. We used it early to shorten cold starts and removed it later. - Prefix caching for shared prompts.
--enable-prefix-caching(on by default in current vLLM) reuses the KV cache of an identical prompt prefix. Put stable content first: system prompt, then schema, then the per-request data. - Quantize the weights. FP8 weights halve decode bytes. Ada and Hopper GPUs run FP8 natively; on Ampere (A40, A6000), vLLM runs FP8 weights through weight-only kernels, which still cut memory traffic. Re-run your eval after quantizing. JSON-heavy outputs show quality loss early.
-
Speculative decoding for copy-heavy outputs. A revise call returns a scene that is mostly the input scene. N-gram prompt lookup proposes the next tokens from the prompt itself and the model verifies several per pass:
--speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 4}'It helps most at low concurrency, where the GPU has spare compute. At high concurrency it competes with the batch.
- Write less. Output length decides latency and cost. A tighter schema with fewer redundant fields beats any GPU upgrade.
--max-num-batched-tokens controls chunked prefill: how many prompt tokens join one step. Higher values give faster time-to-first-token for long prompts. Lower values give smoother decode for the streams already running. Our first deployment used 8192.
9. Serverless traps
Each of these cost us time or money on RunPod Serverless:
- Cold start is minutes, and you pay for it. Ours is 3 min 28 s: container start, 19 GB of weights from a network volume, and CUDA graph capture. Billing runs from worker start, not from first token.
- The idle timeout is billed. A worker waits
idleTimeoutseconds for the next job before it stops. At 300 s it adds $0.10 per burst on a $1.22/h worker. At 5 s you pay for more cold starts. Set it from your traffic gap distribution. - FlashBoot helps only a warm fleet. It keeps recently stopped workers ready for a fast restart. After a long quiet period, you get a full cold start.
workersMin=1is a 24/7 pod at serverless prices. At $1.22/h that is ~$890/month, double a secure pod with the same GPU. If you need always-on, rent a pod.- The network volume pins your region. Workers can only start in the volume’s data center. If that region runs out of your GPU type, jobs queue. We widened the pool to five 48 GB types (A40, RTX A6000, RTX 6000 Ada, L40, L40S). That fixed capacity, but the Ada and L40-class cards bill $1.75/h instead of $1.22/h, so the same job can cost 43% more depending on which card it lands on.
- Use the async route. The synchronous
/runsyncand OpenAI-compatible routes held the HTTP connection through a cold start and timed out. Submit to/run, keep the job ID, poll/status/{id}. Make the submission idempotent so a retry does not start a second paid job. - Worker defaults do not always apply.
DEFAULT_CHAT_TEMPLATE_KWARGShad no effect on our worker version, and the model started thinking. Sendchat_template_kwargson every request and test the raw output. - Set
executionTimeoutabove your longest job. A 7,400-token answer at 31.5 tokens/s takes about 4 minutes, plus a cold start. Ours is 900 s.
10. The decision rule
- Prototype on the API. Use caching and the Batch API before you think about GPUs.
- Fine-tune when the format is the problem, not the price. That is why we trained the 9B.
- Serve on serverless while traffic is bursty and low. Accept the cold start or pay for a warm worker during working hours.
- Move to a pod when the GPU is busy more than ~half the day. Below that, serverless’s per-second billing wins; above it, the pod’s 2× lower hourly price wins.
- Measure tokens per second at your real concurrency. It is the only variable in the formula you control, and it moves the answer by 10×.