Split a 27B FP8 model in half across two 22GB cards, turn on a 256K context, turn on MTP speculative decoding. The server comes up, it talks, it calls tools, the KV cache fits. Then I run the benchmark and get 46 tok/s.

The fork's docs say 80. That is a big enough gap to investigate — because “local models are slow” and “my configuration is wrong” are two completely different conclusions, and they look the same.

Three numbers

Configuration first, otherwise the numbers mean nothing:

ItemValue
Model / precisionQwen3.8-27B-FP8 · W8A16 (FP8 weights, FP16 activations)
Parallelism / speculativeTP=2 · MTP/3 · FLASHINFER attention
Context / KV262,144 tokens · GPU KV cache 276,788 tokens
VRAM footprintGPU0 21.5 GB / GPU1 21.1 GB
Power cap180 W (default 250 W, maximum 280 W)
Software stackvLLM 0.29.1rc0 SM75 fork · PyTorch 2.13+cu130 · CUDA 13.0 · driver 580
46.1tok/s · thinking on
52.1tok/s · thinking off
1,978tok/s prefill @4K
276,788tokens KV cache

46 and 52 are both medians over three or more runs with warmup discarded, not the best run.

The cost of thinking

“Thinking” mode (a reasoning trace) decodes about 13% slower: 52.1 → 46.1. The reason has to do with MTP — speculative decoding depends on the draft token acceptance rate, and a reasoning trace is less predictable than prose, so drafts get rejected more often and the average number of tokens per step drops.

That gap may be invisible on other machines; on this one it is very stable, because it is not bounded by bandwidth, it is bounded by compute per step (next section).

Why not 80

Two reasons, neither mysterious.

First, that 80 is a synthetic benchmark. The fork's docs say so themselves: both inputs are synthetic samples built for high speculative acceptance, used to compare throughput under equal conditions, not representative of real inference. Real draft acceptance is lower, so each step emits fewer tokens.

Second, the workstation is held at 180W. MTP/3 verifies three draft tokens per step, a textbook compute-bound load — exactly the kind the power wall is least forgiving about. Under load both cards sit pinned at 180W (peaks 182W / 179W), utilization 100%, SM clocks capped at 1845 / 1830 MHz. Meaning: the cards are not below their clocks because of heat, but because of power.

Pinned at 180W

GPUPeakMeanCapMax SM clockUtilization
0182 W178 W180 W1,845 MHz100%
1179 W175 W180 W1,830 MHz100%

180W is not a hardware limit: the default cap is 250W and the maximum is 280W. It is a policy I set myself with nvidia-smi -pl plus restore-on-boot, because two full-power cards warm this room within half an hour.

This is a textbook trade: predictability (quiet, cool, no tripped breaker) in exchange for throughput. I write it down because the next time you see a figure like “local 27B at 46 tok/s”, the first question should be about the power cap, not about quantization.

MTP/2 and MTP/3

Intuition says more draft tokens means faster. Measured, it does not:

DraftsThinkingMedianMinMaxStd dev
MTP/2off50.849.451.70.88
MTP/2on46.143.750.42.41
MTP/3off51.245.556.84.27
MTP/3on45.642.649.42.34

The difference is 0.4–0.5 tok/s, an order of magnitude smaller than the spread between runs. MTP/2 and MTP/3 are statistically indistinguishable on the workstation.

The MTP/3 “thinking off” group also exposes a sampling trap: six runs drifting from 56.8 all the way down to 45.5, the first few a cold-card gift, the steady state at 50–51. MTP/2 has σ 0.88 and settles to the same steady state quickly. So I left the configuration on the default MTP/3 — not because it is faster, but because there is no reason to change it.

Prefill is fine

If the power wall were the only cause, prefill should sag too. It does not:

Prompt lengthTime to first tokenPrefill throughput
74122 ms~600 tok/s (overhead-dominated)
~1K940 ms1,244 tok/s
~4K2.3 s1,978 tok/s
~16K13.2 s1,354 tok/s

The fork's reference prefill for this model is 1,409–1,653 tok/s; we get 1,978 at 4K, above it. That is a good counter-check on the earlier explanation: prefill is less sensitive to “a few extra tokens of compute per step”, and so less dependent on power headroom.

What it does not prove

These numbers do not show that a 2080 Ti is unsuited to local inference, and they do not show that MTP is useless — they show that at 180W, on this driver, on this fork, a 27B FP8 model reliably gives 46 tok/s, a 256K context, a 21GB-class VRAM footprint, and working tool calls.

They also cannot be compared directly with “a Mac with 48GB unified memory” or “a single 3090”, because context length, quantization and power cap all differ. To compare, you have to pin those three variables first.

The most useful conclusion is small: when you report tok/s, you have to report the power cap and the context length at the same time, otherwise the number describes only an unconstrained ideal case.

Method / Data comes from measured records in vllm-2080ti/RESULTS.md on the workstation: streaming requests, TTFT separated from decode by first-token time, fixed prompt and output lengths, at least five runs per item with median plus min/max/stdev, and one warmup discarded after every configuration change. The full configuration table is on the site's Local page.