Local / text stack
27B, 256K, 180 W.
The reference page for the text model that runs here: the exact serving configuration, the throughput it produces, and the one number that explains the gap to the documentation.
01 / Configuration
The resolved serving profile.
Not a suggested config: this is the profile the benchmark numbers were produced with, on the driver the workstation boots today.
| Setting | Value |
|---|---|
| Model | Qwen3.8-27B-FP8 · W8A16 (FP8 weights, FP16 activations) |
| Quantisation | fp8 e4m3 · weight block 128×128 · dynamic activations |
| KV cache | FP8 · 276,788 tokens resident |
| Context | 262,144 tokens (256K) |
| Parallelism | TP=2 × PP=1 · 2 ranks on 2 cards |
| Speculative decoding | MTP / 3 draft tokens |
| Attention / linear attn | FLASHINFER · FlashQLA legacy prefill (SM75) · Triton GDN decode fallback |
| CUDA graphs | PIECEWISE · capture size 4 |
| Prefix caching | enabled |
| Sequences / tokens | max_num_seqs 1 · batched tokens 2,048 |
| GPU memory utilisation | 0.94 — the desktop holds ~740 MiB on GPU0 |
| Power cap | 180 W per card (default 250 W, maximum 280 W) |
| Software stack | vLLM 0.29.1rc0 SM75 fork · PyTorch 2.13+cu130 · Triton 3.7.1 · CUDA 13.0.88 · driver 580.178.04 |
| OS | Ubuntu 24.04.5 LTS · kernel 7.0.0-43 |
02 / Throughput
Decode, prefill, and the cost of thinking.
Streaming requests, TTFT separated from decode, fixed prompt and output length, median of three or more runs with warmup discarded.
| Decode test | Run 1 | Run 2 | Run 3 | Median |
|---|---|---|---|---|
| Thinking on · 512 tokens | 47.5 | 45.9 | 46.1 | 46.1 tok/s |
| Thinking off · 512 tokens | 52.1 | 52.9 | 51.1 | 52.1 tok/s |
| Prompt tokens | TTFT | Prefill throughput |
|---|---|---|
| 74 | 122 ms | ~600 tok/s (overhead-dominated) |
| 1,170 | 940 ms | 1,244 tok/s |
| 4,500 | 2,275 ms | 1,978 tok/s |
| 17,830 | 13,167 ms | 1,354 tok/s |
03 / Ablation
MTP/2 vs MTP/3.
Six runs per cell at the same power cap. The difference is smaller than the run-to-run spread, which is the actual result.
| Draft tokens | Thinking | Median | Min | Max | Stdev |
|---|---|---|---|---|---|
| MTP/2 | off | 50.8 | 49.4 | 51.7 | 0.88 |
| MTP/2 | on | 46.1 | 43.7 | 50.4 | 2.41 |
| MTP/3 | off | 51.2 | 45.5 | 56.8 | 4.27 |
| MTP/3 | on | 45.6 | 42.6 | 49.4 | 2.34 |
ReadMTP/2 and MTP/3 are statistically indistinguishable here. The profile stays at MTP/3 because there is no measured reason to change it, not because it is faster.
04 / Power
Both cards pinned at the cap.
The 180 W limit is a deliberate choice, and it is the single largest term in the gap between measured decode and the fork's published figure.
| GPU | Peak draw | Average draw | Limit | Max SM clock | Utilisation |
|---|---|---|---|---|---|
| 0 | 182 W | 178 W | 180 W | 1,845 MHz | 100% |
| 1 | 179 W | 175 W | 180 W | 1,830 MHz | 100% |
NoteUtilisation is 100% while clocks cap at the limit: this is a power budget, not a thermal one. Raising the cap to 250 W raises decode and the room temperature.
05 / Reproduce
Bringing it back up.
The model volume is an NTFS partition that does not auto-mount. Two commands, then the server.
udisksctl mount -b /dev/nvme0n1p4 # model volume, not auto-mounted
cd ~/vllm-2080ti && ./llm.sh start # detached vLLM server on loopback
./llm.sh status # server, model and GPU overview
./llm.sh bench # throughput benchmark
./llm.sh stop # stop the model, leave the workstation on
06 / Caveats
What these numbers assume.
Every figure on this page is conditional on three things: the power cap, the driver, and the context length.
One cap, one driver
180 W per card and driver 580.178.04. Change either and the decode figure moves; prefill barely does.
256K is not free
The KV cache holds 276,788 tokens, so a full 256K context runs at about 1.06× concurrency. Long context here means one request at a time.
One machine
These are medians from a used two-socket Xeon with SM75 cards. Useful for deciding what to run here, not for a datacentre comparison.