Norwin YuAI usage research
中文 AI usage research

Local / text stack

27B, 256K, 180 W.

The reference page for the text model that runs here: the exact serving configuration, the throughput it produces, and the one number that explains the gap to the documentation.

Qwen3.8-27B-FP8262,144 tokensTP=2 · MTP/3Measured / Nov 2026

01 / Configuration

The resolved serving profile.

Not a suggested config: this is the profile the benchmark numbers were produced with, on the driver the workstation boots today.

Text inference serving configuration
SettingValue
ModelQwen3.8-27B-FP8 · W8A16 (FP8 weights, FP16 activations)
Quantisationfp8 e4m3 · weight block 128×128 · dynamic activations
KV cacheFP8 · 276,788 tokens resident
Context262,144 tokens (256K)
ParallelismTP=2 × PP=1 · 2 ranks on 2 cards
Speculative decodingMTP / 3 draft tokens
Attention / linear attnFLASHINFER · FlashQLA legacy prefill (SM75) · Triton GDN decode fallback
CUDA graphsPIECEWISE · capture size 4
Prefix cachingenabled
Sequences / tokensmax_num_seqs 1 · batched tokens 2,048
GPU memory utilisation0.94 — the desktop holds ~740 MiB on GPU0
Power cap180 W per card (default 250 W, maximum 280 W)
Software stackvLLM 0.29.1rc0 SM75 fork · PyTorch 2.13+cu130 · Triton 3.7.1 · CUDA 13.0.88 · driver 580.178.04
OSUbuntu 24.04.5 LTS · kernel 7.0.0-43

02 / Throughput

Decode, prefill, and the cost of thinking.

Streaming requests, TTFT separated from decode, fixed prompt and output length, median of three or more runs with warmup discarded.

Decode throughput
Decode testRun 1Run 2Run 3Median
Thinking on · 512 tokens47.545.946.146.1 tok/s
Thinking off · 512 tokens52.152.951.152.1 tok/s
Prefill latency and throughput
Prompt tokensTTFTPrefill throughput
74122 ms~600 tok/s (overhead-dominated)
1,170940 ms1,244 tok/s
4,5002,275 ms1,978 tok/s
17,83013,167 ms1,354 tok/s

03 / Ablation

MTP/2 vs MTP/3.

Six runs per cell at the same power cap. The difference is smaller than the run-to-run spread, which is the actual result.

MTP draft-token ablation
Draft tokensThinkingMedianMinMaxStdev
MTP/2off50.849.451.70.88
MTP/2on46.143.750.42.41
MTP/3off51.245.556.84.27
MTP/3on45.642.649.42.34

ReadMTP/2 and MTP/3 are statistically indistinguishable here. The profile stays at MTP/3 because there is no measured reason to change it, not because it is faster.

04 / Power

Both cards pinned at the cap.

The 180 W limit is a deliberate choice, and it is the single largest term in the gap between measured decode and the fork's published figure.

Power and clocks under load
GPUPeak drawAverage drawLimitMax SM clockUtilisation
0182 W178 W180 W1,845 MHz100%
1179 W175 W180 W1,830 MHz100%

NoteUtilisation is 100% while clocks cap at the limit: this is a power budget, not a thermal one. Raising the cap to 250 W raises decode and the room temperature.

05 / Reproduce

Bringing it back up.

The model volume is an NTFS partition that does not auto-mount. Two commands, then the server.

udisksctl mount -b /dev/nvme0n1p4     # model volume, not auto-mounted
cd ~/vllm-2080ti && ./llm.sh start    # detached vLLM server on loopback
./llm.sh status                       # server, model and GPU overview
./llm.sh bench                        # throughput benchmark
./llm.sh stop                         # stop the model, leave the workstation on

06 / Caveats

What these numbers assume.

Every figure on this page is conditional on three things: the power cap, the driver, and the context length.

01

One cap, one driver

180 W per card and driver 580.178.04. Change either and the decode figure moves; prefill barely does.

02

256K is not free

The KV cache holds 276,788 tokens, so a full 256K context runs at about 1.06× concurrency. Long context here means one request at a time.

03

One machine

These are medians from a used two-socket Xeon with SM75 cards. Useful for deciding what to run here, not for a datacentre comparison.