Local / 2026
Local environment & configuration.
Two machines and two operating systems. The workstation is the one that runs the models, so its numbers are recorded rather than estimated: what fits, what it decodes at, and which cap is actually in the way.
01 / Local environment
Two machines, one workflow.
A portable studio and a used-parts workstation. The Mac is where the experiments are made; the 2080 Ti pair is where the models come home to.
| Machine | Compute | Memory | Storage | OS | Role |
|---|---|---|---|---|---|
| MacBook ProThe quiet half | Apple silicon | 48 GB unified | 1 TB SSD | macOS | Design, code and writing; daily carry. |
| WorkstationThe loud half | 2 × RTX 2080 Ti · 22 GB each · NVLink NV2 | 44 GB VRAM + 94 GB DDR4 | 1 TB NVMe + 3 TB HDD | Ubuntu 24.04.5 LTS | Local inference, video generation and heat. |
22 GB each, 44 GB total. NVLink NV2 with two bonded links at 25.78 GB/s, so a tensor split is not paying PCIe. But the cards sit on different NUMA nodes, and the host thread that launches a kernel costs more than the link does.
Text decode is power-capped: both cards sit pinned at 180 W at 100% utilisation, so clocks stop before memory bandwidth does. Video generation is a bandwidth-and-synchronisation problem instead.
Good: a 27B FP8 model at 256K context, 20B video at 640×352, embeddings, classification, small LoRA runs. Not: 70B at usable context, batch > 1 at 256K, or native 1344×768 video without a chunked pipeline.
Built from used parts: 2 × Xeon E5-2696 v4 (44C / 88T), 2 × RTX 2080 Ti at 22 GB, 94 GB DDR4, one 1 TB NVMe. Still cheaper than renting.
02 / Text stack
What the model actually decodes at.
The serving configuration below is the one that ran the benchmarks: a 27B model across both cards with a 256K context, on the driver the workstation boots today.
| Item | Value |
|---|---|
| Runtime | vLLM 0.29.1rc0 SM75 fork · PyTorch 2.13 · CUDA 13.0 |
| Model | Qwen3.8-27B-FP8 · W8A16 (FP8 weights, FP16 activations) · FP8 KV cache |
| Parallelism | TP=2 across both cards · MTP/3 speculative decoding |
| Context | 262,144 tokens · GPU KV cache holds 276,788 |
| System | Ubuntu 24.04.5 · kernel 7.0.0-43 · driver 580.178.04 (open module) |
| Power cap | 180 W per card (default 250 W, maximum 280 W) |
NoteMedians of five or more runs, warmup discarded, at the 180 W cap. The fork's published ~80 tok/s is a synthetic best case with high speculative acceptance; real MTP acceptance is lower, and a power limit penalises MTP because verifying several draft tokens per step is compute-heavy.
MTP/2 and MTP/3 are statistically the same here (50.8 vs 51.2 tok/s off; 46.1 vs 45.6 on). The draft count is not the bottleneck — the power cap, memory bandwidth and the Triton fallback that SM75 needs for decode are.
03 / Video stack
A 20B video model on 22 GB cards.
MiniMax H3 is 20.11B parameters over 50 blocks. It does not fit on one card, so the second card is not spare memory — it has to take half the work, or the run is just slower.
| Run | Per step | Clip | Peak VRAM | Peak temp |
|---|---|---|---|---|
| Single card · 640×352 | ~51 s | 243 s | 19.1 GB | 78 °C for 4 min |
| TP2 · 640×352 | ~14 s | 82 s | 11.6 + 19.7 GB | 75 °C for 1.4 min |
| TP2 · 864×480 | 19.8 s | — | 19.7 GB | — |
Speedup2.97× on the smoke sample, 2.25× on the 864×480 film. Both measured against the same single-card workflow; numbers from the repository's own performance log.
The first TP2 attempt only reached 1.31×. The sharding was correct; the host thread was the problem. The W4A8 GEMM blocks the thread that launches it, so with a single launch thread the two cards' GEMMs serialise — measured overlap 5%. One host thread per rank: 91% overlap, 2.25×, output bit-identical.
04 / Caveats
What these numbers are not.
Every figure above comes from the workstation, at this power cap, on this driver. Remove one of those and the number changes.
One cap, one driver
The 180 W limit is a choice, not a hardware limit. Raising it to 250 W raises decode; the temperatures above are what keeps it at 180.
Not a leaderboard
These are single-machine medians on a used two-socket Xeon. They are useful for deciding what to run here, not for comparing against a datacentre.
Watched by the workstation
The same NVML fields behind these numbers are read by sensorium, the terminal monitor in the experiments list — no nvidia-smi subprocess in the loop.