Norwin YuAI usage research
中文 AI usage research

Local / 2026

Local environment & configuration.

Two machines and two operating systems. The workstation is the one that runs the models, so its numbers are recorded rather than estimated: what fits, what it decodes at, and which cap is actually in the way.

02 machines44 GB VRAM94 GB DDR4Snapshot / Nov 2026

01 / Local environment

Two machines, one workflow.

A portable studio and a used-parts workstation. The Mac is where the experiments are made; the 2080 Ti pair is where the models come home to.

Computer specifications
MachineComputeMemoryStorageOSRole
MacBook ProThe quiet halfApple silicon48 GB unified1 TB SSDmacOSDesign, code and writing; daily carry.
WorkstationThe loud half2 × RTX 2080 Ti · 22 GB each · NVLink NV244 GB VRAM + 94 GB DDR41 TB NVMe + 3 TB HDDUbuntu 24.04.5 LTSLocal inference, video generation and heat.
Two cards, one pool

22 GB each, 44 GB total. NVLink NV2 with two bonded links at 25.78 GB/s, so a tensor split is not paying PCIe. But the cards sit on different NUMA nodes, and the host thread that launches a kernel costs more than the link does.

Where the ceiling is

Text decode is power-capped: both cards sit pinned at 180 W at 100% utilisation, so clocks stop before memory bandwidth does. Video generation is a bandwidth-and-synchronisation problem instead.

Good at / not good at

Good: a 27B FP8 model at 256K context, 20B video at 640×352, embeddings, classification, small LoRA runs. Not: 70B at usable context, batch > 1 at 256K, or native 1344×768 video without a chunked pipeline.

Built from used parts: 2 × Xeon E5-2696 v4 (44C / 88T), 2 × RTX 2080 Ti at 22 GB, 94 GB DDR4, one 1 TB NVMe. Still cheaper than renting.

02 / Text stack

What the model actually decodes at.

The serving configuration below is the one that ran the benchmarks: a 27B model across both cards with a 256K context, on the driver the workstation boots today.

Local text inference configuration and measured throughput
ItemValue
RuntimevLLM 0.29.1rc0 SM75 fork · PyTorch 2.13 · CUDA 13.0
ModelQwen3.8-27B-FP8 · W8A16 (FP8 weights, FP16 activations) · FP8 KV cache
ParallelismTP=2 across both cards · MTP/3 speculative decoding
Context262,144 tokens · GPU KV cache holds 276,788
SystemUbuntu 24.04.5 · kernel 7.0.0-43 · driver 580.178.04 (open module)
Power cap180 W per card (default 250 W, maximum 280 W)
46tok/s decode · thinking on
52tok/s decode · thinking off
1,978tok/s prefill at 4K
21.5 / 21.1GB VRAM at 256K

NoteMedians of five or more runs, warmup discarded, at the 180 W cap. The fork's published ~80 tok/s is a synthetic best case with high speculative acceptance; real MTP acceptance is lower, and a power limit penalises MTP because verifying several draft tokens per step is compute-heavy.

MTP/2 and MTP/3 are statistically the same here (50.8 vs 51.2 tok/s off; 46.1 vs 45.6 on). The draft count is not the bottleneck — the power cap, memory bandwidth and the Triton fallback that SM75 needs for decode are.

03 / Video stack

A 20B video model on 22 GB cards.

MiniMax H3 is 20.11B parameters over 50 blocks. It does not fit on one card, so the second card is not spare memory — it has to take half the work, or the run is just slower.

MiniMax H3 measured on one card versus two-card tensor parallelism
RunPer stepClipPeak VRAMPeak temp
Single card · 640×352~51 s243 s19.1 GB78 °C for 4 min
TP2 · 640×352~14 s82 s11.6 + 19.7 GB75 °C for 1.4 min
TP2 · 864×48019.8 s—19.7 GB—

Speedup2.97× on the smoke sample, 2.25× on the 864×480 film. Both measured against the same single-card workflow; numbers from the repository's own performance log.

The first TP2 attempt only reached 1.31×. The sharding was correct; the host thread was the problem. The W4A8 GEMM blocks the thread that launches it, so with a single launch thread the two cards' GEMMs serialise — measured overlap 5%. One host thread per rank: 91% overlap, 2.25×, output bit-identical.

04 / Caveats

What these numbers are not.

Every figure above comes from the workstation, at this power cap, on this driver. Remove one of those and the number changes.

01

One cap, one driver

The 180 W limit is a choice, not a hardware limit. Raising it to 250 W raises decode; the temperatures above are what keeps it at 180.

02

Not a leaderboard

These are single-machine medians on a used two-socket Xeon. They are useful for deciding what to run here, not for comparing against a datacentre.

03

Watched by the workstation

The same NVML fields behind these numbers are read by sensorium, the terminal monitor in the experiments list — no nvidia-smi subprocess in the loop.