The goal was specific: run a 125B-class MoE at 128K context on two 22GB 2080 Ti cards, at UD-IQ4_XS. It is an order of magnitude bigger than a 27B dense model, and it works by loading only the experts that get hit; the price is that every configuration change makes the engine read 35 to 55GB back into memory, one to three minutes.

So the question stopped being “which parameter is faster” and became “how should I schedule these dozen-odd restarts”. This note is about the second question, plus two hardware-related things I found along the way.

What three minutes means

Three minutes is not expensive by itself. What is expensive is how it changes the way you experiment:

  • change one parameter at a time → a dozen-odd restarts → forty minutes, waiting only;
  • decide to “just try one more switch” → another three minutes, and the baseline for this restart is already dirty;
  • discover halfway through that the measurement method is wrong → not an edit to the script, a full rerun.

So the first decision was not which parameter to pick, but fixing the measurement protocol so that the data from each restart is directly comparable: streaming requests; TTFT separated from decode by first-token time; fixed prompt and output lengths; at least five runs per item with median and min/max/stdev; and one warmup discarded after every configuration change.

The protocol was inherited from the previous project — the 27B work was measured the same way, including the numbers that the power cap explained away in 80 and 46. Without one protocol the restart cost doubles, because you have to come back and re-measure.

One restart, one group of tests

There is one orchestration rule: one engine configuration, one full group of tests. So the experiments are laid out in batches, not as a parameter table.

BatchConfigurationWhat this group answers
0Single card · 128K · images onBaseline: quality score and speed, the full case set
1Single card + BF16 tensor core switchWhat the 2080 Ti-specific optimization is actually worth (off/on alternating inside the batch)
2Single card vs layer split vs hand-set split pointsIs the second card extra capacity, or also faster
3Explicitly turn on the two optimizations the second card losesCan what was lost be recovered
4Two concurrent streamsIs total tok/s worth making each stream slower
5KV precision 8-bit / k8v4 / q4_0How many experts the saved VRAM buys, and how much quality drops
6Chunk size and reserveBest prefill chunk value
7Head-to-head vLLM on the same hardwareTradeoffs between the two architectures for agent work (different models, not a speed contest)

The value of this table is not that it is complete, it is that it admits the restart cost: every extra batch is another three minutes; so merge what can be merged, and alternate inside a batch what can be alternated.

Side note

For the concurrency batch, the docs give 82 tok/s for one stream and 89.2 combined for two (+8.8%), but each stream drops to about 44.6. So “use both cards fully” and “one person iterating on code faster” are two different things, and you have to decide which one you want.

Two optimizations the second card loses

This was the least comfortable thing I found in the docs: the engine has two automatic optimizations that only apply on a single card.

OptimizationSingle cardLayer split (two cards)
Prefill CPU offloadOn by defaultOff by default, must be set explicitly
PCIe split ratio auto-tuningOn by defaultNot used with CUDA layer split
Routing tail skipOnAlso applies

In other words, adding the second card does not just add memory, it removes two optimizations. If you only benchmark end-to-end “one card vs two cards”, you fold those two effects into a single number and then draw a wrong conclusion — in either direction.

That is why batch 3 exists: turn the lost optimizations on explicitly first, then compare. Otherwise group B in the A/B is not “more hardware”, it is “more hardware minus two default optimizations”.

The second hardware finding is more ironic: the two cards have NVLink NV2 between them, measured P2P bandwidth 44 GB/s. And layer split is the mode the docs say explicitly does not need NVLink — each card computes its own layers and passes them down serially, a few hundred KB per window.

The result: a 44 GB/s link sits essentially idle under layer split. What would use it is a different mode — split different experts across the two cards within the same token and compute them at once, with activations and results over P2P. The official docs have a “a 2080 Ti as helper is slower” conclusion, but it measured the old host-buffer version, and that machine's two cards could not NVLink at all.

So this is a textbook case of an official conclusion that does not apply to the workstation: dual 2080 Ti + NVLink + a slow CPU is exactly the ideal hardware for that mode, and it has never been measured that way (not in public records, at least). It is scheduled as batch 8.

The other half of 449 MB

One more, dumber thing. Before starting, I looked at free memory on the two NUMA nodes:

node0  free 449 MB      ← GPU0 is on this node
node1  free 12.4 GB     ← GPU1 is on this node

The page cache had eaten node0. If the expert pool allocates per node, GPU0's half lands in remote memory and every access takes an extra hop. This is not a model problem, it is a placement problem.

A more dramatic reference comes from the docs: thread affinity alone (without changing a line of kernel) took a single 5090 from 142.82 to 207.75 tok/s, +45%. On a dual-socket machine with 88 threads there is far more room to place things wrong than on a single-socket desktop.

How cases are scored

There is quality as well as speed. One rule I hold to: if a question can be scored by a program, never score it by eye. Cases fall into seven classes, scored from strongest to weakest:

ClassContentScoring
ADeterministic reasoningExtract the last number, exact match
BCode (reverse a linked list, LRU, binary search bounds, fix an off-by-one)Extract the code → run unit tests
CInstruction following (output only JSON, exactly three items, ban one word)Parse + count + character check
DChinese ability (explanation, translation, long-document summary)Keyword-table coverage + human spot check
ELong-context retrieval (8K/32K/64K/128K × five positions)Does the answer contain the planted unique string
FTool calls (single turn, multi turn, parameter types, not calling when it should not)JSON schema validation; both endpoints tested
GVision (bar chart, table, text, counting)The image is drawn by PIL, so the answer is the number that was drawn

Class G deserves its own note: generate the image with a program, and ground truth needs no human labelling and admits no argument. That is far more reliable than asking “what is in this image” about a real screenshot — with the latter you are never sure whether the model was wrong or the label was.

What is not measured yet

At this point I have to be clear: the batch table in this note is a plan, not a result. What is settled is only a handful of things — the restart cost, the two optimizations that go away with a second card, NVLink sitting idle under layer split, the NUMA imbalance, and the KV VRAM budget at 128K (about 2.21GB at 8-bit, 1.71GB for k8v4, 1.21GB for q4_0, saving 23% / 45%).

The real questions — “is two cards actually faster”, “can peer tier put NVLink to use”, “how many points does KV compression cost” — are all still pending. Once measured, this note becomes a data table; until then it is an experiment design document, which is also a thing worth writing down.

Method / Hardware and cost figures come from research/strata/EXPERIMENTS.md on the workstation and from the engine docs: 2 × RTX 2080 Ti 22GB, NVLink NV2 (measured P2P 44 GB/s), dual Xeon E5-2696 v4 (44C/88T), 94GB DDR4, driver 580 series. The batch table and the case set are an unexecuted plan; nothing in this note presents them as finished results.