MiniMax H3 is a 20.11B joint streaming video and audio Transformer: 50 blocks, hidden 5376, FFN 14336. After W4A8 quantisation the DiT weights take 10.96 GiB on disk, which fits inside one 22GB card. It does produce clips.
The problem: the other card sat idle next to it. It was a 2U fan moving air.
It fits, but it is not used
On one card, a smoke sample of 640×352 × 124 frames × 4 steps runs at about 51 seconds per step, 243 seconds for a 4-step clip. GPU1 at 97% utilisation, peak memory 19.1GB, the temperature sitting at 78 °C for four minutes. It was not that it could not run. It was that it could not run like that all afternoon.
So the goal was plain: get the second card into the loop. Not as a bigger memory pool — the model already fit — but to share the compute and, incidentally, cut the time one card spends hot.
The split was not the mistake
The TP2 split was the standard one: qkv, out and fc1 split by column, fc2 by row, with a symmetric allreduce at the end of each block. Both cards hold the full input and output shapes across their 25 blocks, and half of every matrix.
It ran at 1.31×. The split logic was correct and the numbers were right, but the GPUs sat idle half the time. It did not look like communication cost — allreduce was only 13 ms/block, 3% of 420 ms — it looked like the two cards taking turns.
The block was not in the interconnect
The repository's probes/ exists for this question: run the same GEMM on both cards concurrently, then switch to one thread per card, and record the time overlap in both cases. 95% overlap means both cards are computing at the same time; 5% overlap means they simply queued.
| Test | dev0 | dev1 | Concurrent time | Overlap |
|---|---|---|---|---|
| dense fp16 GEMM | 61.9 | 58.0 | 61.0 | 95% |
| W4A8 GEMM | 103.3 | 96.8 | 194.7 | 5% |
| W4A8 GEMM, one thread per card | 106.5 | 97.1 | 106.8 | 91% |
Look at the second row: each card's GEMM alone is about 100 units, and running them concurrently takes 194.7 — exactly the sum. They did not run in parallel; they ran twice in series.
The cause was not NVLink, and not CUDA's stream scheduling. It was the W4A8 GEMM in comfy-kitchen: after launching the kernel it blocks the host thread that called it until the kernel finishes. The whole forward pass has one host thread, so the first GEMM blocks that thread and the second rank's kernel can only queue. The dense fp16 path does not block, which is why it reached 95% overlap — one split, two outcomes.
One thread per card
The fix is a few lines: give each rank its own host thread (_par in tp2.py), so two threads launch the two cards' kernels. Overlap went from 5% to 91%, concurrent time from 194.7 to 106.8 — close to the single-card time, which is actual parallelism.
It is a pure scheduling change and touches no numbers: the output is bit-identical, frame MAE 0.0000. That matters — it shows the 2.97× was not bought by computing less.
Results
| Metric | Single card | TP2 |
|---|---|---|
| Time per step | ~51 s | ~14 s |
| 4-step clip, end to end | 243 s | 82 s |
| Speedup | 1× | 2.97× |
| GPU utilisation | GPU0 idle / GPU1 97% | 57% / 72% |
| Peak memory | 19.1 GB | 11.6 + 19.7 GB |
| Time running hot | 78 °C · 4 minutes | 75 °C · 1.4 minutes |
At output resolution (864×480, ref2va) the curve is the same story on another scale: 44.6 s/step on one card, 34.0 s (1.31×) for the first TP2, 19.8 s (2.25×) after one thread per card. Inside a fixed block, qkv plus attention is 211 ms, MLP 196 ms, allreduce 13 ms — 420 ms/block in total.
Do not judge whether parallelism worked by the end-to-end speedup. Measure overlap first: are the two cards' kernels really running at the same time, or just alternating. An end-to-end 1.31× can come from many causes. 5% overlap has one.
Does this carry over?
Yes, with limits. What holds here is the premise that the degree of parallelism is set by how the host side issues calls, not only by how the weights are split. Any kernel or library that blocks its caller turns parallelism into queueing once it is dropped into a single-threaded forward pass. The dense fp16 versus W4A8 comparison is the same model, the same split, with only the kernel implementation changed.
The parts that do not carry over need saying too: these numbers come from one machine, two 22GB 2080 Ti cards, a 250W power limit, driver 580 series, CUDA 13.0, fixed seed 42. They do not prove that other cards or other models will also reach 2.97×. The two cards also sit on different NUMA nodes, so half of the 13 ms allreduce is topology, and a dual-socket board would change it.
What is worth keeping is the diagnostic order: first confirm the split is correct (the numbers agree), then confirm the kernels really are concurrent (overlap), and only then optimise communication. Done the other way round, you spend a lot of time tuning a bandwidth problem that does not exist.
Method / The data comes from the NorwinYu/minimax-h3-dual-2080ti repository's docs/PERFORMANCE.md and probes/; the numbers were measured on real renders on the workstation, not estimated. Code is MIT; model weights are not distributed with the repository and follow MiniMax's own licence. How overlap is measured is documented in docs/ARCHITECTURE.md.