The workstation under my desk has two RTX 2080 Ti cards, 44GB of VRAM in total, and an electricity bill that makes me want to close the door. It's built from spare parts and it's louder than a normal conversation. But it changed how I understand local AI more than any benchmark chart.
The first lesson arrived before the first model loaded: 22 plus 22 does not equal 44, at least not any 44 that means anything.
The arithmetic illusion
Two cards give you 44GB of total VRAM, but they are not one 44GB card. The distinction isn't pedantry. It decides which models run comfortably, which only run slowly, and which leave you staring at CUDA out of memory at one in the morning.
A model that fits on a single card avoids card-to-card communication entirely. A model split across two cards has to coordinate at every layer boundary that crosses the split. The workstation has NVLink — NV2, two bonded links at 25.78 GB/s each — so the coordination doesn't go over PCIe. Fast wires aren't free wires, though: the allreduce per block is still 13 ms, the two cards sit on different NUMA nodes, and the host thread that launches kernels is sometimes more expensive than the link. The second card isn't free memory. It's memory you have to schedule to use.
VRAM is a budget, not a number. Context, KV cache, activations and the framework itself each take a line.
So “will it fit in 44GB” is rarely the right question. The right question is: at the context length I actually need, what's the largest model that runs, is the speed still usable, and does the headroom survive the next driver update?
What actually fits
Here's the rough map I use. The numbers are 4-bit inference weight sizes; they exclude KV cache, framework, vision tower and fragmentation. Treat them as a starting point, not a guarantee.
| Model size | 4-bit weights | Where it fits | Reality |
|---|---|---|---|
| 7B–8B | about 4–5GB | Single card | Comfortable. Leaves room for long context and batch. |
| 13B–14B | about 7–9GB | Single card | Comfortable. 22GB is not the issue. |
| 27B–32B | about 16–20GB | Single card, tight | Runs at moderate context; KV cache decides the outcome. |
| 70B | about 35–40GB | Two cards | Runs on paper at short context, very slowly. Not a daily driver. |
7B is the sweet spot. A quantized 7B/8B leaves several GB for context and batch, the speed is usable, and the cost of a failed run is minutes rather than an afternoon. 14B sits comfortably on a 22GB card. 32B is tight on one card and has headroom on two. 70B is a demo, not a daily driver.
This is also why “a local model is a cheap GPT” fails. A 7B isn't a worse frontier model; it's a different instrument: private, predictable, always on, and good within a narrow band of tasks. Use it on that band.
The Turing limits
The 2080 Ti is Turing, not Ampere. That has practical consequences: no BF16, so modern training recipes need translation; many inference kernels are tuned for Ampere or newer, and while they run on Turing they aren't always the fast path. Flash attention, quantized kernels and fused operators all have to be verified against the actual card, not against the README.
Memory bandwidth is still decent — about 616 GB/s per card — and FP16 tensor cores are there. Inference, especially quantized inference, is far from unusable. Fine-tuning is a different story: no BF16 plus the 22GB per-card ceiling turns every experiment into careful accounting. Small LoRAs are fine. Full fine-tuning is not what the workstation is for.
Then there's the unglamorous physics: two 250W cards (I cap mine at 180W), dual Xeons and spinning disks produce heat and noise, and make the room noticeably warmer than the rest of the house. It's a space heater that answers questions occasionally.
The bottleneck moves
At short context the bottleneck is usually decode: one token at a time, limited by memory bandwidth, and the 2080 Ti's bandwidth is the ceiling. At long context, prefill and KV cache take over. A model that's fast at a 4k context can feel like a different machine at 32k, even though the weights haven't changed.
So a local inference benchmark that reports a single tokens/s figure is close to useless for product decisions. What you need is tokens/s at your context length, your batch, your real prompt distribution. Everything else is a feeling.
Time to first token, decode speed at 4k / 16k / 32k, peak VRAM, and the point where the workstation starts swapping. A local model that's only fast on an empty prompt is a demo, not a tool.
There's a second, quieter bottleneck: the person. A local model that takes 90 seconds to answer changes the rhythm of work. You stop asking small questions, start batching them, start writing better prompts, because the feedback loop is expensive. That constraint can be a feature — but only if you design around it instead of pretending it isn't there.
Routing is design
The most useful thing the workstation taught me isn't how to run large models locally. It's how to route between local and cloud without romance on either side.
| Work | Where | Why |
|---|---|---|
| Private notes, drafts, embeddings | Local | Data never leaves the desk; cheap at volume too. |
| Classification, extraction, filtering | Local | Small models are enough, and iteration is fast. |
| Hard reasoning, fuzzy design | Cloud | Frontier capability is still frontier. |
| Long-context research | Cloud | KV cache and context windows are not a fair fight. |
| Image and motion generation | Cloud / rent | Consumer VRAM has the wrong shape. |
The rule is boring: private, high-volume, well-defined work goes local first; fuzzy reasoning, long context and visual generation go to the cloud. The interesting part is the boundary, and every project moves it a little.
It's an instrument, not a data center
I've stopped trying to turn the workstation into a small cloud. It isn't one. It's an instrument, for questions that need the data to stay home, that need a hundred cheap iterations, or that simply want to hear the fans spin up.
That framing makes the constraints useful. Limited VRAM forces quantization and batch discipline; slow decode forces tighter prompts and tighter scope; 22GB per card forces you to choose models that respect the split. The workstation isn't a limit on ambition. It's a set of design constraints you can actually hear.
The best local setup isn't the one that runs the largest model. It's the one that makes the next experiment cheap enough to try.
A checklist
- Pin the stack. Driver, CUDA, PyTorch and inference server versions go in the notes. “It worked last week” is not a reproduction strategy.
- Measure at real context. Record time to first token, decode speed and peak VRAM at 4k, 16k and 32k.
- Leave headroom. If the model occupies 43.5GB of 44GB, it doesn't fit. The next prompt will find the edge.
- Keep the model card. Quantization scheme, chat template, context ceiling, the tasks it's actually good at.
- Separate experiments from serving. A Jupyter notebook and a systemd unit have different failure modes.
- Don't leave keys on the workstation. Local isn't safe. Env files, keys and tokens need the same discipline as a server.
The 44GB problem has no clean solution. It has a set of working trade-offs, a few models genuinely worth keeping, and a daily reminder that capability is always shaped by the workstation that runs it. That isn't a limitation. It's the design brief.
Hardware / 2 × RTX 2080 Ti (22GB each, NVLink NV2 dual link, 2 × 25.78 GB/s), 94GB DDR4, dual Xeon E5-2696 v4 (44C / 88T), Ubuntu 24.04.5 LTS. The numbers are 4-bit inference weight-size estimates and exclude KV cache, activations and framework overhead. No tokens/s claim is made here; measure your own workload.