Strata / local inference
Multi-GPU on old cards.
Upstream: Niko1221/Strata . Strata runs a large model on consumer hardware. My contribution is a series of benchmark reports and docs notes for two RTX 2080 Ti 22 GB cards on a dual-socket Xeon E5-2696 v4.
- Focus
- Layer split vs one card, --batch, vLLM
- Hardware
- 2× 2080 Ti 22 GB · dual Xeon
- Form
- 5 upstream PRs
- Data
- Raw requests, configs, concurrency
Layer split vs one card, UD-IQ4_XS: decode 40.8 → 60.4 tok/s (+48%); 32K fresh prompt 768 → 1,269 tok/s (+65%).