This workstation has 88 threads, two 22GB cards, seven disk partitions. It runs inference, ComfyUI and builds, and it makes a lot of noise and heat. I wanted one dashboard: what each core is doing at a glance, how hot the two cards are right now, and whether power is pinned against the 180W wall or not.
The ready-made answer is to wire nvidia-smi, sensors, free and iostat together. I tried it. What came out had two problems: it was slow, and it lied.
Why not call a subprocess
Slow is measured: one nvidia-smi call takes about a hundred milliseconds, and sensors burns a hundred to three hundred milliseconds of CPU on a dual-socket Xeon. For a dashboard refreshing at 1 second across six views, that means the monitor itself eats a large slice of a core — to find out what the workstation is busy with, first make the workstation busy.
The lying part is subtler: different tools define “used memory” differently, and they sample at different moments. Drawn side by side it looks orderly, but it is a collage of four sampling instants.
So sensorium has one hard rule: fork no subprocesses. CPU and memory come from /proc; temperature, fans, voltage and power come from /sys/class/hwmon; GPUs go through the in-process NVML bindings. One sample is one timeline, not a jigsaw.
Where the data comes from
Six views map to six kinds of data source:
| View | Source | What it shows |
|---|---|---|
| Overview | CPU + memory + GPU summary | load, both package temperatures, VRAM, power, trend curves |
| CPU | /proc/stat, /sys/devices/system/cpu, RAPL | 88-thread activity matrix, per-core clock and temperature, context switches and interrupts |
| GPU | NVML (in-process) | SM/VRAM/encoder-decoder utilization, VRAM and history, temperature thresholds, fans, power and cap, PCIe generation and width, throttle reasons, processes holding VRAM |
| Sensors | full /sys/class/hwmon enumeration | temperature, fan speed, voltage, current, power, with critical limits and alarm bits |
| Storage & Net | /proc/diskstats, /proc/net/dev | throughput, IOPS, mount-point capacity, NIC link speed |
| Processes | /proc/<pid> | top processes, CPU%, RSS, command line (anonymizable) |
One NVML trap deserves its own paragraph: identity fields and time series do not refresh at the same rate. Model, BIOS, serial number, power cap and temperature thresholds never change for the life of the card, yet they were read through the C library every frame just like utilization. The first version read every field every frame, and GPU sampling cost 97 milliseconds.
Tiered sampling
The fix is unglamorous — group by how often a value changes:
| Field | Refresh rate | Reason |
|---|---|---|
| Model / BIOS / serial / cap / thresholds | Read once and cached | Hardware properties, they do not change |
| Utilization / VRAM / temperature / fan / power | Every frame | The curves need them |
| Frequency / Power Limit / PCIe / encoder-decoder | Every 2 frames | Slow to change |
| Clock / P-State / throttle reasons | Every 4 frames | State-machine level changes |
| Energy counters | Every 30 frames | A single read is expensive |
Result: GPU sampling drops from about 97 milliseconds to about 15, and nothing on screen changes.
The same idea went into the sensors and the process table. Sensor labels and critical limits are read once and rescanned every ten seconds so hotplug works; each frame reads only *_input. Per-core temperature is a cross-core MSR read every time, which is expensive on 44 cores, so it refreshes every two frames. Walking a thousand /proc/<pid>/stat entries takes twenty to thirty milliseconds, so the process table samples every three seconds — which also makes per-process CPU% more accurate.
The cost of a monitoring tool can almost always be cut by an order of magnitude once you ask which values actually change every frame. Group first, then optimize.
Drawing it myself
Rendering takes the same route: no relayout of the whole tree, write straight into ratatui's Buffer. A full 120×40 repaint is 0.2–0.35 milliseconds — three orders of magnitude cheaper than one nvidia-smi read. That is also why this tool can sit at a 1-second refresh and go almost unnoticed.
The cost
Measured on the workstation, release build, with the workstation itself loaded during sampling (load ≈ 2.5, jobs on the GPU):
| View | Share of one core | Per-frame render |
|---|---|---|
| Overview | 3.5 – 5.0% | 0.34 ms |
| Sensors | 3.9 – 5.1% | 0.31 ms |
| Storage & Net | 4.0 – 5.0% | 0.25 ms |
| GPU | 4.1 – 5.3% | 0.26 ms |
| CPU | 4.5 – 6.4% | 0.31 ms |
| Processes | 4.7 – 6.1% | 0.21 ms |
Against the whole workstation that is 0.05%–0.07%, with about 24 MiB resident and a 1.2 MB binary (strip + LTO). The workstation has 88 threads, so “4% of one core” sounds scarier than “0.06% of the workstation” — both framings are written down because they answer different questions.
Checking accuracy
The most dangerous failure for a monitoring tool is not a crash, it is quietly showing the wrong number. So it ships --json, which dumps one sample verbatim to be diffed field by field against an independent calculation:
- every memory field matches byte for byte;
- total CPU utilization differs from an independent count by 0.14%;
- network throughput differs by 2% in one window-misaligned comparison (891 MiB/s vs the independent count), while a self-built constant 8 MiB/s loopback stream reads exactly 8.00 MiB/s;
- only
--benchreports “steady state” and “first sample” (NVML init, first full sensor scan, first/procwalk) separately — otherwise the one-off cost is averaged in and looks flattering.
There is a more important principle: if it cannot read something, it says so. Most distributions make /sys/class/powercap/*/energy_uj root-only, so a normal user cannot get CPU package power; it prints unavailable (energy_uj needs root) instead of drawing a zero line. This board does not load a Super-I/O fan driver and has no /dev/ipmi0, so chassis fan speed is unreadable — so that field stays empty.
The credibility of a monitoring tool rests on how quiet it is when data is missing.
What it cannot read
The known gaps are in the README rather than in my head: CPU package power needs root; chassis fans need a driver such as nct6775; GPUs are NVIDIA only (AMD's amdgpu shows up as a sensor but has no utilization or VRAM); per-core temperature updates at half the display refresh rate, because every read across 44 cores is a cross-core MSR read.
Then there is privacy: this tool will be asked to paste its output into an issue. --demo turns anonymization on automatically — hostname becomes workstation, process command lines keep only the program name, NIC names lose the embedded MAC, GPU serial numbers are cleared, home paths become /home/user. Hardware specs stay; those explain the problem, they are not personal information.
Method / All figures come from the NorwinYu/sensorium README and --bench output; the test machine is dual Xeon E5-2696 v4 (44C/88T), 2 × RTX 2080 Ti 22GB, 94GB DDR4, loaded during sampling. The view screenshots are rendered offline from sensorium --frame --demo ANSI output and can be reproduced with tools/render-frame.py in the repository.