Storage & I/O Performance
iowait is a symptom. Knowing what it's a symptom of is the job.
Overview
iowait numbers without context are noise. An I/O bottleneck you cannot attribute to a specific process and device is an I/O bottleneck you cannot fix. Answer the five questions below with the tool-level precision of someone who has diagnosed a live I/O performance incident.
Read before you answer
Storage I/O is the most common performance bottleneck in production Linux systems and the most frequently misdiagnosed. High iowait (the CPU time spent waiting for I/O to complete, visible in `top` as the `wa` field) tells you that processes are blocked on I/O — but not which processes, which devices, or what kind of I/O. Diagnosing an I/O problem requires layering tools: `iostat -x 1` for per-device utilisation, await, and queue depth; `iotop` for per-process I/O attribution; `blktrace`/`blkparse` for block-level I/O traces; and `vmstat 1` for the relationship between I/O, swap, and CPU states. A system with 80% iowait is not necessarily broken — a bulk data import legitimately dominates I/O; but 80% iowait on a system that should be serving interactive web requests is a production emergency.
SSD write endurance is a reliability concern that most administrators underestimate until a drive fails unexpectedly. SSDs have a finite number of program-erase cycles per NAND cell, measured in DWPD (Drive Writes Per Day) or TBW (Terabytes Written). Consumer SSDs (including many used in cloud VMs and budget VPS providers) have significantly lower endurance than enterprise SSDs. The practical implications are: log-heavy applications writing multiple gigabytes per hour can exhaust consumer SSD endurance in months, not years; NVMe drives in high-write database workloads require enterprise-grade hardware rated for the actual write load; and monitoring write amplification and drive health via SMART data (`smartctl -a /dev/sda`) gives advance warning before a drive hits its TBW limit. `nvme smart-log /dev/nvme0` reports percentage_used which directly indicates remaining endurance — a drive at 95% used is approaching end of life.
Log management is one of the most controllable sources of unnecessary I/O in production systems. Uncontrolled log growth has produced more unplanned outages than most categories of hardware failure. The correct architecture for logs is: application logs written to journald or a structured log aggregator (not raw files that accumulate indefinitely); logrotate or journald size/time limits configured for all file-based logs; tmpfs for genuinely ephemeral data (session files, PID files, temporary uploads, cache files that can be regenerated) to eliminate disk I/O entirely for that category; and monitoring of filesystem utilisation with alerting well before 100% — a full filesystem on the root partition causes systemd, journald, and many applications to fail in difficult-to-diagnose ways. The combination of a disk-full event and aggressive log writes is one of the most reliable ways to turn a manageable incident into an unrecoverable one.