Process Management & Resource Limits
Ulimits, cgroups, and the difference between a watchdog and a prayer.
Overview
Resource limits without measurement are guesses. Resource limits without restart policies are hopes. And a watchdog that restarts a crashing process indefinitely is not a reliability feature — it is deferred failure with extra steps. Answer the five questions below with the operational depth of someone who has debugged a production resource exhaustion incident.
Read before you answer
Linux provides a layered system for controlling what resources processes can consume: ulimits at the process level, cgroups at the group level, and systemd resource controls as a management interface over cgroups for service-managed processes. Understanding all three layers — and which one to use for which problem — is essential for running stable multi-service systems where one misbehaving process cannot exhaust resources for everything else. The most common production failure pattern is not a single process crashing, but a runaway process consuming all available memory, CPU, or file descriptors and degrading every other service on the host until the system becomes unresponsive.
Ulimits (`ulimit -a` to view, `/etc/security/limits.conf` to configure persistently) control per-process limits including: `nofile` (maximum open file descriptors — the most commonly hit limit for network-heavy applications, often requiring an increase from the default 1024 to 65536 or higher); `nproc` (maximum threads/processes per user); `stack` (stack size); and `core` (core dump size). Hard limits are enforced by the kernel and cannot be exceeded; soft limits are the current effective limit and can be raised by the process itself up to the hard limit. Modern systemd services set limits via `LimitNOFILE=`, `LimitNPROC=`, etc. in the service unit file, which is the preferred method for service-specific limits and does not require modifying system-wide configuration files.
Cgroups (control groups) provide hierarchical resource accounting and limiting at the group level. Cgroups v2, the unified hierarchy now default in modern distributions, provides: `cpu.max` for CPU bandwidth limiting (useful for preventing a CPU-intensive process from starving others); `memory.max` for hard memory limits with OOM kill within the cgroup rather than system-wide; `io.max` for I/O bandwidth limits; and `pids.max` for process count limits. Systemd exposes cgroups through service unit directives like `CPUQuota=`, `MemoryMax=`, `IOWeight=`, and `TasksMax=`. A well-configured production service unit should specify both resource limits (to bound the blast radius of a runaway process) and a restart policy (`Restart=on-failure`, `RestartSec=5`, `StartLimitIntervalSec=`, `StartLimitBurst=`). The common mistake is a watchdog restart policy with no limits: if the service is crashing due to a memory condition, restarting it every five seconds does nothing but consume more resources on the way to a harder crash.