Hardcore Linux Administrative Architectures
Linux Fleet Operations Under Load
Most Linux incidents are not mysteries. They are memory pressure, storage faults, kernel regressions, and resource contention that were visible before they became outages. This guide covers the operational layer that keeps a fleet predictable: cgroup limits that fail gracefully, filesystem audits that preserve evidence, immutable rollouts with a real rollback, and the measurement discipline that tells you which of those is actually broken.
Kernpunten
- Start with pressure, not free memory. PSI gives you the stall signal that predicts user-visible latency, while free memory alone says almost nothing about whether the host is healthy.
- Give every service a memory ceiling and a reclaim cushion. memory.high triggers reclaim before memory.max triggers the OOM killer, which turns a hard failure into a slower and observable one.
- Read the OOM report before restarting anything. It names the killed task, its cgroup, and the allocation context, which is usually enough to separate a leak from a working set that outgrew its limit.
- Keep one previous kernel and its matching modules installed on rolling-release hosts. Recovery from a bad kernel is a boot menu selection if the old kernel is present and effectively impossible if it is not.
- Audit storage before repairing filesystems. SMART counters, kernel logs, and power-loss history explain corruption better than fsck output, and repairing first can destroy the evidence you need.
- Separate immutable images from persistent state. Image rollouts are safe only when databases, uploads, and local caches live outside the image and have their own backup and restore path.
- Test eviction and throttling at the limit. A node that behaves well at sixty percent utilization can fail unpredictably at one hundred, and the failure mode is what users experience.
- Measure eBPF overhead with tracing on and off. Kernel telemetry is powerful and not free, especially on high-frequency events and busy network paths.
- Pin toolchains and build inputs for every image. Reproducibility is what makes a rollback trustworthy, because you can rebuild the exact artifact that previously worked.
- Keep an out-of-band console path for every host. Network-based recovery fails exactly when the network configuration is the thing that broke.
Veelgestelde vragen
Why did the OOM killer kill an important process?
Its choice follows memory score and cgroup context, which often differs from what operators expect. Set oom_score_adj deliberately and give critical services reserved memory.
Should I use hard memory limits or proportional shares?
Use shares for bursty workloads that can tolerate slowdowns, and hard limits for services that must not exceed a budget. Many fleets need both on different tiers.
How do I verify a rollback actually works?
Boot the previous kernel or image deliberately, confirm services start and data is intact, then repeat the upgrade behind a staging gate. An untested rollback is a hope, not a control.
Does eBPF tracing slow down production?
It can if the probe runs on a hot path or exports too many events. Aggregate in kernel maps, sample, and compare latency with tracing enabled and disabled.
What fails first on a dense host?
Disk and inode exhaustion from logs and images, followed by noisy-neighbor contention. Both are cheaper to prevent with quotas and limits than to diagnose during an outage.
Hoe Kinetikatech helpt bij hardcore linux administrative architectures
Dit is ons dagelijks werk: Operating Linux fleets under memory pressure, storage faults, kernel regressions, and dense multi-tenant load. Onze dienst Serverbeheer en Linux-serverbeveiliging dekt het volledige traject: beoordeling, implementatie, 24/7-monitoring en noodrespons binnen 30 minuten.