Hardcore Linux Administrative Architectures
Linux Fleet Operations Under Load
Most Linux incidents are not mysteries. They are memory pressure, storage faults, kernel regressions, and resource contention that were visible before they became outages. This guide covers the operational layer that keeps a fleet predictable: cgroup limits that fail gracefully, filesystem audits that preserve evidence, immutable rollouts with a real rollback, and the measurement discipline that tells you which of those is actually broken.
핵심 내용
- Start with pressure, not free memory. PSI gives you the stall signal that predicts user-visible latency, while free memory alone says almost nothing about whether the host is healthy.
- Give every service a memory ceiling and a reclaim cushion. memory.high triggers reclaim before memory.max triggers the OOM killer, which turns a hard failure into a slower and observable one.
- Read the OOM report before restarting anything. It names the killed task, its cgroup, and the allocation context, which is usually enough to separate a leak from a working set that outgrew its limit.
- Keep one previous kernel and its matching modules installed on rolling-release hosts. Recovery from a bad kernel is a boot menu selection if the old kernel is present and effectively impossible if it is not.
- Audit storage before repairing filesystems. SMART counters, kernel logs, and power-loss history explain corruption better than fsck output, and repairing first can destroy the evidence you need.
- Separate immutable images from persistent state. Image rollouts are safe only when databases, uploads, and local caches live outside the image and have their own backup and restore path.
- Test eviction and throttling at the limit. A node that behaves well at sixty percent utilization can fail unpredictably at one hundred, and the failure mode is what users experience.
- Measure eBPF overhead with tracing on and off. Kernel telemetry is powerful and not free, especially on high-frequency events and busy network paths.
- Pin toolchains and build inputs for every image. Reproducibility is what makes a rollback trustworthy, because you can rebuild the exact artifact that previously worked.
- Keep an out-of-band console path for every host. Network-based recovery fails exactly when the network configuration is the thing that broke.
자주 묻는 질문
Why did the OOM killer kill an important process?
Its choice follows memory score and cgroup context, which often differs from what operators expect. Set oom_score_adj deliberately and give critical services reserved memory.
Should I use hard memory limits or proportional shares?
Use shares for bursty workloads that can tolerate slowdowns, and hard limits for services that must not exceed a budget. Many fleets need both on different tiers.
How do I verify a rollback actually works?
Boot the previous kernel or image deliberately, confirm services start and data is intact, then repeat the upgrade behind a staging gate. An untested rollback is a hope, not a control.
Does eBPF tracing slow down production?
It can if the probe runs on a hot path or exports too many events. Aggregate in kernel maps, sample, and compare latency with tracing enabled and disabled.
What fails first on a dense host?
Disk and inode exhaustion from logs and images, followed by noisy-neighbor contention. Both are cheaper to prevent with quotas and limits than to diagnose during an outage.
hardcore linux administrative architectures에 대한 Kinetikatech의 지원
저희는 이것을 수백 번 처리했습니다: Operating Linux fleets under memory pressure, storage faults, kernel regressions, and dense multi-tenant load. 서버 관리 및 Linux 서버 보안와 함께하면 평가, 구현, 24시간 연중무휴 모니터링, 그리고 중요한 때에 30분 이내 대응이라는 완전한 사이클을 제공합니다.