Navigating the Shift to cgroup v2 in Kubernetes

The shift to cgroup v2 in Kubernetes is a significant evolution that addresses resource management challenges in containerized environments. Control groups (cgroups) are a kernel feature that helps manage system resources, ensuring that applications run smoothly without stepping on each other's toes. With Kubernetes v1.25, support for cgroup v2 became stable, and from v1.35, the kubelet defaults to not starting on cgroup v1 nodes, pushing users towards the newer model.

Kubernetes utilizes cgroups to allocate resources like CPU and memory to containers. The kubelet coordinates changes between Pod-level and container cgroups, allowing for dynamic adjustments. For instance, when memory reservations are increased, it creates headroom before container limits grow, while decreases constrain containers before shrinking the Pod-level boundary. Key configuration parameters include failCgroupV1, which controls kubelet behavior on cgroup v1 nodes, and memoryReservationPolicy, which determines how memory reservations are handled. The introduction of Memory QoS as an alpha feature in Kubernetes v1.22 adds tiered memory protection, separating memory throttling from memory reservation.

In production, be cautious with alpha features like Memory QoS. The Kubernetes recommendation is to avoid enabling alpha features in production environments unless you thoroughly test them. If you decide to use Memory QoS with tiered reservations, account for hard-reserved memory before enabling the TieredReservation feature gate. Additionally, ensure your environment meets the prerequisites, including a Linux node with cgroup v2 enabled and a compatible kernel version. The kubelet logs warnings when Memory QoS is enabled on affected kernels, so stay vigilant about your configurations.