Zero-Downtime Migration: From Ingress NGINX to Envoy Gateway

In the world of Kubernetes, ensuring zero downtime during migrations is a critical concern. When transitioning from Ingress NGINX to Envoy Gateway, the challenge lies in maintaining service availability while shifting traffic. This migration approach allows you to run both the Ingress and the HTTPRoute at the same time, using weighted DNS records to manage traffic without a hard cutover. This means you can confidently test the new configuration while still serving requests through the old one.

The process involves using ExternalDNS and AWS Route 53 to set up weighted DNS records. Initially, you assign a high weight to the Ingress and zero to the HTTPRoute. This configuration makes the HTTPRoute live but not yet receiving traffic. Once you’re confident in the new setup, you simply swap the weights. This method avoids the need to create or delete DNS records, making the transition smooth and controlled. For instance, you might set the Ingress weight like this: external-dns.alpha.kubernetes.io/aws-weight: "100" and the HTTPRoute like this: external-dns.alpha.kubernetes.io/aws-weight: "0". After the swap, you can revert easily if any issues arise, as there’s no resource to redeploy or DNS record to recreate.

In production, this method is straightforward but requires careful monitoring during the transition. The rollback process is particularly advantageous, allowing you to revert changes quickly if anything goes wrong. This migration strategy is effective and minimizes risk, but always ensure you have proper observability in place to catch any potential issues early.