Injecting Chaos: My LFX Mentorship with kgateway

In today's cloud-native environments, ensuring service resilience is non-negotiable. That's where kgateway comes in, providing a robust foundation for managing traffic in Kubernetes. My journey through the LFX Mentorship Program centered on enhancing kgateway by integrating chaos engineering support, specifically through HTTP fault injection. This capability allows platform teams to deliberately introduce failures like delays and aborts, validating their services' resilience before these issues arise in production.

The implementation involved extending the TrafficPolicy to support fault injection. This included delay injection, abort injection using both HTTP and gRPC status codes, and response rate limiting. Additionally, I introduced a per-route override to disable these features when necessary. Under the hood, this functionality maps to the Envoy HTTP fault filter, which is added to the filter chain but remains disabled by default. This selective enabling per route or virtual host allows for granular control over how and when to test service resilience.

In production, it's crucial to understand that chaos engineering is not just about injecting faults; it's about learning how your system behaves under stress. With five years of experience in cloud-native API gateways and networking, I was well-prepared to navigate the complexities of Go, Kubernetes, and Envoy Proxy. However, be mindful of the potential pitfalls of chaos engineering—it's easy to overwhelm your system or misinterpret the results if not done thoughtfully.