Scaling OpenTelemetry: Skyscanner's Collector Strategy Across 24 Clusters

In a world where observability is paramount, Skyscanner has developed a robust strategy for managing telemetry across 24 production clusters. The challenge lies in handling vast amounts of data while ensuring that services can communicate effectively. Skyscanner's architecture addresses this by using a central DNS endpoint with intelligent routing via Istio, allowing services to send telemetry to a single address regardless of their global location or cluster. This setup simplifies the process and enhances the reliability of data collection.

The architecture employs two distinct collector patterns: the Gateway Collector, which handles bulk OTLP traffic, and the Agent Collector, which scrapes Prometheus endpoints from services that don't support OTLP natively. The Gateway Collector processes the majority of telemetry, while the Agent Collector ensures that all services are covered. Metrics are transformed into semantic convention names, such as http.client.duration and http.server.duration, and aggregated by cluster, service name, and HTTP status code. Key configuration parameters like dimensions_cache_size (default: 15,000,000) and metrics_flush_interval (default: 30s) are crucial for optimizing performance and resource usage.

In production, understanding how to configure these collectors is vital. The connectors configuration allows for detailed aggregation and dimension management, which can significantly enhance your observability capabilities. Be aware that while this architecture is powerful, it requires careful tuning to avoid performance bottlenecks. Skyscanner's approach exemplifies how to effectively scale observability in a complex environment, but it demands a solid understanding of both the tools and the underlying infrastructure.