Seamless Distributed Tracing for CI Pipelines with OpenTelemetry
In today's fast-paced development environment, understanding the performance and behavior of CI pipelines is crucial. Distributed tracing provides insights into how workflows execute, allowing you to identify bottlenecks and optimize processes. With OpenTelemetry, you can achieve this without ever touching a single workflow file, making it a game-changer for teams looking to enhance observability.
The core mechanism involves setting up an OpenTelemetry Collector configured with the githubreceiver component. This collector sits behind a single org-level GitHub webhook, converting incoming workflow_run and workflow_job events into OTLP spans. Each workflow is represented as an outer span, with jobs and steps nested as child spans. Key configuration parameters include the webhook endpoint, path, and secret, along with the endpoint and authorization for the OTLP exporter. For example, you would set the endpoint to 0.0.0.0:19418 and the path to /events, ensuring GitHub can deliver webhooks to your collector.
In production, there are several important considerations. The collector requires a publicly reachable endpoint, so plan for IP allowlisting or a WAF to restrict access to GitHub’s webhook source ranges. Additionally, setting up an org-level webhook necessitates admin access, which should be confirmed early in the process. Be aware that the scrapers block in the configuration is required for validation, even if it seems unrelated to tracing. This can lead to unnecessary troubleshooting time if overlooked. Lastly, if you're using GitHub Enterprise Server, validate webhook delivery behavior before proceeding, as it may differ from github.com.
Key takeaways
- →Configure the OpenTelemetry Collector with the githubreceiver component to enable tracing.
- →Set the webhook endpoint to 0.0.0.0:19418 and path to /events for GitHub integration.
- →Ensure your collector has a publicly reachable endpoint for GitHub webhook delivery.
- →Include a dummy entry in the scrapers block to avoid validation errors.
- →Confirm org admin access early to facilitate webhook setup.
Why it matters
Implementing distributed tracing in CI pipelines enhances visibility into workflow performance, leading to faster identification of issues and improved overall efficiency.
Code examples
1receivers:
2 github:
3 webhook:
4 endpoint: 0.0.0.0:19418
5 path: /events
6 secret: ${env:GITHUB_WEBHOOK_SECRET}
7 scrapers: # required even if you only want tracing, a dummy entry is enough
8 scraper:
9 github_org: ${env:GITHUB_ORG}
10
11exporters:
12 otlp:
13 endpoint: ${env:TRACE_BACKEND_ENDPOINT}
14 headers:
15 authorization: ${env:TRACE_BACKEND_API_KEY}
16
17service:
18 pipelines:
19 traces:
20 receivers: [github]
21 exporters: [otlp]When NOT to use this
The official docs don't call out specific anti-patterns here. Use your judgment based on your scale and requirements.
Want the complete reference?
Read official docsIndustry-standard certifications built by the people behind Linux and Kubernetes. Earn the CKA — the gold standard Kubernetes administrator cert. OpsCanary readers get 30% off year-round with code OPSCANARY3.
Get CKA certified →Unlocking Observability: OpenTelemetry and Kubernetes Events with OpenChoreo
In a microservices world, debugging can feel like finding a needle in a haystack. OpenChoreo leverages OpenTelemetry for distributed tracing, allowing you to track requests across multiple services while capturing persistent Kubernetes events that might otherwise slip through the cracks.
Unlocking Kubernetes: The Workload Conformance Working Group Explained
The Workload Conformance Working Group is setting a new standard for Kubernetes workloads, addressing a critical gap in runtime behavior. This initiative will deliver objective tests for workload behavior, ensuring consistency across Kubernetes versions.
Measuring Developer Experience on EKS: Unlocking DORA Metrics
Want to elevate your developer experience on Amazon EKS? Focus on DORA metrics like Deployment Frequency and Change Failure Rate to gain actionable insights. This article dives into how to connect your CI/CD events for effective measurement.
Get the daily digest
One email. 5 articles. Every morning.
No spam. Unsubscribe anytime.