observabilitytracingPractitioner

Unlocking Tempo 3.1: Kafka Enhancements and TraceQL Metrics

5 min read Grafana BlogReviewed for accuracy

Practitioner — Hands-on experience recommended

Tempo 3.1 exists to enhance observability by improving how you ingest and manage trace data. With the rise of distributed systems, the need for secure, efficient data handling has never been more critical. This release tackles those challenges head-on with community-contributed Kafka client improvements, enabling secure connections and reducing data transfer costs. You can now use TLS and multiple SASL mechanisms for authentication, ensuring that your data remains safe while in transit. Additionally, rack-aware fetching allows consumers to read from nearby replicas, optimizing performance and cost.

The new TraceQL metrics feature is a significant addition, allowing you to query ad-hoc metrics directly from trace data. This means you can derive insights without waiting for batch processing. The introduction of sampling-aware metrics queries improves accuracy by accounting for sampling rates at query time. Furthermore, trace redaction can now be performed using a TraceQL query, enabling you to efficiently remove sensitive data without waiting for retention periods to expire. This is particularly useful in compliance-heavy environments where data privacy is paramount.

In production, you need to be aware of a few gotchas. Ensure that all schedulers and workers are running Tempo 3.1 before using the --start and --end flags; otherwise, older workers may ignore your time window, leading to data loss. Also, be cautious with the experimental hint requiring vParquet4 blocks or later. These details can make or break your observability strategy, so keep them front of mind as you integrate these new features into your stack.

Key takeaways

→Implement TLS and SASL mechanisms for secure Kafka connections.
→Utilize rack-aware fetching to optimize data transfer costs.
→Leverage TraceQL for real-time metrics queries from trace data.
→Perform trace redaction using TraceQL to manage sensitive information efficiently.
→Ensure all components are updated to Tempo 3.1 to avoid data loss during queries.

Why it matters

This release significantly enhances the security and efficiency of data ingestion in observability stacks, allowing teams to manage sensitive information and derive insights faster. These improvements can lead to better compliance and performance in production environments.

Code examples

YAML

1ingest:
2  kafka:
3    address: kafka.example.com:9093
4    topic: tempo-traces
5    sasl_mechanism: SCRAM-SHA-512
6    sasl_username: ${KAFKA_USERNAME}
7    sasl_password: ${KAFKA_PASSWORD}
8    tls_enabled: true
9    tls_ca_path: /etc/tempo/kafka-ca.pem
10    tls_cert_path: /etc/tempo/kafka-client.crt   # optional, for mTLS
11    tls_key_path: /etc/tempo/kafka-client.key    # optional, for mTLS

YAML

ingest:
  kafka:
    client_rack: us-east-1a

Bash

tempo-cli redact \
  --tenant=<TENANT_ID> \
  --query '{span.attribute = "<leaked PII>"}' \
  --dry-run \
  <SCHEDULER_ADDRESS>:<GRPC_PORT>

When NOT to use this

The official docs don't call out specific anti-patterns here. Use your judgment based on your scale and requirements.

Want the complete reference?

Read official docs

Test what you just learned

Quiz questions written from this article

Take the quiz →

DigitalOcean Serverless InferenceSponsor

OpenAI & Anthropic-compatible inference API — no GPU provisioning needed. 55+ models, pay-per-token with no minimums. VPC + zero data retention by default.

Try Serverless Inference →

Unlocking Tempo 3.1: Kafka Enhancements and TraceQL Metrics

Key takeaways

Why it matters

Code examples

When NOT to use this

More on this topic

Maximize Your Trace Budget with Volumetric Sampling in Grafana Cloud

Tempo 3.0: Unlocking Scalable Observability with TraceQL Metrics

Mastering Jaeger Tracing for Effective Deployment Observability