Unlocking Observability: Embedding Metrics in AWS Logs

In today's cloud-native environments, observability is crucial for maintaining system health and performance. Traditional logging often leaves you with a wealth of data but little actionable insight. By embedding metrics directly within your logs, you can bridge this gap, allowing for more granular monitoring and faster incident response.

The CloudWatch embedded metric format enables you to generate custom metrics asynchronously in the form of logs written to CloudWatch Logs. This means you can embed custom metrics alongside detailed log event data. CloudWatch automatically extracts these metrics, allowing you to visualize and set alarms on them. To make this work, ensure you have the necessary permission: logs:PutLogEvents. Interestingly, you don’t need the cloudwatch:PutMetricData permission, simplifying your permission management.

However, be cautious with your metric extraction configuration. It directly impacts your custom metric usage and billing. If you inadvertently create metrics based on high-cardinality dimensions, such as requestId, you could end up with a multitude of custom metrics—one for each unique dimension combination. This can lead to unexpected costs and complexity in your monitoring setup.