Provisioning Grafana: Mastering Synthetic Monitoring
Provisioning Grafana is essential for automating the setup and management of your observability tools. It allows you to define data sources and configurations in a structured way, reducing manual overhead and ensuring consistency across environments. This is particularly important in production, where a misconfigured data source can lead to gaps in monitoring and alerting.
Grafana utilizes a provisioning system that reads configuration files from specific directories. You can manage data sources by placing YAML files in the provisioning/datasources directory. Grafana reads its default configuration from <WORKING DIRECTORY>/conf/defaults.ini and custom configuration from <WORKING DIRECTORY>/conf/custom.ini. You can override the custom configuration path using the --config option. Notably, if you set prune: true in your provisioning file, Grafana will automatically delete data sources that are no longer listed, keeping your environment tidy. This is a powerful feature that helps prevent clutter from outdated configurations.
In production, remember to use environment variables judiciously. They can be used for configuration values but should not be relied upon for keys or larger parts of your configuration structure. Also, be aware that the default installation paths for configuration files may differ based on your package manager. For example, the Deb and RPM packages place the configuration file at /etc/grafana/grafana.ini. This can affect how you set up your provisioning system, especially in environments with strict configuration management practices.
Key takeaways
- →Utilize YAML configuration files in the provisioning/datasources directory for managing data sources.
- →Enable the prune feature to automatically delete unlisted data sources and maintain a clean environment.
- →Use environment variables for configuration values, but avoid them for sensitive keys or complex structures.
- →Override the default configuration path with the --config option to customize your setup.
- →Be mindful of the installation paths for configuration files based on your package manager.
Why it matters
Effective provisioning in Grafana ensures that your observability stack is always in sync with your infrastructure. This minimizes downtime and enhances your ability to respond to incidents quickly.
Code examples
1datasources:
2 - name: Graphite
3 url: http://localhost:$PORT
4 user: $USER
5 secureJsonData:
6 password: $PASSWORD1# Configuration file version
2apiVersion: 1
3
4# List of data sources to delete from the database.
5deleteDatasources:
6 - name: Graphite
7 orgId: 1
8
9# Mark provisioned data sources for deletion if they are no longer in a provisioning file.
10# It takes no effect if data sources are already listed in the deleteDatasources section.
11prune: true
12
13# List of data sources to insert/update depending on what's
14# available in the database.
15datasources:
16 - name: Graphite
17 type: graphite
18 access: proxy
19 orgId: 1
20 uid: my_unique_uid
21 url: http://localhost:8080
22 user:
23 database:
24 basicAuth:
25 basicAuthUser:
26 withCredentials:
27 isDefault:
28 jsonData:
29 graphiteVersion: '1.1'
30 tlsAuth: true
31 tlsAuthWithCACert: true
32 secureJsonData:
33 tlsCACert: '...'
34 tlsClientCert: '...'
35 tlsClientKey: '...'
36 password:
37 basicAuthPassword:
38 version: 1
39 editable: false1datasources:
2 - name: Graphite
3 secureJsonData:
4 password1: $PASSWORD # Resolved as Pa$sw0rd
5 password2: ${PASSWORD} # Resolved as Pa
6 password3: 'Pa$$sw0rd' # Resolved as Pa$sw0rd
7 password4: 'Pa$sw0rd' # Resolved as PaWhen NOT to use this
The official docs don't call out specific anti-patterns here. Use your judgment based on your scale and requirements.
Want the complete reference?
Read official docsOpenAI & Anthropic-compatible inference API — no GPU provisioning needed. 55+ models, pay-per-token with no minimums. VPC + zero data retention by default.
Try Serverless Inference →Managing Synthetic Monitoring Checks as Code with Terraform and Grafana Cloud
Take control of your synthetic monitoring with Terraform and Grafana Cloud. Learn how to prototype checks in the Grafana UI and then export them as Terraform resources for seamless management. This approach ensures your monitoring checks are versioned and reproducible.
Mastering Grafana Alerting: A Deep Dive into Synthetic Monitoring
Grafana Alerting is crucial for maintaining system reliability and performance. By leveraging alert rules and notification policies, you can effectively monitor your services. This article breaks down how to set up alert rules that fire based on specific conditions, ensuring you never miss critical issues.
Mastering the Multi-Target Exporter Pattern for Observability
The multi-target exporter pattern is a game changer for monitoring diverse systems. It allows you to scrape metrics from multiple targets using a single exporter, simplifying your observability stack. Dive into how it works and what you need to know to implement it effectively.
Get the daily digest
One email. 5 articles. Every morning.
No spam. Unsubscribe anytime.