Taming Secret Sprawl in Multi-Account Kubernetes with External Secrets Operator
Managing secrets in a multi-account Kubernetes setup can lead to chaos if not handled properly. Secret sprawl occurs when secrets are scattered across various accounts and clusters, making it difficult to maintain security and compliance. The External Secrets Operator (ESO) addresses this challenge by providing a Kubernetes-native reconciliation model that synchronizes secrets from external secret management systems, like Bitwarden, directly into the Kubernetes Secret API.
At its core, ESO operates through three main components: a centralized secret management system (Bitwarden), the External Secrets Operator running in each Kubernetes cluster, and the Kubernetes Secrets that ESO generates and maintains. When you create an ExternalSecret resource, ESO retrieves the specified secret from Bitwarden and creates or updates the corresponding Kubernetes Secret. This process is governed by a reconciliation loop that continuously checks for updates based on a configured refresh interval, which defaults to 15 minutes. You can customize this behavior using parameters like refreshInterval to ensure your secrets are always current.
In production, you need to be aware of a few key points. First, ensure you have a Kubernetes cluster (EKS, AKS, GKE, etc.) and that you’ve installed both kubectl and helm. The installation of ESO involves adding the external-secrets Helm repository and deploying it with the necessary configurations. Be cautious with your access tokens; granting write access can lead to unintended secret creation in Bitwarden. The version of this tool is set to evolve, so keep an eye on updates and changes to ensure compatibility with your existing systems.
Key takeaways
- →Utilize the External Secrets Operator to synchronize secrets from Bitwarden into Kubernetes seamlessly.
- →Set the `refreshInterval` parameter to control how often secrets are updated, with a default of 15 minutes.
- →Deploy ESO using Helm to simplify installation and management of Custom Resource Definitions.
Why it matters
In production, managing secrets effectively is crucial for maintaining security and operational efficiency. ESO helps eliminate the risks associated with secret sprawl, ensuring that your applications have the necessary credentials without manual overhead.
Code examples
1helm repo add external-secrets https://charts.external-secrets.io
2
3helm install external-secrets external-secrets/external-secrets \
4 --namespace external-secrets \
5 --set installCRDs=true \
6 --set "bitwarden-sdk-server.enabled=true"1cat <<EOF | kubectl apply -f -
2apiVersion: external-secrets.io/v1
3kind: ExternalSecret
4metadata:
5 name: app-payment-creds
6 namespace: app-backend
7spec:
8 refreshInterval: 15m # Automatically rotate every 15 minutes
9 secretStoreRef:
10 name: bitwarden-global-store
11 kind: ClusterSecretStore
12 target:
13 name: payment-creds # The na
14EOF1cat <<EOF | kubectl apply -f -
2apiVersion: external-secrets.io/v1
3kind: ClusterSecretStore
4metadata:
5 name: bitwarden-global-store
6spec:
7 provider:
8 bitwardensecretsmanager:
9 apiURL: https://vault.bitwarden.eu./api
10 identityURL: https://vault.bitwarden.eu./identity
11 auth:
12 secretRef:
13 credentials:
14 key: token
15 name: bitwarden-access-token
16 namespace: external-secrets
17 bitwardenServerSDKURL: https://bitwarden-sdk-server.external-secrets.svc.cluster.local:9998
18 caProvider:
19 type: Secret
20 name: bitwarden-ca-certs
21 key: ca.crt
22 namespace: external-secrets
23 organizationID: <YOUR_ORGANIZATION_ID>
24 projectID: <YOUR_PROJECT_ID>
25EOFWhen NOT to use this
The official docs don't call out specific anti-patterns here. Use your judgment based on your scale and requirements.
Want the complete reference?
Read official docsUnified observability — logs, uptime monitoring, and on-call in one place. Used by 50,000+ engineering teams to ship faster and sleep better.
Try Better Stack free →Mitigating Staleness in Kubernetes Controllers: What You Need to Know
Kubernetes v1.36 introduces key features to tackle staleness in controllers, directly impacting your cluster's reliability. By leveraging atomic FIFO processing and the new ConsistencyStore, controllers can ensure they act on the most current data. This is a game-changer for production environments where stale data can lead to cascading failures.
Building a Memcached Operator with Go: A Practical Guide
Operators are a powerful way to extend Kubernetes, and building one with Go can streamline your application management. This guide walks you through creating a Memcached operator, focusing on the Custom Resource Definition (CRD) and the controller's role in reconciliation.
Mastering Admission Control in Kubernetes: What You Need to Know
Admission control is a critical gatekeeper in Kubernetes, ensuring that only valid requests reach your cluster. Understanding the difference between mutating and validating admission controllers can save you from costly misconfigurations.
Get the daily digest
One email. 5 articles. Every morning.
No spam. Unsubscribe anytime.