Taming Secret Sprawl in Multi-Account Kubernetes with External Secrets Operator
Managing secrets in a multi-account Kubernetes setup can lead to chaos if not handled properly. Secret sprawl occurs when secrets are scattered across various accounts and clusters, making it difficult to maintain security and compliance. The External Secrets Operator (ESO) addresses this challenge by providing a Kubernetes-native reconciliation model that synchronizes secrets from external secret management systems, like Bitwarden, directly into the Kubernetes Secret API.
At its core, ESO operates through three main components: a centralized secret management system (Bitwarden), the External Secrets Operator running in each Kubernetes cluster, and the Kubernetes Secrets that ESO generates and maintains. When you create an ExternalSecret resource, ESO retrieves the specified secret from Bitwarden and creates or updates the corresponding Kubernetes Secret. This process is governed by a reconciliation loop that continuously checks for updates based on a configured refresh interval, which defaults to 15 minutes. You can customize this behavior using parameters like refreshInterval to ensure your secrets are always current.
In production, you need to be aware of a few key points. First, ensure you have a Kubernetes cluster (EKS, AKS, GKE, etc.) and that you’ve installed both kubectl and helm. The installation of ESO involves adding the external-secrets Helm repository and deploying it with the necessary configurations. Be cautious with your access tokens; granting write access can lead to unintended secret creation in Bitwarden. The version of this tool is set to evolve, so keep an eye on updates and changes to ensure compatibility with your existing systems.
Key takeaways
- →Utilize the External Secrets Operator to synchronize secrets from Bitwarden into Kubernetes seamlessly.
- →Set the `refreshInterval` parameter to control how often secrets are updated, with a default of 15 minutes.
- →Deploy ESO using Helm to simplify installation and management of Custom Resource Definitions.
Why it matters
In production, managing secrets effectively is crucial for maintaining security and operational efficiency. ESO helps eliminate the risks associated with secret sprawl, ensuring that your applications have the necessary credentials without manual overhead.
Code examples
1helm repo add external-secrets https://charts.external-secrets.io
2
3helm install external-secrets external-secrets/external-secrets \
4 --namespace external-secrets \
5 --set installCRDs=true \
6 --set "bitwarden-sdk-server.enabled=true"1cat <<EOF | kubectl apply -f -
2apiVersion: external-secrets.io/v1
3kind: ExternalSecret
4metadata:
5 name: app-payment-creds
6 namespace: app-backend
7spec:
8 refreshInterval: 15m # Automatically rotate every 15 minutes
9 secretStoreRef:
10 name: bitwarden-global-store
11 kind: ClusterSecretStore
12 target:
13 name: payment-creds # The na
14EOF1cat <<EOF | kubectl apply -f -
2apiVersion: external-secrets.io/v1
3kind: ClusterSecretStore
4metadata:
5 name: bitwarden-global-store
6spec:
7 provider:
8 bitwardensecretsmanager:
9 apiURL: https://vault.bitwarden.eu./api
10 identityURL: https://vault.bitwarden.eu./identity
11 auth:
12 secretRef:
13 credentials:
14 key: token
15 name: bitwarden-access-token
16 namespace: external-secrets
17 bitwardenServerSDKURL: https://bitwarden-sdk-server.external-secrets.svc.cluster.local:9998
18 caProvider:
19 type: Secret
20 name: bitwarden-ca-certs
21 key: ca.crt
22 namespace: external-secrets
23 organizationID: <YOUR_ORGANIZATION_ID>
24 projectID: <YOUR_PROJECT_ID>
25EOFWhen NOT to use this
The official docs don't call out specific anti-patterns here. Use your judgment based on your scale and requirements.
Want the complete reference?
Read official docsIndustry-standard certifications built by the people behind Linux and Kubernetes. Earn the CKA — the gold standard Kubernetes administrator cert. OpsCanary readers get 30% off year-round with code OPSCANARY3.
Get CKA certified →Mastering Custom Resources in Kubernetes: A Guide for Operators
Custom resources in Kubernetes allow you to extend the API and tailor it to your needs. By combining them with custom controllers, you create a powerful declarative API that can manage complex applications. Dive into how this works and what you need to watch out for in production.
Forensic Container Checkpointing on Amazon EKS: What You Need to Know
Forensic container checkpointing is a game changer for stateful applications running on Amazon EKS. By leveraging the Kubelet Checkpoint API and CRIU, you can capture a container's full runtime state seamlessly. This article dives into the mechanics and real-world implications of implementing this powerful feature.
Why Your Controller's Cache Keeps the API Server Running Smoothly
Ever wonder how your Kubernetes controller can handle hundreds of calls per second without crashing the API server? The secret lies in the controller-runtime's local cache mechanism, which uses a combination of informers and stores to optimize data access. Dive in to understand the mechanics behind this powerful feature.
Get the daily digest
One email. 5 articles. Every morning.
No spam. Unsubscribe anytime.