How to manage Kubernetes secrets with Vault securely

Why the default Kubernetes secret model fails in production

Managing sensitive data, such as database passwords and SSL certificates, is one of the biggest challenges when operating modern distributed systems. In the cloud-native ecosystem, Kubernetes uses a built-in resource called a Secret to store this information. In practice, this means any key your application needs to function is stored in plain text, merely disguised by a basic encoding scheme called Base64, which anyone can reverse in seconds. To make matters worse, these credentials are saved in a central cluster database called etcd, usually without encryption at rest by default in standard installations. If an attacker manages to read this database or breach a neighboring pod with broad permissions, they gain full access to your entire infrastructure.

To shield production environments against these vulnerabilities, organizations adopt dedicated identity management and encryption tools, with HashiCorp Vault being the most robust industry standard. Vault acts as a heavily fortified digital vault that centralizes the custody of passwords, API keys, and certificates. It does much more than just store data: it can generate credentials on-demand, providing temporary database usernames and passwords that automatically expire after a few minutes. This approach drastically minimizes the impact if a token is intercepted over the network. If you want to learn how to structure modern applications with high operational reliability, check out our guide on how to design decoupled microservices with Domain-Driven Design.

How the External Secrets Operator connects the vault to the cluster

Although Vault is excellent for storing secrets, it does not natively speak the Kubernetes API language. This is precisely where the External Secrets Operator comes in—an intelligent controller installed inside your cluster that acts as a communication bridge. In practice, it monitors resources you define and fetches protected data directly from the external vault, injecting it automatically into the local ecosystem. This allows you to continue using standard application configuration structures without rewriting initialization code to deal with complex third-party APIs.

The operation of this bridge relies on two main concepts: the SecretStore, which defines where the operator should connect and authenticate with the vault, and the ExternalSecret, which specifies exactly which data to retrieve and into which local secret it should be mapped. When you apply these configuration files to your cluster, the operator makes a secure request using encrypted protocols, fetches the updated password, and creates the corresponding Kubernetes object. To ensure this automation workflow runs smoothly alongside the rest of your infrastructure, you can also explore the recommendations in our article on how to use GitOps to manage Kubernetes clusters.

Implementing dynamic injection step by step

To put this architecture into practice, the first step is configuring authentication between your Kubernetes cluster and Vault, typically utilizing native Kubernetes Service Accounts to ensure only authorized pods obtain tokens. Next, you install the synchronization operator using package managers or direct manifests. Below is a practical configuration example of a resource that instructs the operator to fetch an API key from the external vault and save it locally.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-secret
  namespace: production
spec:
  refreshInterval: '1h'
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: app-db-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: secret/data/production/postgres
        property: password

After applying this manifest, the operator executes the fetch cycle periodically, ensuring any change made in the central vault is reflected in the runtime environment without requiring manual application restarts. However, it is important to monitor controller logs to identify potential permission failures or temporary network unreachability between servers.

Security best practices for production environments

Operating secrets at scale demands discipline and strict access permissions granted to each system component. A fundamental golden rule is the principle of least privilege, which dictates that each microservice should only see the keys strictly necessary for its operation, never having access to other modules' credentials. Furthermore, avoid storing secrets in traditional environment variables inside pod manifest files, as any monitoring tool or error diagnostic process can accidentally expose this information in operating system logs.

Another essential practice to guarantee cluster integrity is the periodic rotation of certificates and access keys. With the combined use of Vault and External Secrets, you can configure policies so database tokens and security certificates expire automatically every few hours. When the credential expires, the operator fetches a fresh version and updates the Kubernetes object, triggering a graceful restart of affected pods. This routine ensures that even if an old key is compromised during a data leak, its useful window of exploitation by malicious actors is extremely short.

Final considerations on resilience and operations

Adopting a robust credential management strategy transforms the security posture of any organization running Kubernetes in production. Although the initial learning curve to configure Vault and External Secrets may seem complex, the gains in auditing, regulatory compliance, and operational peace of mind vastly outweigh the implementation effort. Keeping keys out of version control and automating dynamic injection shields applications from everyday human oversight.

Ultimately, cloud information security does not rely on a single silver bullet tool, but rather on a layered architecture where isolated failures are contained rapidly. By eliminating static credentials and adopting short lifespans for passwords and certificates, your engineering team gains the stability required to scale complex systems with absolute confidence in production behavior.