Insights · Security Architecture
Secrets management in production: vaults, rotation and workload identity
Centralize every secret, prefer dynamic credentials over long-lived ones, rotate automatically, and rehearse break-glass access.
The leak data makes this non-optional. A mixed-methods study of secret handling (Krause et al., arXiv 2022) found 30.3% of surveyed developers had encountered secret leaks in their repositories — and that developers systematically struggle to estimate how dangerous a leaked credential is. Scanning public GitHub issue reports, another 2024 study confirmed 5,881 genuine secrets among 54,148 candidates (Ahmed et al., arXiv). And in infrastructure code specifically, SLAC’s static analysis of 50,323 IaC scripts found 46,600 security-smell occurrences — including 7,849 hard-coded passwords (Rahman et al., ACM TOSEM). Secrets leak because they are everywhere; the cure is to make them nearly nowhere.
What belongs in a vault — and what the vault buys
The OWASP Secrets Management Cheat Sheet is the rulebook: never hardcode; “centralize and standardize”; least privilege per secret; automate rotation; treat CI/CD as production (pipeline secrets are crown jewels); and shift left with secret scanning so nothing reaches the repository in the first place. A vault (HashiCorp Vault is the reference design) earns its operational cost by doing four things scattered storage can’t: it centralizes secrets, rotates credentials, generates credentials on demand, and audits every client interaction.
- Dynamic secrets. Instead of storing a database password, the vault mints one per connection — short TTL, auto-revoked, zero standing credential to steal. The database sees many short-lived users instead of one immortal one.
- Encryption as a service. The transit engine performs crypto operations without storing anything — “never persists the data you send it” — with versioned keys and re-encryption (
rewrap) that never exposes plaintext; NIST’s AES-GCM guidance effectively caps a key version at ~2³² encryptions, which transit rotation handles mechanically. - Audit. Every read of every secret, by every identity, logged. When a credential shows up on a paste site, “who could have read it” is a query, not an investigation.
Auto-unseal: the root-of-trust decision
A vault’s own root key is the deepest secret in the estate, and unsealing it at boot is where architecture shows. Vault’s seal documentation lays out the hierarchy — encryption key, wrapped by root key, unsealed by key shares — and the trade: Shamir shares keep humans in the loop but “unsealing it using Shamir seals is a manual process” (and a dependency on humans being awake); auto-unseal delegates custody to a KMS/HSM-class device, buying automation at the price of a hard dependency on that provider. On the E-Tax programme this was a recorded decision — SoftpHSM as the explicitly interim auto-unseal mechanism, with migration to a hardened KMS/HSM as the superseding path — because interim arrangements without records become permanent ones. The ADR practice exists precisely for decisions like this: root-of-trust placement, with a named expiry.
Rotation: automate it, and mind the cryptoperiod
OWASP’s rotation logic is window-shrinking: “restarting the application does not itself revoke stolen credentials” — only rotation does, and it must be scheduled, not heroic. AWS Secrets Manager states the managed pattern: “configure an automatic rotation schedule … replace long-term secrets with short-term ones,” with rotation lambdas and runtime retrieval replacing anything hardcoded. For the cryptographic layer, NIST SP 800-57 Part 1 Rev. 5 supplies the authority on cryptoperiods — the bounded lifespan of a key’s originator- and recipient-usage periods. One nuance the cheat sheet preserves from NIST: human password rotation on a fixed schedule is out of fashion — rotate human credentials on evidence of compromise, but machine credentials on schedule, always.
Workload identity: delete the shared secret entirely
The end state is not better-managed static secrets; it is no static secrets. SPIFFE — the CNCF “universal identity control plane for distributed systems” — gives every workload a “strongly attested, cryptographic identity”: a SPIFFE ID (a URI naming the workload in a trust domain) proven by an SVID — “the document with which a workload proves its identity” — preferably a short-lived X.509 certificate. Services authenticate to each other with mTLS bound to workload identity (the zero-trust east-west posture), and pull their application secrets with that identity instead of shared tokens. This is the mechanism that makes default-strict mTLS across hundreds of services administrable: identities are issued, rotated and revoked by the platform, not configured by humans.
In Kubernetes specifically, know what you have: the Secrets documentation is blunt that base64 encoding “obscures them but does NOT provide any useful level of confidentiality,” that Secrets live unencrypted in etcd unless encryption-at-rest is configured, and that list/watch on Secrets or a privileged container exposes far more than any one workload needs. Minimum hygiene: etcd encryption on, least-privilege RBAC, and external secret injection from the vault rather than hand-managed manifests.
Break-glass: the exception you rehearse
When the vault is down (or the automation that depends on it is), you need credentials outside the system — OWASP’s break-glass guidance is specific: emergency credentials in a secondary system, physically or logically separated from the primary, tested, with every opening alarmed and audited. Break-glass that has never been exercised is a second outage waiting inside the first one.
No secrets in code, config, or pipeline variables — scanning enforces it in CI; every machine secret has an owner, a TTL and an automatic rotation; vault root-of-trust is a recorded decision with a named interim; workloads authenticate by identity, not shared tokens; break-glass lives outside the estate and has been opened in a drill.
Sources
- HashiCorp Vault documentation
- HashiCorp Vault, “Seal/Unseal” — Shamir vs auto-unseal
- HashiCorp Vault, “Transit secrets engine” — encryption as a service
- OWASP Secrets Management Cheat Sheet
- SPIFFE — Secure Production Identity Framework For Everyone
- SPIFFE concepts — SPIFFE IDs and SVIDs
- Kubernetes documentation, “Secrets”
- AWS, “What is AWS Secrets Manager?”
- NIST SP 800-57 Part 1 Rev. 5, “Recommendation for Key Management” (2020)
- Krause, Klemmer, Huaman, Wermke, Acar & Fahl, “Pushed by Accident: Strategies of Handling Secret Information in Source Code Repositories” (arXiv, 2022)
- Ahmed, Rahman, Wahab, Uddin & Shahriyar, “Secret Leak Detection in Software Issue Reports using LLMs” (arXiv, 2024)
- Rahman, Rahman, Parnin & Williams, “Security Smells in Ansible and Chef Scripts” (ACM TOSEM; arXiv)