Design a Secrets Manager & KMS (Vault / AWS Secrets Manager style)

Envelope Encryption Flow

1. Key Hierarchy (KMS)

Key hierarchies, envelope boundaries, and rotation grace periods form the operational focus of enterprise secrets architectures.

The cryptographic architecture organizes into three distinct tiers: root keys never export from hardware boundaries, customer master keys wrap data encryption keys, and data encryption keys protect secret payloads.

• Root KMS Key (RKK): Generated inside hardware security modules and physically non-exportable, signing and wrapping regional customer master keys within FIPS 140-3 Level 3 or an equivalent certified HSM boundary boundaries.
• Customer Master Key (CMK): Provisioned per tenant or per environment, used exclusively to wrap and unwrap data encryption keys rather than encrypting secret payloads directly.
• Data Encryption Key (DEK): A random 256-bit AES key generated per secret version, encrypting payloads via AES-256-GCM and stored only in wrapped ciphertext form.
• CMK rotation: Annual automated rotation provisions a new key version, while a background re-wrap job unwraps data encryption keys using the previous version and re-wraps them under the new version without exposing secret plaintexts.

2. Secret Lifecycle API

Every lifecycle operation is authenticated and authorized against path policies. Mutations are durably committed before returning, while asynchronous audit publication and indexing do not extend the request path unnecessarily.

• Create and update: Accepts secret payloads over TLS, executes envelope encryption, persists the ciphertext blob and metadata row, designates the version as CURRENT, and emits an audit event.
• Read: Validates workload authorization, loads version metadata, requests KMS to unwrap the data encryption key, decrypts the blob in memory, emits an audit event, and returns the plaintext payload while zeroizing volatile memory.
• List: Returns metadata collections filtered by tenant namespace, omitting secret values to prevent bulk exfiltration.
• Delete: Enforces a soft-delete retention window of 30 days. After the retention and backup lifecycle requirements are satisfied, deleting the wrapped data encryption key provides cryptographic erasure while leaving the underlying ciphertext irrecoverable.
• Idempotency and audit durability: Create and update honor an Idempotency-Key. Read and deny events are appended to a local durable WAL or spool before the response completes, then published to Kafka asynchronously with retry and replay. If durable audit staging is unavailable, the request fails rather than silently losing the event.

3. Dynamic Secrets Engine ⭐

Dynamic secrets eliminate static passwords by provisioning ephemeral database roles on demand with automatic lease expiration.

POST /v1/dynamic/database/creds (lease request)

1. AuthZ: principal must have dynamic-creds:issue on database role
2. Dynamic engine connects to target DB with stored admin creds (from static secret)
3. CREATE USER svc_abc123 WITH PASSWORD 'random' VALID UNTIL now() + interval '1 hour'
4. GRANT readwrite TO svc_abc123
5. Store lease record: { lease_id, username, expires_at, secret_id_ref }
6. Return { username, password, lease_id, lease_duration: 3600 }
7. Audit: dynamic_lease.issued

Renew: POST /v1/dynamic/leases/{lease_id}/renew: extends VALID UNTIL (max 24h total)
Revoke: DELETE /v1/dynamic/leases/{lease_id}: atomically transition the lease to revoke_pending, then DROP USER and mark revoked, where retries are idempotent

Revoker daemon (every 30s):
  SELECT leases WHERE expires_at < NOW() AND status = 'active'
  → claim active lease with compare-and-set → DROP USER → mark revoked only after successful revocation → audit dynamic_lease.revoked

By provisioning unique database users per workload instance, the blast radius of a credential leak is confined to a single short-lived lease. The trade-off requires direct network connectivity from the secrets cluster to target database engines and bootstrapping admin credentials stored securely within the envelope store.

4. Rotation Scheduler

Zero-downtime rotation requires versioned credentials with overlapping grace windows so existing workload instances continue functioning while new instances adopt fresh credentials.

Zero-downtime rotation for static secret (e.g., DB password):

  T0: Secret v3 is CURRENT (password: old_pw)
  T1: Rotation job generates new_pw, creates v4 as PENDING
  T2: Rotation hook updates upstream (e.g., ALTER USER in PostgreSQL with admin creds)
  T3: Smoke test: dynamic engine verifies new_pw works (SELECT 1)
  T4: Transactionally CAS v3 CURRENT → v4 CURRENT and v3 → DEPRECATED (grace_until = T4 + 48h)
  T5: Workloads still on v3 succeed until grace ends. The SDK polls version every 5 min, and repeated retries are idempotent
  T6: After grace: v3 → DISABLED (reads return 410). Audit confirms zero v3 reads before destructive cleanup

Rollback: if smoke test fails at T3, delete v4, alert operator, v3 remains CURRENT. Every state transition is guarded by the secret version and expected current-version value so retries cannot promote a stale version.

Dynamic secret rotation (Vault-style):
  - Each lease is independent. There is no shared password across services
  - Lease TTL is 1h. Renewal is allowed up to max_ttl 24h
  - On expiry: background revoker claims the lease with compare-and-set, DROP USER in DB, then marks it revoked. Failed DB revocation remains retryable

• Rotation policies: Specify rotation intervals, grace period durations, and webhook target URLs for custom upstream systems.
• Coordination and idempotency: Leverages Leader Election to coordinate one scheduler per secret, while transactional compare-and-set state transitions make retries safe if a worker crashes after an upstream mutation.
• Verification smoke testing: Prohibits promoting a new version to CURRENT until an end-to-end verification query succeeds against the target database.

5. Policy Engine & RBAC

Path-based access policies enforce explicit deny precedence and tenant boundary verification before evaluating allow rules.

Policy example (path-based, Vault ACL style):

  path "secret/data/prod/db/*" {
    capabilities = ["read"]
    allowed_principals = ["role:payment-service", "role:order-service"]
  }

  path "secret/data/prod/db/admin" {
    capabilities = ["read"]
    allowed_principals = ["role:break-glass"]
    require_mfa = true
    max_ttl = 3600
  }

Evaluation order:
  1. Deny rules (explicit deny wins)
  2. Tenant boundary check (principal.tenant_id == resource.tenant_id)
  3. Allow rules by longest path match
  4. Default deny

Cache: policy decisions in Redis (TTL 60s), including a policy version in cache keys and invalidating through pub/sub. A durable policy version check prevents stale allows after a subscriber reconnects

6. Audit Pipeline

Audit durability is non-negotiable. All access attempts, including rejected requests, are durably staged before the request is considered complete and then published asynchronously to Kafka so the Kafka path does not block the read path.

Topic: secret-access-events
  Partitions: 128 (partition by tenant_id for per-tenant ordering during compliance export)
  Retention: 7 days hot replay buffer, while compliance retention lives in immutable WORM storage for 7 years
  Events: secret.read, secret.write, secret.rotate, dynamic_lease.issued, dynamic_lease.revoked,
          break_glass.access, kms.decrypt, policy.denied
  Payload: { event_id, tenant_id, secret_id, version, principal_id, principal_ip, trace_id, outcome, timestamp }

Consumer groups:
  1. audit-indexer: ClickHouse immutable store, compliance dashboards
  2. siem-forwarder: Splunk/Datadog security analytics
  3. anomaly-detector: bulk read, geo anomaly, break-glass usage
  4. billing-metering: API call counts per tenant (optional)

Sync path: GET /secrets/{id} → authZ → load ciphertext metadata → unwrap DEK if not cached → decrypt → durable audit staging → async Kafka publish
Async path: rotation scheduler, cross-region replication, re-wrap jobs

DLQ: secret-access-events-dlq (alert on any message because audit loss is P0)

Audit records correlate with Distributed Tracing via a trace identifier on every event. The compliance export API allows authorized auditors to extract cryptographically signed logs partitioned by tenant.

7. Cross-Region Replication and Failover

Multi-region deployments operate with active-active read capacity across regions, while mutation leadership relies on Leader Election to ensure single-region write coordination per secret.

• Metadata replication: Synchronizes PostgreSQL state or CockroachDB tables across regions asynchronously for the normal path, while critical writes can wait for a configured replication quorum when the RPO requires it.
• Blob replication: Leverages cross-region object storage replication with eventual consistency bounded by a five-second replication lag SLO.
• Multi-region keys: Uses provider-supported multi-region KMS keys or an equivalent regional key hierarchy so each region can perform local cryptographic operations without shipping secret plaintext across regions.
• Failover coordination: Follower regions serve local read traffic during normal operation and promote to primary coordinators upon leader loss.