How to rotate secrets safely

Topic: Security basics

Rotate passwords, API keys, and certificates on a schedule or after a suspected leak without causing outages. Add the new secret first, update consumers, then revoke the old one. Use this when implementing rotation or responding to a compromise.

Intent: How-to

Quick answer

  • Add or generate the new secret; deploy it to the secret manager or config so all consumers can read it. Update the application or service to use the new value (restart or reload if needed).
  • Verify that the new secret works (health check, test login, test API call). Then revoke or delete the old secret so it can no longer be used. Log the rotation and who performed it.
  • For zero-downtime rotation, support two valid secrets briefly (e.g. two API keys) so you can switch consumers in batches; then revoke the old key when all are updated.

Prerequisites

Steps

  1. Create new secret

    Generate a new password, key, or token; add it to the vault or secret manager. Do not revoke the old one yet; both may be valid during the transition.

  2. Update consumers

    Deploy the new secret to all services that use it (restart, reload config, or pull from vault). If multiple instances, roll out so each picks up the new secret before the old is revoked.

  3. Verify and revoke old

    Confirm every consumer works with the new secret (health checks, smoke tests). Then revoke or delete the old secret. Monitor for errors in case a consumer was missed.

  4. Document and alert

    Log rotation time and operator; update runbooks. Set a schedule for the next rotation or trigger rotation after an incident; alert if rotation fails.

Summary

Rotate by adding the new secret, updating all consumers, verifying, then revoking the old one. Support overlapping validity for zero-downtime where possible. Use this for scheduled rotation or after a leak.

Prerequisites

Steps

Step 1: Create new secret

Generate the new credential and store it in the vault. Leave the old one valid until consumers are updated.

Step 2: Update consumers

Deploy the new secret to all services; restart or reload so they use it. Roll out in batches if needed.

Step 3: Verify and revoke old

Confirm all consumers work with the new secret; then revoke the old one. Monitor for failures.

Step 4: Document and alert

Log the rotation; update runbooks. Schedule or trigger rotation; alert on failures.

Verification

New secret is in use everywhere; old secret is revoked; no outages; rotation is logged and repeatable.

Troubleshooting

Consumer still using old secret — Find the missed instance or config; update and retry revoke. Downtime during rotation — Use overlapping validity (two keys) and roll out consumers before revoking.

Next steps

Continue to