Runbook: credential rotation (Atlas database users)

Applies to: MongoDB Atlas database users on the askflorence-staging and askflorence-prod clusters, and every consumer that carries their connection strings.

Status: written 2026-08-07 from an executed rotation of three leaked staging users (app_audit_writer, app_write, app_read). Before that date no rotation runbook existed; ENG-462 rotated the whole estate and left nothing written down. This is the procedure that should have existed then.

Founder-gated. Rotating a credential breaks every consumer holding the old one until it is updated and restarted. Do not begin without Taha.

0. Hard rules

From CLAUDE.md "Security rules". These are not negotiable and every step below is shaped by them.

Never put a secret value in a commit message, PR, Linear, Slack, a chat window, a log line, a screenshot, or a terminal transcript. Reference secrets by name (staging/mongodb/app-write).
Treat anything shareable, indexable, cacheable, or screenshottable as public.
Never git add . or git add -A — worktrees carry a gitignored .env.local symlink and it will get staged.
If a secret leaks: revoke and reissue first, update consumers second, scrub history third. Rotation comes first; history scrubbing is hygiene.

Three mechanical rules that follow from those:

Do	Don't	Why
`atlas api databaseUsers updateDatabaseUser --file <body>`	`atlas dbusers update --password <value>`	The flag puts the password in `argv` (visible in `ps`) and shell history. Its `--role` flag also replaces preexisting roles.
`aws secretsmanager put-secret-value --secret-string "file://$tmp"`	`--secret-string "<value>"`	Same argv exposure.
`put-secret-value`	`create-secret`	`create-secret` defaults to the AWS-managed KMS key. The shells already exist in Terraform (`infra/envs//secrets.tf`) with a CMK and `lifecycle.ignore_changes = [secret_string]`; you are writing a new version*.

Generation pattern (never cat the file, never echo the value):

bash

PW_FILE=$(mktemp); chmod 600 "$PW_FILE"
openssl rand -base64 96 | tr -dc 'A-Za-z0-9' | head -c 40 > "$PW_FILE"

Alphanumeric-only is deliberate: the password gets embedded in a URI, and restricting the alphabet removes any percent-encoding question.

The delivery pattern (mktemp -d + trap + chmod 600 + file://) is implemented in scripts/aws/populate-staging-secrets.sh and scripts/aws/rotate-atlas-user.sh. Read one before writing anything new.

1. Inventory first — this is where rotations go wrong

Do not trust infra/atlas/access-matrix.ts for this step. It is the declared model, and on 2026-08-07 it was wrong in a way that would have caused an outage: it attributed MONGODB_REFERENCE_URI to a user (app_read_staging) that no secret had referenced since ENG-279.

Derive the truth from the live secret values. Print only the username and host:

bash

AWS_PROFILE=<profile> aws secretsmanager get-secret-value \
  --secret-id <name> --query SecretString --output text \
| python3 -c "import sys,re; s=sys.stdin.read().strip(); \
m=re.match(r'mongodb(?:\+srv)?://([^:]+):[^@]+@([^/?]+)',s); \
print(f'{m.group(1)}  @  {m.group(2)}' if m else '(unparsed)')"

Sweep both AWS accounts (askflorence-staging, askflorence-prod) and all mongo-ish secrets, not just the <env>/mongodb/* prefix — other apps (saloon, registry, switchboard, pulse, telegraph) keep their URIs outside that prefix.

Then cross-check the wiring:

bash

npm run audit:atlas-env-vars   # declared bindings vs Terraform
npm run audit:ecs-task-def     # declared secrets vs ECS bindings

And enumerate live ECS consumers — the app service is not the only one:

bash

AWS_PROFILE=<profile> aws ecs list-task-definition-families --status ACTIVE
AWS_PROFILE=<profile> aws ecs describe-task-definition --task-definition <fam> \
  --query 'taskDefinition.containerDefinitions[].secrets[?contains(name,`MONGO`)].[name,valueFrom]' \
  --output text

Write the result down before touching anything.

Traps that have actually bitten us

One secret, many bindings. staging/mongodb/app-write backs ten ECS env-var bindings: MONGODB_WRITE_URI plus six legacy aliases (MONGODB_URI_PLANS_WRITE, _SURVEY_WRITE, _AGENTS_WRITE, _AGENTS_ADMIN, _WAITLIST_WRITE, _HUBSPOT_SYNC_WRITE) on the app task, plus three on staging-app-smoke-task. MONGODB_URI_AUDIT_READ aliases mongodb/app-read. The fan-out is at the binding layer, so one put-secret-value covers all of them — but you must know they exist to reason about blast radius.

The cross-cluster trap. prod/mongodb/reference-uri carries the staging cluster's app_read user, reached over PrivateLink (askflorence-staging-pl-0.*). Rotating staging app_read is therefore a production-touching change: it requires redeploying askflorence-prod-app, not just staging. It backs getReferenceDb() → /api/providers/covered + /api/drugs/covered.

Per-secret hosts differ. The same user appears behind an SRV hostname in staging and a PrivateLink hostname in prod. Never copy one secret's value into another. Substitute only the password component of each secret's existing URI so its host and query params survive verbatim.

Consumers that self-heal. .github/workflows/playwright.yml and nightly-drift-check.yml fetch staging/mongodb/app-write from Secrets Manager at run time. No stored CI secret to update.

Consumers that don't. The canonical ~/Developer/askflorence/.env.local (every worktree symlinks to it) and any already-running local dev process. Those need the file updated and the process restarted.

2. Order of work

Rotate one user at a time. Never bulk-rotate. Order by blast radius, smallest first, so the first rotation is a rehearsal of the mechanics:

app_audit_writer — no live consumer (getAuditDb() is Phase 5). Safest.
app_read / app_write — live consumers.

For each user: generate → update Atlas → write every Secrets Manager target → update .env.local → redeploy every consuming service → verify → move on.

Timing exception. Between the Atlas password change and the ECS redeploy, running tasks hold a dead credential. If two users both back the same service, rotating them back-to-back and issuing one redeploy is shorter total degradation than two sequential rolling deploys. Decide deliberately and say so.

Use the tool rather than doing it by hand:

bash

bash scripts/aws/rotate-atlas-user.sh <username> --dry-run   # preflight only
bash scripts/aws/rotate-atlas-user.sh <username>

It refuses to proceed unless every declared target already carries the user being rotated, so a mis-mapped target aborts before the Atlas password changes rather than stranding a live consumer.

Then redeploy every service that binds the affected secrets:

bash

AWS_PROFILE=askflorence-staging aws ecs update-service \
  --cluster askflorence-staging --service askflorence-staging-app --force-new-deployment
AWS_PROFILE=askflorence-staging aws ecs wait services-stable \
  --cluster askflorence-staging --services askflorence-staging-app

Secrets are resolved at task start, and the task definition references the secret ARN without a version, so a forced deployment picks up AWSCURRENT. No Terraform apply and no new task-definition revision are required for a value-only rotation.

Finally, shred every temp file (rm -f) and confirm the trap fired.

3. Verification ladder

A rotation that authenticates from your laptop but leaves the running service on a stale secret is a broken rotation that looks fine. Climb the whole ladder.

bash

# 1. secret hygiene: no trailing newline, no literal \n, not a placeholder
AWS_PROFILE=askflorence-staging node scripts/audit/validate-secrets.js
AWS_PROFILE=askflorence-prod    node scripts/audit/validate-secrets.js

# 2. declared-vs-wired: nothing orphaned, nothing missing
npm run audit:atlas-env-vars
npm run audit:ecs-task-def

# 3. the credential itself: new works everywhere, OLD IS REJECTED
node scripts/aws/verify-atlas-credential.mjs <username> --old <old-uri-file>

# 4. the full gate
npm run preflight
npm test
PLAYWRIGHT_BASE_URL=https://origin.stage.askflorence.health npm run test:e2e

# 5. the proof: end-to-end against the deployed service, bypassing CloudFront
SMOKE_TARGET_URL=https://origin.stage.askflorence.health \
MONGODB_WRITE_URI=... npm run smoke:post-deploy

# 6. live access control unchanged
npx tsx scripts/audit/staging-cluster-drift.ts

Smoke checks map onto the users you touched: POST /api/eligibility → app_read; POST /api/waitlist + /api/agents/discovery → app_write; POST /api/providers/covered + /api/drugs/covered → getReferenceDb().

Two things the ladder will not tell you unless you look

The smoke does NOT catch a dead reference credential. Since ENG-330, both reference fallbacks catch the error and return an empty Map, so the routes degrade to CMS-only output with a [ref-db] WARN instead of 500ing. The smoke asserts only that data.coverage is an array, so it passes either way. To actually prove the reference path, assert on the enrichment field:

bash

curl -s -X POST https://askflorence.health/api/providers/covered \
  -H 'Content-Type: application/json' \
  -d '{"npis":["1962994962"],"plan_id":["68781UT0200014"],"year":2026}' \
| python3 -c "import json,sys; c=json.load(sys.stdin)['data']['coverage']; \
print('ENRICHED' if any(x.get('network_tier') for x in c) else 'DEGRADED')"

Run it against prod — that is the only thing that exercises the PrivateLink path with the rotated credential. ENRICHED is the pass.

(Note: /api/drugs/covered with the smoke's Utah fixture plan legitimately returns no drug_tier — that rxcui/plan pair is not in formularies_staging. Use the providers check as the reference-path signal.)

PrivateLink targets are not reachable from a laptop. verify-atlas-credential.mjs reports them as SKIPPED, which is correct, not a gap: it is the same Atlas user, so authenticating against the public SRV host proves the password. The network path is proven by the prod curl above.

Atlas propagation is eventually consistent

A freshly rotated password can take tens of seconds to propagate. During the 2026-08-07 rotation, app_audit_writer and app_write authenticated immediately while app_read briefly showed the old password working and the new one failing — which reads exactly like a failed rotation.

Retry the verification before concluding anything. Only treat it as a real failure if it persists across several minutes. Do not "fix" it by rotating again; that just moves the target.

Confirm the old credential is dead

This is the half people skip, and it is the half that proves the rotation landed rather than silently no-oping. rotate-atlas-user.sh writes the old URI to $ROTATE_OLD_URI_OUT for exactly this purpose. Attempting auth with it must be rejected. Shred the file immediately after.

4. Close-out

Shred every temp file holding an old or new credential.
Restart any running local dev process so it picks up the new .env.local.
If you deleted a user, check for an orphaned custom role. Deleting an Atlas user does not delete the custom role it held. The role lingers, grants nothing while unattached, and quietly inflates the access surface an auditor reads. Two had accumulated this way before anyone looked (role_reader_reference from app_read_staging, and role_reader_local_staging from ENG-462's app_read_local_staging). Sweep for them:
bash
```
atlas customDbRoles list --projectId <id> -o json   # every custom role
atlas dbusers list --projectId <id> -o json         # roles actually held
```
Any role in the first list that appears in no user's roles[] — and is not an inheritedRoles target of another role — is an orphan. Delete it.
Update infra/atlas/access-matrix.ts if any user state changed, then npm run docs:atlas to regenerate the human-readable matrix.
Add a change-log entry to docs/infrastructure/change-log.md — what rotated, which consumers moved, verification result. Names only, never values. (ENG-462 rotated the estate and wrote nothing down; the next person paid for it.)
Tell anyone sharing the environment to restart their sessions.

5. If a rotation goes wrong

New credential rejected everywhere, old one still works. Almost certainly Atlas propagation (see above). Wait and retry before doing anything else.

New credential works, service still failing. The running tasks have not been replaced. Confirm the deployment actually rolled:

bash

AWS_PROFILE=<profile> aws ecs describe-services --cluster <c> --services <s> \
  --query 'services[0].deployments[].[status,rolloutState,runningCount,desiredCount]' --output text

Expect a single PRIMARY / COMPLETED.

A target was missed. Re-run the inventory sweep in §1. Every secret carrying the user must have been rewritten; a missed one leaves a consumer on a dead credential.

You need to roll back. You cannot recover the old password — Atlas stores a hash, and the old value is intentionally shredded. Roll forward: rotate again and make sure every target is in the map this time. Secrets Manager keeps prior versions (AWSPREVIOUS), so a mis-written secret value can be recovered even though the password itself cannot:

bash

AWS_PROFILE=<profile> aws secretsmanager get-secret-value \
  --secret-id <name> --version-stage AWSPREVIOUS --query SecretString --output text