S3 agent survey uploads - setup

The /agent-discovery survey accepts optional blank-template uploads (client consent forms, Medicaid-rejection attestations, other verification forms). Files are written via src/app/api/agents/discovery/upload/route.ts to the existing askflorence-data S3 bucket in the management account (askflorencehealth, 778477254880).

Using the same bucket as PUF source files so the Phase 4-5 account-based migration (see aws-organizations.md) moves everything in one coordinated cut.

Target location

• Bucket: askflorence-data
• Account: 778477254880 (management / askflorencehealth)
• Region: us-east-1
• Prefix: agent-survey-uploads/{client_consent|medicaid_attestation|other_verification}/{ts}-{randomId}-{safeName}

Bucket hardening (one-time)

If not already applied, ensure the following on askflorence-data:

1. Block public access - all four settings on.
2. Versioning - enabled.
3. Default encryption - SSE-KMS with a customer-managed CMK (preferred) or SSE-S3 AES256 (fallback). Applied 2026-04-21: alias/askflorence-data -> arn:aws:kms:us-east-1:778477254880:key/88df2ce4-b694-4181-91b1-d0efc107429a in the management account, annual rotation on. Historical note: bucket started on SSE-S3 AES256 (pre-migration); switched to SSE-KMS with this CMK during AWS migration Phase 2 intake work.
4. Deny non-SSL bucket policy (matches askflorence-org-config-* pattern).
5. Lifecycle - for agent-survey-uploads/ prefix: abort incomplete multipart uploads after 1 day; transition to Glacier Instant Retrieval after 180 days; expire non-current versions after 90 days.
6. Object Lock - consider enabling governance-mode retention of 6 years to satisfy the HIPAA retention target (CLAUDE.md). Requires bucket-level Object Lock enabled at creation, so may need a new bucket rather than retrofit - tracked for Phase 4-5.

IAM user for the Vercel runtime

Create a dedicated user vercel-agent-survey-uploader with programmatic-only access and the minimum policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PutAgentSurveyUploads",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::askflorence-data/agent-survey-uploads/*"
    },
    {
      "Sid": "UseKmsCmkIfConfigured",
      "Effect": "Allow",
      "Action": ["kms:GenerateDataKey", "kms:Encrypt"],
      "Resource": "arn:aws:kms:us-east-1:778477254880:key/<CMK-ID>"
    }
  ]
}

The second statement is only needed if the bucket uses SSE-KMS with a customer-managed CMK. Drop it if the bucket uses SSE-S3.

Admin read access is a separate user/role and is not scoped in this runbook - tracked with the Phase 5 admin dashboard.

Vercel env vars (Production + Preview)

Set the following on the askflorence Vercel project:

Var	Value
AWS_REGION	us-east-1
S3_AGENT_SURVEY_BUCKET	askflorence-data
S3_AGENT_SURVEY_KMS_KEY_ID	(optional) CMK ARN if SSE-KMS; leave unset to fall back to SSE-S3
AWS_ACCESS_KEY_ID	from the new IAM user
AWS_SECRET_ACCESS_KEY	from the new IAM user

Don't reuse these creds for other work - this user should only ever call s3:PutObject under the upload prefix.

Sanity check after deploy

# From a machine with admin creds:
aws s3 ls s3://askflorence-data/agent-survey-uploads/ --recursive

Submit a dummy entry on /agent-discovery with a blank PDF; verify the object appears under the client_consent/, medicaid_attestation/, or other_verification/ prefix with the expected KMS encryption header:

aws s3api head-object --bucket askflorence-data \
  --key agent-survey-uploads/client_consent/<your-key>

Expect ServerSideEncryption: aws:kms (if CMK configured) or AES256.

Malware scanning

Enabled 2026-04-21 as part of AWS migration Phase 2.5. Every object uploaded under agent-survey-uploads/ is scanned by GuardDuty Malware Protection for S3 within ~60 seconds and tagged with the result.

• Plan ID: d4ced6e0c14fe707c26d in mgmt account 778477254880
• Tag applied: GuardDutyMalwareScanStatus — values NO_THREATS_FOUND (clean) or THREATS_FOUND (malware detected)
• Full details: docs/infrastructure/guardduty-setup.md

To manually inspect the scan result for a specific upload:

aws s3api get-object-tagging --bucket askflorence-data --key <key> \
  --query 'TagSet[?Key==`GuardDutyMalwareScanStatus`].Value' --output text

If the tag value is THREATS_FOUND, quarantine the object immediately and follow the PHI-deletion procedure below regardless of whether the object appears to contain PHI. Alert on this via the EventBridge rule that Phase 11 will add.

PHI posture

Agents are warned in-app to upload blank templates only and must tick a per-file confirmation before the upload button enables. If a file is later found to contain PHI, delete manually:

aws s3 rm s3://askflorence-data/<offending-key>
aws s3api delete-object --bucket askflorence-data \
  --key <offending-key> --version-id <version-id>  # remove from versioning too

Log the deletion + reason in agent_audit_log when Phase 5 lands (see CLAUDE.md).

Verification, drift detection, and incident response (ENG-324)

Added 2026-05-15 as part of ENG-286 audit follow-up. Existing setup is correctly configured — this section formalizes the re-verification cadence, drift detection, and incident response so future operators have a paint-by-numbers procedure.

Verified configuration (as of 2026-05-15)

Field	Value	How to re-verify
Plan ID	d4ced6e0c14fe707c26d	aws guardduty list-malware-protection-plans
Status	ACTIVE	aws guardduty get-malware-protection-plan
Bucket	askflorence-data	--query 'ProtectedResource.S3Bucket.BucketName'
ObjectPrefixes	["agent-survey-uploads/"]	--query 'ProtectedResource.S3Bucket.ObjectPrefixes'
Actions.Tagging.Status	ENABLED	--query 'Actions.Tagging.Status'
Resource lifecycle	CLI-managed (tag ManagedBy=cli-phase2.5)	--query 'Tags'
Terraform drift detection	infra/envs/management/guardduty-malware-protection.tf	cd infra/envs/management && terraform plan

Quarterly re-verification

Run every quarter (or after any GuardDuty / IAM change):

aws guardduty get-malware-protection-plan \
  --malware-protection-plan-id d4ced6e0c14fe707c26d \
  --region us-east-1 \
  --profile askflorence-mgmt

Confirm every field above matches. If anything drifts — investigate immediately. The Terraform postcondition in infra/envs/management/guardduty-malware-protection.tf will fail at the next terraform plan (passive guard), but proactive checks beat passive ones.

EICAR end-to-end smoke test (proves the control actually works)

EICAR is the industry-standard benign test string detected by every malware scanner. Use it to verify the full pipeline: upload → GuardDuty scan → detection → object tag. Run quarterly alongside the verification check above OR after any change to the plan.

# Create the EICAR test file
echo 'X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*' > /tmp/eicar-test.txt

# Upload to the protected prefix
TIMESTAMP=$(date +%Y%m%d%H%M%S)
KEY="agent-survey-uploads/_eicar-test/eicar-${TIMESTAMP}.txt"
aws s3 cp /tmp/eicar-test.txt "s3://askflorence-data/${KEY}" --profile askflorence-mgmt

# Wait 2-5 minutes for GuardDuty to scan, then verify the tag landed
sleep 180
aws s3api get-object-tagging \
  --bucket askflorence-data \
  --key "${KEY}" \
  --profile askflorence-mgmt
# Expected TagSet:
#   - GuardDutyMalwareScanStatus = THREATS_FOUND
#   - GuardDutyFindingId = <finding-uuid>

# Verify a finding was created
DETECTOR_ID=$(aws guardduty list-detectors --region us-east-1 --profile askflorence-mgmt --query 'DetectorIds[0]' --output text)
aws guardduty list-findings \
  --detector-id "${DETECTOR_ID}" \
  --finding-criteria '{"Criterion":{"type":{"Eq":["Object[malware]:S3Object/Malicious"]}}}' \
  --region us-east-1 \
  --profile askflorence-mgmt

# CLEANUP: delete the test file
aws s3 rm "s3://askflorence-data/${KEY}" --profile askflorence-mgmt
rm -f /tmp/eicar-test.txt

If GuardDuty does NOT tag the object within 5 minutes:

1. The control is misconfigured — likely IAM permission issue on GuardDutyMalwareProtectionS3Role or KMS-decrypt failure
2. Open a Linear issue immediately; treat as a P1 security incident
3. DO NOT mark the verification as passing until resolved

Terraform drift check

From infra/envs/management/:

terraform plan

The data.external.guardduty_malware_protection_plan data source fetches the live plan config and four postcondition blocks assert: status is ACTIVE, bucket matches, prefix scope matches, tagging is ENABLED. Any drift = terraform plan fails with a clear error message pointing back to this runbook.

Drift detection is passive (only runs when someone runs terraform plan in the management env). For active detection, the quarterly verification above is the primary control.

Incident response — malware detected on a REAL upload

When GuardDuty creates a Object[malware]:S3Object/Malicious finding on a real (non-EICAR) object:

Immediate (within minutes)

1. Identify the affected object — pull the finding's Service.ResourceDetails.S3BucketDetails[0] ARN + key.
2. Quarantine the object — DO NOT delete (forensics needed). Move to a quarantine prefix:

   aws s3 mv \
     s3://askflorence-data/agent-survey-uploads/<key> \
     s3://askflorence-data/_quarantine/<key> \
     --profile askflorence-mgmt

3. Flag the agent's record — pre-Phase-5: manually update Mongo agent_waitlist_submissions or agent_survey_responses row with flagged_malware_upload: true + timestamp. Phase 5+: disable the account.
4. Notify — post to ops, file Linear P0 issue tagged security-incident.

Forensics (within hours)

5. Pull access logs from CloudTrail for the upload event (who, when, source IP, user agent).
6. Cross-reference with the agent's other activity in Mongo — any other suspicious patterns?
7. Document in the Linear issue per docs/security-compliance/incident-response-plan.md.

Recovery + lessons (within days)

8. Was the malware signature one the magic-byte sniff (src/app/api/agents/discovery/upload/route.ts) should have caught? If yes, harden the magic-byte detection.
9. Decide on retroactive re-scan policy — GuardDuty auto-scans new uploads only; historical re-scan requires a separate workflow.
10. Update this runbook with what was learned.

Rotation — GuardDuty Malware Protection plan

If the plan ever needs to be recreated (scope expansion, region change, etc.):

1. Delete the old plan via CLI — this stops scanning, so coordinate carefully:

   aws guardduty delete-malware-protection-plan \
     --malware-protection-plan-id d4ced6e0c14fe707c26d \
     --region us-east-1 \
     --profile askflorence-mgmt

2. Recreate via aws guardduty create-malware-protection-plan ... with new scope.
3. Capture the new plan ID; update infra/envs/management/guardduty-malware-protection.tf local.guardduty_malware_plan_id.
4. Update this runbook's "Verified configuration" table.
5. Run EICAR smoke test to verify new plan is scanning.
6. CloudTrail audit trail: aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=CreateMalwareProtectionPlan.

Compliance refs

• HIPAA Security Rule §164.308(a)(1) — security management process
• SOC 2 CC6.6 — logical and physical access controls
• CMS EDE Phase 3 §6 — content scanning of uploads

For auditors asking "show me malware scanning is in place on user uploads": this runbook + the Terraform drift detection + the most recent quarterly EICAR test output is the evidence package.

Cross-references

• ENG-286 — audit doc parent
• ENG-324 — this verification work
• ENG-289 / PR-C — magic-byte sniff on /api/agents/discovery/upload route (pre-GuardDuty file-type validation)
• infra/envs/management/guardduty-malware-protection.tf
• Incident Response Plan
• GuardDuty Setup