Comprehensive Plan Data Audit

Status: Active. Initial run April 14, 2026. Scope: All 30 federal (FFM) states, ~28K zip codes, ~4K plans Goal: Continuously prove our database matches Healthcare.gov accuracy without runtime CMS dependency

Source of truth

• Federal states (30): CMS Marketplace API at marketplace.api.healthcare.gov/api/v1/
• NY: NY DFS Exhibit 23 + NYSOH (separate methodology, see new-york-2026.md)
• Other SBE states (zip→county only): CMS Marketplace API /counties/by/zip (the same endpoint that backs Tier 1). Deeper plan-level data lives on each SBE marketplace and is out of scope.

Tiered audit structure

Tier	Scope	Method	Cadence
1 — Zip → County (federal-30)	Every unique federal-state zip (~18.8K)	CMS /counties/by/zip vs our zip_county federal-state docs	Monthly
1.5 — Zip → County (SBE-state)	Every unique SBE-state zip (~13K)	CMS /counties/by/zip vs our zip_county SBE docs (with sbeRedirect)	Monthly
2 — Zip → Premium	Every federal zip-county entry (~28K)	CMS /plans/search vs our cheapest Silver	Monthly
3 — Scenario matrix	200 stratified counties × 10 scenarios	Realistic households w/ Medicaid auto-adjustment	Monthly
4 — Internal integrity	All 4K plans + all 28K zips	Pure DB queries, no CMS calls	Every commit

Standard test parameters

Unless otherwise noted, all audits use:

• Year: 2026 (current enrollment year)
• Market: Individual
• State filter: 30 federal FFM states
• Match tolerance:
  • Tier 1: exact set equality (FIPS codes)
  • Tier 2: $0.01 (cheapest Silver age 35)
  • Tier 3: $0.50 (allows for rounding in household premium calculations)

Medicaid flow handling (CRITICAL)

Problem: CMS API does not return APTC-discounted plans for users below 138% FPL. It returns is_medicaid_eligible: true with no APTC. Our app handles this differently — it auto-bumps the income to 138.5% FPL + $500 and shows what plans would look like if denied Medicaid (e.g., immigration status).

Audit approach: When a scenario triggers Medicaid (income < 138% FPL):

1. Compute effectiveIncome = autoAdjustIncome(income, householdSize)
2. Query CMS with effectiveIncome (not original income)
3. Compare against our DB output (which uses the same auto-adjustment)
4. Match on cheapest plan + premium

This compares apples-to-apples: the plans CMS shows at the adjusted income vs the plans we show after auto-adjustment.

Auto-adjustment formula (matches src/lib/owned-plans.ts):

const FPL_BASE = 15650;        // 2025 FPL for 2026 enrollment
const FPL_PER_PERSON = 5500;
const fpl = FPL_BASE + FPL_PER_PERSON * (householdSize - 1);
if (income < fpl * 1.38) {
  return Math.round(fpl * 1.385) + 500;
}
return income;

CMS API rate limit handling

CMS doesn't publish exact rate limits. Conservative target:

• Sustained: 25 req/sec
• Burst: 30 req/sec
• Concurrency: 5 parallel requests
• Backoff: Exponential on 429/503 (1s → 2s → 4s → 8s)
• Timeout: 30s per request
• Max retries: 4

Implementation:

• scripts/audit/lib/rate-limiter.js — token bucket
• scripts/audit/lib/cms-client.js — retry wrapper with stats tracking

Resumability

All Tier 1, 2, and 3 audits write progress to /tmp/audit-progress-{tier}.json every 100 items. If a script dies, simply re-running it picks up where it left off. On successful completion, the progress file is deleted.

Output

Each audit run produces:

1. Stdout summary — pass/fail counts + per-state breakdown
2. JSON report at audit-tier-{N}-results.json (root of repo)
3. Documentation update in docs/validation/audit/tier-{N}-*.md
4. Atlas snapshot (manual safeguard) before running

Failure policy

Flag, don't block. Failures are documented and tracked in GitHub issues. Production rollouts only block if Tier 2 falls below 99% match rate.

Reproducibility

• Tier 4: identical output on the same DB state (deterministic)
• Tier 1, 2, 3: ≥99% identical on consecutive runs (small variance from CMS-side data updates is normal)
• Atlas snapshots allow full DB rollback if needed

Scripts reference

Script	Purpose
scripts/audit/tier-1-zip-county.js	Zip → county resolution audit (federal-30)
scripts/audit/tier-1-5-sbe-zip-county.js	Zip → county resolution audit (SBE-state)
scripts/audit/tier-2-zip-premium.js	Zip → premium audit
scripts/audit/tier-3-scenarios.js	Household scenario matrix
scripts/audit/tier-4-integrity.js	Internal data consistency
scripts/audit/lib/rate-limiter.js	Token bucket rate limiter
scripts/audit/lib/cms-client.js	CMS API wrapper
scripts/audit/lib/progress.js	Save/resume progress
scripts/audit/lib/db-helpers.js	Shared MongoDB helpers