Streamer App — Production Launch Checklist

Audience

Engineering, DevOps, anyone running the production cutover.

Target launch: May 15, 2026 Scope: publishing the streamer-facing application (app.verifluence.io + api.verifluence.io) on the production tier and flipping the marketplace from pre-launch onboarding to live.

---

Current state (May 7, 2026)

Component	Lives in	Public URL
Frontend (CF Pages: verifluence-app)	CF Pages	app.verifluence.io → currently proxies to api-stage.verifluence.io
Admin (CF Pages: verifluence-admin)	CF Pages	admin.verifluence.io → currently proxies to api-stage.verifluence.io
API	k8s stage namespace	api-stage.verifluence.io
Worker	k8s stage namespace	(internal)
Webhooks server	k8s stage namespace	wh.stage.verifluence.io
Postgres (stage)	k8s stage namespace	vf-pg-rw.stage (internal)
Postgres (prod)	k8s production namespace	vf-pg-rw.production (internal) — empty / no app releases pointing at it
ClickHouse (prod)	k8s production namespace	clickhouse-vf.production
Cloudflared tunnel	k8s production namespace	exposes *.verifluence.io

Implication: publishing to production = (a) provisioning api/worker/webhooks/migrator in the production namespace, (b) flipping CF Pages API_URL from stage → prod, and (c) flipping LD flags.

---

1. Code & schema readiness

• [ ] All main-branch CI is green (api typecheck, frontend typecheck, admin typecheck, storybook build)
• [ ] npx tsc --noEmit clean in api/, frontend/, admin/
• [ ] Image tag pinned for cutover — capture today's ghcr.io/verifluence/api:main-<sha>-<ts> so flux image-policy doesn't pull mid-launch
• [ ] Apply baseline + trailing migrations to production Postgres (handled automatically by the production migrator HelmRelease — see operations/clusters/prod/production/README.md). For manual bootstrap:
  • [ ] Create vfmigrator + vfapp roles per README step 3a
  • [ ] node dist/migrate.js — auto-detects fresh DB, applies api/schema/parts/00_baseline.sql, baselines 0001–0124, runs trailing migrations
  • [ ] Verify SELECT count(*) FROM _migrations ≥ count of ls api/migrations/*.sql (everything trailing the baseline applied)
• [ ] Apply ClickHouse schemas 01–05 to production CH:
  • 01_kick_streamer_probe.sql, 02_streamer_trust_score_history.sql, 03_events.sql, 04_impressions.sql, 05_sumsub_webhooks.sql
• [ ] Smoke-test rollback script for last 3 migrations on a snapshot

2. Production k8s releases (operations repo)

Create mirrors of the stage/releases/*.yaml under clusters/prod/production/releases/, with namespace: production and prod env values:

• [ ] production/releases/api.yaml
• [ ] production/releases/worker.yaml
• [ ] production/releases/webhooks.yaml
• [ ] production/releases/migrator.yaml
• [ ] Image policies & vf-charts HelmRepository accessible from production namespace
• [ ] HPA/PDB for api + worker + webhooks (replicaCount: 2 minimum)
• [ ] Add an HTTPRoute under clusters/prod/production/routes/http/ for:
  • [ ] api.verifluence.io → api.production.svc:80
  • [ ] wh.verifluence.io → webhooks.production.svc:80
• [ ] Verify Envoy Gateway listener prod-verifluence-https exists (TLS cert from cert-manager)

3. Production secrets & env

For every value below, rotate — do not reuse stage credentials.

• [ ] SESSION_SECRET — fresh 32-byte random
• [ ] ADMIN_SECRET — fresh 32-byte random; mirror to verifluence-admin Pages secret
• [ ] DATABASE_URL — points at vf-pg-rw.production:5432
• [ ] KICK_CLIENT_ID / KICK_CLIENT_SECRET — new prod app in Kick Developer Portal (see §6)
• [ ] SUMSUB_APP_TOKEN / SUMSUB_SECRET_KEY — production Sumsub tenant (prd: prefix, not sbx:)
• [ ] SUMSUB_WEBHOOK_SECRET — fresh, registered in Sumsub portal
• [ ] SUMSUB_LEVEL_NAME_ID / SUMSUB_LEVEL_NAME_POA — names of prod tenant levels (verify in Sumsub portal)
• [ ] RESEND_API_KEY — production key; verified domain updates.verifluence.io SPF/DKIM/DMARC green
• [ ] S3_* (R2) — prod bucket for streamer documents, with versioning + lifecycle rules
• [ ] PUSHER_* — prod Pusher app
• [ ] CLICKHOUSE_* — points at clickhouse-vf.production
• [ ] SENTRY_DSN — separate prod project; release tagging on
• [ ] SCRAPER_API_KEY / SCRAPEDO_TOKEN — keep stage values or rotate
• [ ] LD_SDK_KEY (if present server-side) — prod LD environment

4. CF Pages cutover

When prod backend is healthy on api.verifluence.io:

• [ ] Edit frontend/wrangler.toml: API_URL = "https://api.verifluence.io"
• [ ] Edit admin/wrangler.toml: API_URL = "https://api.verifluence.io"
• [ ] Set CF Pages secret on verifluence-app: CF_ACCESS_CLIENT_SECRET (new prod service token)
• [ ] Set CF Pages secret on verifluence-admin: CF_ACCESS_CLIENT_SECRET + ADMIN_SECRET
• [ ] CF Pages env vars (production deployment): VITE_LD_CLIENT_ID (prod), any other VITE_* build-time vars
• [ ] Trigger fresh deploys; verify SSR loaders return data on app.verifluence.io

5. Cloudflare Access

• [ ] Create prod CF Access service token (separate from stage); set client_id in wrangler.toml [vars], secret as Pages secret
• [ ] CF Tunnel: route api.verifluence.io and wh.verifluence.io through the production cloudflared deployment
• [ ] CF Access policy on api.verifluence.io — allow service-token bypass for the new prod token only; block stage's token

6. Kick OAuth (prod app)

• [ ] Create separate Kick app for production in Developer Portal
• [ ] Register redirect URLs:
  • https://app.verifluence.io/oauth/kick/callback
  • (no localhost entries on prod app)
• [ ] Capture client_id + client_secret; populate prod env (§3)
• [ ] Verify scopes match the constant in api/src/oauth.ts (KICK_SCOPES)

7. Sumsub (prod tenant)

• [ ] Create prod Sumsub tenant or new app under existing tenant — separate from sandbox
• [ ] Configure two levels with stable names matching SUMSUB_LEVEL_NAME_ID / SUMSUB_LEVEL_NAME_POA
• [ ] Webhook URL → https://wh.verifluence.io/wh/sumsub
• [ ] Generate webhook secret → set SUMSUB_WEBHOOK_SECRET in webhooks env
• [ ] Test fixture flow on staging-tenant against a synthetic streamer to confirm event handlers fire

8. DNS

• [ ] app.verifluence.io → CF Pages verifluence-app (custom domain)
• [ ] admin.verifluence.io → CF Pages verifluence-admin (custom domain)
• [ ] api.verifluence.io → CF Tunnel (production cloudflared)
• [ ] wh.verifluence.io → CF Tunnel (production cloudflared)
• [ ] Verify TLS certs auto-issued (cert-manager on the cluster side, CF on the edge side)

9. LaunchDarkly (prod environment)

Per prelaunch-flags.md, all four flags default false. For launch day:

• [ ] Confirm prod LD environment SDK key in VITE_LD_CLIENT_ID
• [ ] Verify all four flags exist on prod env: streamer-marketplace-enabled, streamer-trust-score-visible, operator-streamer-discovery-enabled, operator-deals-enabled
• [ ] Pre-stage targeting rules — beta cohort emails added but rule disabled
• [ ] Do not flip globally yet. Decide go/no-go after smoke tests in §12.

10. Observability

• [ ] Loki + Promtail / Alloy scraping production namespace pods
• [ ] ServiceMonitor selectors include production namespace
• [ ] Grafana folder for prod with these dashboards loaded:
  • verifluence-prelaunch-onboarding.json
  • api / worker / webhooks dashboards
• [ ] Critical alerts wired to paging channel:
  • api 5xx rate
  • webhook handler errors (Kick + Sumsub)
  • DB connection pool exhaustion
  • cnpg primary failover
  • CF tunnel down
• [ ] Sentry release tagging: deploy hook posts release to Sentry

11. Backups & disaster recovery

• [ ] cnpg Backup resource for prod Postgres; verify scheduled ScheduledBackup runs
• [ ] R2 bucket: object versioning on; lifecycle rule for old versions
• [ ] Test a restore — rehydrate prod backup into a scratch namespace and run SELECT count(*) on each table
• [ ] ClickHouse: confirm CHO replication or scheduled backup
• [ ] Document RPO/RTO targets in this repo

12. Smoke tests (against prod, before flag flip)

Run with a synthetic streamer + operator account; flags still off.

• [ ] GET /api/health returns 200
• [ ] GET /api/build-info returns the pinned image tag
• [ ] Kick OAuth signup end-to-end on app.verifluence.io (creates streamer row in prod DB)
• [ ] KYC: ID + PoA tracks both submit and webhook updates kyc_age_status + kyc_country_status
• [ ] Wallet linking persists
• [ ] /profile/dashboard shows the pre-launch checklist with correct per-track states
• [ ] Operator: invitation flow, account creation, campaign create + fund (escrow deposit reaches contract)
• [ ] Admin panel: KYC review queue lists the synthetic streamer; verdict round-trips
• [ ] Sentry receives a deliberate test error from each pod
• [ ] Grafana shows traffic on the prod api/worker/webhooks dashboards

13. Communication

• [ ] Pre-launch banner copy reflects launch date / final status
• [ ] Status page (if any) updated; maintenance window if needed
• [ ] Beta cohort emailed with timing + URL
• [ ] Internal Slack: launch-day runbook posted in #launch
• [ ] Support team briefed on prelaunch-flags.md scenarios table

14. Launch day flag flip (when §1–§13 are green)

In LD admin (per prelaunch-flags.md §"Recommended rollout sequence"):

• [ ] Stage 1 — flip operator-streamer-discovery-enabled. Wait 30 min, watch dashboards.
• [ ] Stage 2 — flip operator-deals-enabled. Wait. Confirm no operator-side errors in Sentry.
• [ ] Stage 3 — flip streamer-marketplace-enabled + streamer-trust-score-visible together.
• [ ] Verify pre-launch banner self-disappears in browser
• [ ] Watch verifluence-prelaunch-onboarding dashboard → onboarding cohort transitions to active marketplace usage
• [ ] First operator → streamer invitation lands and is acknowledged

15. Post-launch (T+24h)

• [ ] Sentry error volume comparable to stage baseline
• [ ] No spike in kyc_age_status='rejected' driven by misconfigured Sumsub levels
• [ ] No CF Tunnel connection errors
• [ ] Postgres connections stable; no slow-query alerts firing
• [ ] First end-to-end deal completed by a non-internal user

---

Rollback plan

If §12 smoke tests fail or §14 reveals a critical issue:

1. Flag rollback: flip the LD flag(s) you just enabled back to off. UI returns to pre-launch within 30s.
2. DNS rollback: point app.verifluence.io back at the previous-known-good frontend deploy via CF Pages's "Rollback" button (kept as last-known-good before launch).
3. API rollback: revert the image-policy commit in operations; flux re-pins the previous tag; rollout completes in ~3 min.
4. DB rollback: if a migration is the issue, the migration framework supports npm run migrate -- --rollback <version>. Test on stage first.

---

Owners

Section	Owner
§1 Code/schema	TBD
§2 K8s releases	TBD
§3 Secrets	TBD
§4–5 CF Pages + Access	TBD
§6 Kick OAuth	TBD
§7 Sumsub	TBD
§8 DNS	TBD
§9 LaunchDarkly	TBD
§10 Observability	TBD
§11 Backups	TBD
§12 Smoke tests	TBD
§13 Comms	TBD
§14 Flag flip	TBD

---

Related docs

• Milestones
• Pre-launch Feature Flags & Rollout Plan
• Feature flags — engineering reference
• Scaling for 100 streamers