Deployment Guide (Non-Production)¶

Single EC2 instance running the full Floh stack via Docker Compose with Caddy for automatic HTTPS.

TLS by Tier¶

Tier	TLS mode	How configured
Edge (`caddy`)	HTTPS on `:443`, certs from Let's Encrypt	`DEPLOY_DOMAIN`, `DEPLOY_PORTAL_DOMAIN`, `DEPLOY_FORM_BUILDER_DOMAIN`
Admin/portal/form-builder browser traffic	HTTPS at public domains	handled by Caddy reverse proxy
Internal container traffic (`server`, `web`, `portal-web`, `form-builder`)	HTTP on Docker network	default compose networking (`server:7070`, etc.)
Caddy → `portal-bff`	HTTPS on Docker network, verified	`PORTAL_BFF_TLS_CERT` / `PORTAL_BFF_TLS_KEY`, SAN must cover `DEPLOY_PORTAL_DOMAIN`
Caddy → `console-bff`	HTTPS on Docker network, verified	`CONSOLE_BFF_TLS_CERT` / `CONSOLE_BFF_TLS_KEY`, SAN must cover `DEPLOY_DOMAIN`
OIDC redirect / post-logout URLs	HTTPS public URLs	`public/ci.env` (`OIDC_REDIRECT_URI`, `FRONTEND_URL`, `PORTAL_FRONTEND_URL`)

Notes:

Do not set TLS_CERT_FILE / TLS_KEY_FILE for deploy compose. API TLS terminates at Caddy.
The two BFF legs are the exception to "internal traffic is HTTP". Each Authifi gateway serves HTTPS on the Docker network — portal-bff on :7071 from PORTAL_BFF_TLS_CERT / PORTAL_BFF_TLS_KEY, console-bff on :7074 from CONSOLE_BFF_TLS_CERT / CONSOLE_BFF_TLS_KEY — and Caddy verifies each (no tls_insecure_skip_verify) against the matching *_CA_CERT rather than the container's system roots. The two leaves are not interchangeable: each certificate's SAN must cover the tls_server_name Caddy sends for that upstream, which is DEPLOY_PORTAL_DOMAIN for the portal and DEPLOY_DOMAIN for the console. See Generating the portal BFF certificate and Generating the console BFF certificate.
FLOH_INTERNAL_URL in public config should stay http://server:7070 for Docker deploy.
The Floh server runs a periodic TLS health check (every 15 min). If a domain's cert is missing or invalid, the server reloads Caddy via its admin API to trigger a fresh ACME challenge. Set CADDY_ADMIN_URL=http://caddy:2019 in ci.env to enable (already configured).

Architecture¶

Internet
  │
  ├─ :443 ──► Caddy (TLS termination, Let's Encrypt)
  │             ├─ floh-dev.example.com/api/*      ──► server:7070
  │             ├─ floh-dev.example.com/scim/v2/*  ──► server:7070 (inbound SCIM)
  │             ├─ floh-dev.example.com/*           ──► web:8080 (nginx)
  │             ├─ portal.example.com/bff/*         ──► portal-bff:7071 (HTTPS, OIDC login/logout/step-up)
  │             ├─ portal.example.com/api/*         ──► portal-bff:7071 (HTTPS, proxied to server)
  │             ├─ portal.example.com/*               ──► portal-web:8080 (nginx)
  │             └─ forms.example.com/*                ──► form-builder:8080 (nginx)
  │                  (CSP frame-ancestors pinned to floh-dev.example.com)
  │
  └─ :8025 ──► MailHog UI (restrict to your IP)

Inbound SCIM (Entra ID, Okta, etc.) uses the admin domain, not a separate API hostname. IdPs call https://<DEPLOY_DOMAIN>/scim/v2 (for example https://floh.authilize.com/scim/v2). Caddy must proxy /scim/v2/* to server:7070 before the catch-all rule that serves the admin SPA; otherwise provisioning clients receive the Angular index.html instead of SCIM JSON.

Verify after deploy:

curl -sS "https://<DEPLOY_DOMAIN>/scim/v2/ServiceProviderConfig" \
  -H "Authorization: Bearer <scim-token>"

Expect JSON with "patch":{"supported":true}. HTML containing <app-root> means the /scim/v2 route is missing or the deploy has not picked up an updated docker/Caddyfile. Token setup is documented in Entra setup for Floh and Okta setup for Floh.

The form-builder site is the standalone @floh/form-builder-app SPA, iframed by the Floh admin UI to power the Workflow Designer's Visual editor. It MUST live on an origin distinct from DEPLOY_DOMAIN — the host class rejects same-origin embeds at runtime. The deploy workflow bakes https://${DEPLOY_FORM_BUILDER_DOMAIN}/ into the web image as environment.formBuilderEmbedUrl so the Visual toggle is enabled out of the box.

Prerequisites¶

AWS account
A domain name with DNS you can control
GitHub repo: github.com/Authifi/floh

First-time bring-up checklist¶

The numbered steps below cover host setup. This checklist is the dependency order for a domain that has never been deployed, including the identity and certificate work that the host steps do not mention. Each item links to its section.

Skipping an item does not fail the deploy at that point — it fails later, often somewhere unrelated-looking. The right-hand column names what you get if you skip it.

Provisional: verified against the code, not yet run end to end

This ordering was derived by tracing the deploy workflow and the Authifi setup scripts, and each stated failure mode is taken from the actual error path rather than inferred. It has not been executed against a fresh tenant from step 1 to step 13.

Two blockers were found this way rather than by running it — the hand-created transitional resource server in step 8 and the --identity-provider requirement in step 9 — so treat a first real bring-up as the thing that validates this list, and please correct it where reality differs. Run step 9 with --dry-run first; it is the step with the most ways to fail.

#	Do this	Skip it and you get
1	Provision the host: Steps 1–6	—
2	Point DNS at the Elastic IP for all three domains (Step 4)	Caddy cannot complete ACME; TLS verification fails
3	Set the deploy variables (`DEPLOY_DOMAIN`, `DEPLOY_PORTAL_DOMAIN`, `AUTHIFI_ADMIN_TARGET`, …) (Variables)	Preflight fails naming the variable
4	Generate the portal BFF certificate and set its three secrets (recipe)	Every `/bff/*` request 502s
5	Generate the console BFF certificate — separate leaf, SAN covers `DEPLOY_DOMAIN` (recipe)	`console-bff` never becomes healthy; the whole console and API are down
6	Set the remaining secrets (Secrets) — all except `OIDC_CLIENT_SECRET` and `PORTAL_OIDC_CLIENT_SECRET`, which do not exist yet	Preflight fails naming the secret
7	Decide the channel audience identifiers (`FLOH_CONSOLE_AUDIENCE`, `FLOH_PORTAL_AUDIENCE`, `FLOH_MCP_AUDIENCE`) and set them in the environment you run steps 8–9 from	Steps 8–9 create and bind the default resource servers; the deployed stack then requests identifiers that do not exist. Both steps read all three
8	Create the resource servers: `pnpm run sync-authifi-rbac -- --apply` for the three channel ones, plus a resource server whose identifier is `FLOH_RESOURCE_ID`, created by hand (why)	Step 9 refuses to run: `No Authifi resource server has identifier "…"`
9	Register the OIDC clients, passing the deployed origins and an identity provider explicitly (details)	Without the origins, login fails with `invalid_redirect_uri`; without `--identity-provider`, creating a client aborts outright
10	Capture the minted client secrets into `OIDC_CLIENT_SECRET` and `PORTAL_OIDC_CLIENT_SECRET` — the two deferred from step 6	Login fails with `invalid_client`; the value cannot be re-read later
11	Merge `config/public/ci.env` changes (URLs, origins, and the same audience identifiers chosen in step 7)	Wrong redirect URLs or CORS rejections; an audience mismatch with step 8 fails login
12	Run the deploy workflow (Deploying)	—
13	Smoke-test login, not just the home page (below)	A broken auth path reports as a green deploy

Four ordering constraints are easy to get wrong:

Certificates before the first deploy. The health gate blocks on console-bff and portal-bff, so a missing or malformed leaf fails the whole deploy rather than degrading one route.
Resource servers before client registration. On a new tenant, setup-authifi-oidc-clients resolves all three channel resource servers before its first write and refuses to proceed if any is missing, so the RBAC sync has to come first.
Audience identifiers before either Authifi command. Both commands read the identifiers from the environment you invoke them in, and neither reads config/public/ci.env. sync-authifi-rbac reads all three (FLOH_CONSOLE_AUDIENCE, FLOH_PORTAL_AUDIENCE, FLOH_MCP_AUDIENCE); setup-authifi-oidc-clients reads the same three, because it registers floh-mcp-client alongside the browser clients, plus FLOH_RESOURCE_ID for the transitional resource server. So if the deployment needs non-default identifiers, choosing them after those commands leaves the tenant holding resource servers under the old names while the deployed API and BFFs request the new ones. Set the same values in both places.

The transitional resource server¶

setup-authifi-oidc-clients binds each client to its channel resource server and to a shared transitional one whose identifier is FLOH_RESOURCE_ID. It resolves four in total — console, portal, MCP, and transitional — before its first write, and fails closed if any is absent.

sync-authifi-rbac only creates the three channel resource servers, so on a new tenant the transitional one does not exist and step 9 aborts with:

No Authifi resource server has identifier "http://floh.api" for the transitional channel.
Run pnpm run sync-authifi-rbac -- --apply first, or correct FLOH_RESOURCE_ID.

That message is misleading for this one case: rerunning the RBAC sync will never create it. Create a resource server by hand in the Authifi admin UI with its identifier set to your FLOH_RESOURCE_ID value, then rerun step 9. The binding is a leftover from the pre-channel-audience model and is tracked for removal under LSA-9828; the API does not accept this identifier as a token aud. Automating or removing this manual step is tracked in #1188.

Client registration before capturing secrets. Authifi cannot re-read an existing client secret. If you register clients and do not capture the minted values, the only recovery is rotation.

Step 9 has a trap worth spelling out. Run bare, setup-authifi-oidc-clients takes its origin lists from FRONTEND_URL and PORTAL_FRONTEND_URL in your local environment, which on a fresh checkout from .env.example are https://localhost:7072 and https://localhost:7073. The deployed values live in config/public/ci.env, which this checklist does not touch until step 11 and which the script does not read in any case. So the bare command registers localhost callbacks, the first deployed login fails with invalid_redirect_uri, and — because the run reconciles rather than appends — any deployed origins already registered are removed. Always pass --console-origin and --portal-origin explicitly and preview with --dry-run first; the full command, including the localhost origins to keep when the tenant is shared with local development, is in Authifi client registration.

Step 9 has a second trap on a genuinely new tenant: creating the console client requires an identity provider, and the script refuses to guess one. With none supplied it stops before any write, including under --dry-run:

Creating floh-client and floh-mcp-client requires --identity-provider (or AUTHIFI_IDENTITY_PROVIDERS).
Duplicate of an existing console client copies providers automatically.

The message names whichever clients the run would create. floh-mcp-client appears because its initial refresh token comes from an authorization-code flow, so it needs a provider too; the vault client does not (it authenticates with private_key_jwt).

Pass --identity-provider <name> (repeatable) or set AUTHIFI_IDENTITY_PROVIDERS=nih,google. .env.example leaves the variable commented out, so a fresh checkout supplies nothing. The value is the Authifi identity-provider name your tenant federates to — ask whoever owns the tenant rather than guessing, since a wrong provider produces a client nobody can log in through. This constraint only applies when the client does not yet exist; reconciling an existing client copies its providers.

For where each value lives and what overrides what, see the Configuration Map.

The deploy's own TLS check probes https://<domain>/, which Caddy serves from the SPA containers. It therefore passes while the entire authentication path is broken. After every deploy to a new domain, confirm:

https://<DEPLOY_DOMAIN>/ loads the console.
Clicking sign-in reaches the IdP (not an invalid_redirect_uri page).
Completing login returns you to the console authenticated — this is the step that catches a stale OIDC_CLIENT_SECRET.
https://<DEPLOY_PORTAL_DOMAIN>/bff/login redirects rather than 502s.
Complete a portal login and land on an authenticated portal page. Step 4 is not sufficient on its own: the portal client secret is not used until the authorization-code exchange on the callback, so /bff/login redirects perfectly well with a stale PORTAL_OIDC_CLIENT_SECRET and every login then fails with invalid_client. This is the portal equivalent of step 3.

Any failure here maps to an entry in the Failure reference.

Step 1: Launch EC2 Instance¶

Instance type: t3.medium (2 vCPU, 4 GB RAM)
AMI: Ubuntu 24.04 LTS
Storage: 30 GB gp3 EBS
Key pair: Create or select one (save the .pem file)

Step 2: Security Group¶

Allow inbound:

Port	Protocol	Source	Purpose
22	TCP	Your IP	SSH
80	TCP	Anywhere	HTTP (ACME challenges + redirect)
443	TCP	Anywhere	HTTPS
8025	TCP	Your IP	MailHog UI (optional)

Step 3: Elastic IP¶

Allocate an Elastic IP and associate it with your instance. This gives a stable address that survives reboots.

Step 4: DNS¶

Create A records pointing to the Elastic IP:

floh-dev.example.com → <ELASTIC_IP>
portal.floh-dev.example.com → <ELASTIC_IP>
forms.floh-dev.example.com → <ELASTIC_IP> (form-builder SPA — must be a distinct origin from the admin domain)

Caddy will auto-provision Let's Encrypt certificates once DNS resolves.

Step 5: Install Docker and Dependencies¶

ssh -i your-key.pem ubuntu@<ELASTIC_IP>

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker ubuntu
newgrp docker
sudo apt-get install -y jq

docker --version
docker compose version
jq --version

Step 6: Register the Self-Hosted Runner¶

The deploy workflow runs on a self-hosted runner with the deploy label. Register the EC2 instance as a GitHub Actions runner:

Go to Settings → Actions → Runners
Click New self-hosted runner and follow the install steps for Linux x64
Configure the runner with the label deploy
Install and start the runner as a service so it survives reboots:

sudo ./svc.sh install
sudo ./svc.sh start

The workflow authenticates to GHCR automatically via docker/login-action — no manual PAT login required.

Step 7: Configure the GitHub "dev" Environment¶

The deploy workflow pulls all secrets and variables from the GitHub dev environment. Non-sensitive runtime config lives in checked-in public config files (config/public/base.env and config/public/ci.env) and is copied to ~/floh/public/ during each deploy, alongside the portal BFF's docker/bff/portal.json in ~/floh/bff/.

In Settings → Environments, create an environment named dev and add:

Secrets¶

Secret	Value
`DB_PASSWORD`	Strong database password
`REDIS_PASSWORD`	Redis password (leave empty if no auth)
`SMTP_USER`	SMTP username (empty for MailHog)
`SMTP_PASS`	SMTP password (empty for MailHog)
`JWT_SECRET`	64-char hex key (`pnpm run generate-key`)
`SESSION_SECRET`	64-char hex key (`pnpm run generate-key`)
`SESSION_ENCRYPTION_KEY`	64-char hex key (`pnpm run generate-key`)
`AUDIT_CHECKPOINT_KEY`	64-char hex key (`pnpm run generate-key`)
`OIDC_CLIENT_SECRET`	Console (`floh-client`) OIDC client secret — written to `~/floh/env/console.env`
`PORTAL_OIDC_CLIENT_SECRET`	Portal (`floh-portal-client`) OIDC client secret — written to `~/floh/env/portal.env`
`PORTAL_BFF_COOKIE_ENCRYPTION_SECRET`	Portal BFF session-cookie encryption key — any non-empty string; use `pnpm run generate-key`. Written to `~/floh/env/portal.env`
`PORTAL_BFF_TLS_CERT`	PEM certificate the portal BFF serves on `:7071` — written to `~/floh/env/portal.env`
`PORTAL_BFF_TLS_KEY`	PEM private key for the above — written to `~/floh/env/portal.env`
`PORTAL_BFF_CA_CERT`	PEM CA certificate that issued the pair above — written to `~/floh/bff-ca.crt`
`CONSOLE_BFF_COOKIE_ENCRYPTION_SECRET`	Console BFF session-cookie encryption key — written to `~/floh/env/console.env`
`CONSOLE_BFF_TLS_CERT`	PEM certificate the console BFF serves on `:7074` — written to `~/floh/env/console.env`
`CONSOLE_BFF_TLS_KEY`	PEM private key for the above — written to `~/floh/env/console.env`
`CONSOLE_BFF_CA_CERT`	PEM CA certificate that issued the pair above — written to `~/floh/console-bff-ca.crt`
`CONNECTOR_ENCRYPTION_KEY`	64-char hex key (`pnpm run generate-key`)
`CONNECTOR_ENCRYPTION_KEY_PREVIOUS`	Previous key during rotation (empty when not rotating)
`AUDIT_CHECKPOINT_KEY_PREVIOUS`	Previous key during rotation (empty when not rotating)
`AUTHIFI_VAULT_PRIVATE_KEY`	PEM private key for the vault client — only required when `SECRETS_BACKEND=authifi` (see Authifi tenant-secrets vault)

The deploy fails closed if any of DB_PASSWORD, JWT_SECRET, SESSION_SECRET, SESSION_ENCRYPTION_KEY, AUDIT_CHECKPOINT_KEY, CONNECTOR_ENCRYPTION_KEY, OIDC_CLIENT_SECRET, PORTAL_OIDC_CLIENT_SECRET, PORTAL_BFF_COOKIE_ENCRYPTION_SECRET, PORTAL_BFF_TLS_CERT, PORTAL_BFF_TLS_KEY, PORTAL_BFF_CA_CERT, CONSOLE_BFF_COOKIE_ENCRYPTION_SECRET, CONSOLE_BFF_TLS_CERT, CONSOLE_BFF_TLS_KEY, or CONSOLE_BFF_CA_CERT is unset. The BFF TLS PEMs are multiline and are written to ~/floh/env/console.env and ~/floh/env/portal.env single-quoted so the newlines survive; PORTAL_BFF_CA_CERT and CONSOLE_BFF_CA_CERT are public, so they are written to ~/floh/bff-ca.crt and ~/floh/console-bff-ca.crt and bind-mounted instead — into Caddy, which verifies each gateway's certificate, and into the matching BFF, whose healthcheck probes /health over the same TLS hop and would otherwise fail to verify the chain against Node's bundled roots.

Rotating an existing secret¶

Most of these can be replaced freely. Rotating JWT_SECRET, SESSION_SECRET, SESSION_ENCRYPTION_KEY, or PORTAL_BFF_COOKIE_ENCRYPTION_SECRET costs nothing worse than logging everyone out, and the three portal BFF PEMs can be reissued as a set at any time because nothing outside this deployment trusts them. Three are not free, and none of the three fails at deploy time:

DB_PASSWORD cannot be rotated here alone. Compose passes it as POSTGRES_PASSWORD, which Postgres honors only when initializing an empty data directory. The volume persists across deploys, so changing the secret changes what the server sends without changing what the database expects, and the next deploy fails to authenticate. See Rotating DB_PASSWORD.
CONNECTOR_ENCRYPTION_KEY is one-way. Replacing it leaves stored connector credentials undecryptable. Carry the outgoing value into CONNECTOR_ENCRYPTION_KEY_PREVIOUS and follow the connector key rotation runbook, which re-encrypts every stored secret under the new key and ends by clearing _PREVIOUS.
AUDIT_CHECKPOINT_KEY is one-way and, unlike the connector key, has no re-signing step. Re-signing historical checkpoints would defeat the tamper-evidence they exist to provide, so there is no equivalent of rotate-keys here. New checkpoints are signed with the current key, and GET /api/audit-logs/verify-integrity verifies each one against the current key plus AUDIT_CHECKPOINT_KEY_PREVIOUS. That means _PREVIOUS is not a transitional value to clear afterwards — it must stay populated for as long as checkpoints signed with the old key are still in scope for verification, and only one generation back is supported.

Both _PREVIOUS values must be exactly 64 hex characters. A value of any other length is silently ignored rather than rejected, so a truncated or whitespace-padded paste presents as connector secrets or historical checkpoints failing with nothing in the logs pointing at the key.

PORTAL_OIDC_CLIENT_SECRET and OIDC_CLIENT_SECRET have counterparts in Authifi, which cannot re-read an existing client secret. A lost value can only be replaced, not recovered: pnpm run setup-authifi-oidc-clients -- --rotate-secrets mints a new one and captures it.

When SECRETS_BACKEND=authifi, DB_PASSWORD is the one secret that lives in two stores at once — see the dual-source hazard — because the server reads it from the vault while Compose interpolates the GitHub copy into POSTGRES_PASSWORD. Even updating both is not sufficient; the live Postgres role has to be altered too, which the sequence immediately below covers.

The OIDC client secrets behave differently, and the vault copy does no work today. Neither OIDC_CLIENT_SECRET nor PORTAL_OIDC_CLIENT_SECRET is read by the API server — PORTAL_OIDC_CLIENT_SECRET appears nowhere in packages/server/src, and only the non-secret PORTAL_OIDC_CLIENT_ID is looked up. The consumer is the BFF, which Compose feeds from ~/floh/env/portal.env as AUTH_CLIENT_SECRET, and the deploy writes that file from the GitHub secret on both backends. Rotating the GitHub secret is therefore what actually takes effect.

They are still listed as required tenant secrets below, because both runbooks are pinned to SECRET_KEYS by test/unit/architecture/deploy-vault-bootstrap.test.ts and SECRET_KEYS is the set the deployment manages, not the narrower set the server resolves. Populating them keeps the vault self-consistent and costs a redundant copy of a live credential; whether the list should be split is #1192.

Rotating `DB_PASSWORD`¶

POSTGRES_PASSWORD is only an initialization convenience; the password is ordinary role state once the cluster exists, so change it in place:

Set the new value as the DB_PASSWORD secret in the dev environment. This has no effect until the next deploy.
If the stack runs SECRETS_BACKEND=authifi, set it in the vault too, as the secret named FLOH_DB_PASSWORD (the provider strips the AUTHIFI_VAULT_SECRET_PREFIX, default FLOH_). The vault is where the server actually reads this value on that backend, so a GitHub-only update leaves the restarted server sending the old password to a role that has already changed — the same lockout this runbook exists to prevent. Keep the two in sync: the GitHub secret still feeds Compose's POSTGRES_PASSWORD interpolation.
Apply it to the running database:

cd ~/floh
docker compose -p floh -f docker-compose.deploy.yml \
  --env-file .env --env-file env/console.env --env-file env/portal.env \
  exec postgres psql -U floh -d floh -c '\password floh'

\password prompts for the value and sends it as a parameter, so the credential never reaches your shell history or the psql process arguments, and it needs no hand-quoting — an apostrophe in the password would otherwise terminate the SQL literal and reject a perfectly valid credential. Avoid -c "ALTER ROLE floh PASSWORD '…';" for exactly those two reasons.

Re-run the deploy.

The role, database, and user name are all floh because DB_USER and DB_NAME are unset in this deployment and fall through to the compose defaults; if either is ever set, substitute accordingly.

Connections already open are authenticated and survive, so the only exposure is new connections between step 3 and the server restarting.

Do not rotate this by rebuilding the volume. Getting POSTGRES_PASSWORD to apply again means deleting pgdata, which turns a one-statement change into a dump, a destroy, and a restore. Worse, the obvious backup command hides a trap: pg_dumpall includes globals, and the globals dump carries the role's password hash as ALTER ROLE floh WITH PASSWORD 'SCRAM-SHA-256$...'. Restoring it re-applies the old password and silently undoes the rotation, presenting as the same authentication failure that prompted the rotation in the first place.

Generating the portal BFF certificate¶

Caddy proxies to https://portal-bff:7071 and verifies the certificate, so the BFF needs a certificate for <DEPLOY_PORTAL_DOMAIN> that chains to an anchor Caddy trusts. Generate a small private CA once, issue a long-lived leaf from it, and keep ca.key offline — only ca.crt is deployed:

DOMAIN=myfloh.authilize.com

openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
  -keyout ca.key -out ca.crt \
  -subj "/CN=Floh Portal BFF Internal CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
  -addext "keyUsage=critical,keyCertSign,cRLSign"

openssl req -newkey rsa:2048 -nodes -keyout bff.key -out bff.csr \
  -subj "/CN=${DOMAIN}"

openssl x509 -req -in bff.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out bff.crt -days 3650 -sha256 \
  -extfile <(printf "subjectAltName=DNS:%s\nextendedKeyUsage=serverAuth\nbasicConstraints=critical,CA:FALSE" "${DOMAIN}")

gh secret set PORTAL_BFF_TLS_CERT --repo Authifi/floh --env dev < bff.crt
gh secret set PORTAL_BFF_TLS_KEY  --repo Authifi/floh --env dev < bff.key
gh secret set PORTAL_BFF_CA_CERT  --repo Authifi/floh --env dev < ca.crt

The SAN must match the tls_server_name Caddy sends, which is DEPLOY_PORTAL_DOMAIN. A mismatch, or a bare self-signed certificate with no matching PORTAL_BFF_CA_CERT, fails the handshake and makes every /bff/* request return 502. The deploy's TLS verification step will not catch this, because it probes the portal root — which Caddy serves from portal-web, not the BFF. Test https://<DEPLOY_PORTAL_DOMAIN>/bff/login explicitly after a deploy.

These certificates are deliberately long-lived and are not renewed by ACME. Re-run the leaf steps before the 10-year expiry, or whenever DEPLOY_PORTAL_DOMAIN changes.

Generating the console BFF certificate¶

The console BFF needs the same treatment as the portal BFF, but its own leaf with a different SAN. Caddy sends tls_server_name {$DEPLOY_DOMAIN} on this hop (docker/Caddyfile), so the console leaf must cover DEPLOY_DOMAIN — reusing the portal leaf fails the handshake even though both certificates are valid and share a CA.

The blast radius is wider here than on the portal. Caddy routes /api/*, /bff/*, and /authifi/* through console-bff, so a bad certificate takes down the entire admin console and its API, not just login.

You can reuse the CA from the portal section (keep ca.key offline) and issue a second leaf, which is what the dev tier does:

DOMAIN=floh.authilize.com  # DEPLOY_DOMAIN, *not* DEPLOY_PORTAL_DOMAIN

openssl req -newkey rsa:2048 -nodes -keyout console.key -out console.csr \
  -subj "/CN=${DOMAIN}"

openssl x509 -req -in console.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out console.crt -days 3650 -sha256 \
  -extfile <(printf "subjectAltName=DNS:%s\nextendedKeyUsage=serverAuth\nbasicConstraints=critical,CA:FALSE" "${DOMAIN}")

gh secret set CONSOLE_BFF_TLS_CERT --repo Authifi/floh --env dev < console.crt
gh secret set CONSOLE_BFF_TLS_KEY  --repo Authifi/floh --env dev < console.key
gh secret set CONSOLE_BFF_CA_CERT  --repo Authifi/floh --env dev < ca.crt

CONSOLE_BFF_CA_CERT is a separate secret from PORTAL_BFF_CA_CERT even when both hold the same CA — they are mounted to different files (~/floh/console-bff-ca.crt and ~/floh/bff-ca.crt). Set both.

Pipe the file with <. It is the one capture path that cannot reshape the value. Every malformed-PEM incident on this tier has come from copying the text through something else first — a browser paste into the GitHub secrets UI, an editor that escaped the newlines, a value round-tripped through JSON — leaving either literal \n escapes or a base64 blob where the PEM should be. The PEM is then unparseable, and Node fails with ERR_OSSL_PEM_NO_START_LINE inside configSecureContext at BFF startup. (--body "$(cat console.crt)" preserves interior newlines correctly, so it is not itself a corruption source; stdin redirection is still preferred because it keeps the secret out of your shell history and argument list.) The deploy's preflight now catches this shape before writing any host file; see Deploy preflight for TLS secrets.

As with the portal, the deploy's "Verify TLS for all domains" step does not exercise this leg — it probes https://<DEPLOY_DOMAIN>/, which Caddy serves from web. What does catch a broken console BFF is the health gate in "Start all services", which fails the deploy after ~120s with console-bff did not become healthy after startup. Confirm explicitly after a deploy by loading the console and completing a login.

Deploy preflight for TLS secrets¶

Before writing any environment file, the deploy runs scripts/validate-bff-tls-secrets.mjs against all seven PEM secrets. It fails the job in seconds — naming the variable, the problem, and the fix — for:

PEM that is unparseable because newlines became literal \n escapes, the value was base64-encoded, or the value arrived wrapped in quote characters
a private key that does not pair with its certificate
a leaf that was not issued by the CA in its *_CA_CERT secret
a leaf whose SAN does not cover the domain Caddy requests (the reused-portal-leaf mistake)
a certificate that is expired or not yet valid

It never prints secret values. Run the same check locally against files before setting secrets:

DEPLOY_DOMAIN=floh.authilize.com DEPLOY_PORTAL_DOMAIN=myfloh.authilize.com \
CONSOLE_BFF_TLS_CERT="$(cat console.crt)" \
CONSOLE_BFF_TLS_KEY="$(cat console.key)" \
CONSOLE_BFF_CA_CERT="$(cat ca.crt)" \
PORTAL_BFF_TLS_CERT="$(cat bff.crt)" \
PORTAL_BFF_TLS_KEY="$(cat bff.key)" \
PORTAL_BFF_CA_CERT="$(cat ca.crt)" \
node scripts/validate-bff-tls-secrets.mjs && echo "TLS secrets OK"

Variables¶

Variable	Value
`DEPLOY_DOMAIN`	`floh.authilize.com`
`DEPLOY_PORTAL_DOMAIN`	`myfloh.authilize.com`
`ACME_EMAIL`	Email for Let's Encrypt expiry warnings (optional)
`SECRETS_BACKEND`	`env` (default when unset) or `authifi`. See Authifi tenant-secrets vault.
`AUTHIFI_ADMIN_TARGET`	Fixed tenant-scoped `<authifi-base>/auth/admin/tenants/<positive-tenant-id>` target for the console BFF.

The workflow writes three host files on every deploy: ~/floh/.env (API + infra), ~/floh/env/console.env, and ~/floh/env/portal.env. Manual edits to those files are overwritten. Rollback is this PR's three-file layout together with matching compose --env-file flags — a leftover single .env from before LSA-9829 is not a compatible host layout.

For the console BFF, NODE_CONFIG owns the bffProxy array and the named bff.tokenSlots. Every slot must include scopes as a JSON string array — BFF 3.4.0 rejects a missing or non-array scopes field at startup. The flohApi primary slot uses the same login scopes as auth.authorizationParams.scope in docker/bff/console.json (openid, profile, email, offline_access). Do not reintroduce standalone BFF_PROXY_TARGET or BFF_PROXY_PATH entries for the console service: node-config's custom-environment-variable precedence would override the array shape with scalar values and silently corrupt the proxy contract. AUTHIFI_ADMIN_TARGET is the remaining GitHub var for the tenant-pinned Authifi admin HTTP API. The admin token slot resource is OIDC_ISSUER (Authifi API RSID). jwksUri is OIDC_JWKS_URI from ci.env (the issuer's discovery jwks_uri). Deploy fetches {OIDC_ISSUER}/.well-known/openid-configuration and fails if jwks_uri disagrees. The workflow writes OIDC_JWKS_URI into ~/floh/.env and AUTHIFI_ADMIN_TARGET into ~/floh/env/console.env.

Repository-level variables¶

In addition to the dev environment vars above, the deploy workflow reads one repository-level variable (Settings → Variables → Actions, not the dev environment). Repository-level scope keeps the matrix build job out of the dev environment so it does not write dev deployment events on every push or inherit any future dev protection rules.

Variable	Value
`DEPLOY_FORM_BUILDER_DOMAIN`	`forms.authilize.com` — required. Must resolve to a distinct origin from `DEPLOY_DOMAIN`. Baked into the web image as `environment.formBuilderEmbedUrl`.

config/public/ci.env contains non-secret URL and OIDC settings. Note that it outranks ~/floh/.env for every non-secret, so changing a non-secret in the GitHub environment usually has no effect — see the Configuration Map.

FRONTEND_URL, PORTAL_FRONTEND_URL, ALLOWED_ORIGINS, ALLOWED_PORTAL_ORIGINS
OIDC_ISSUER, OIDC_CLIENT_ID, OIDC_REDIRECT_URI, PORTAL_OIDC_CLIENT_ID
OIDC_AUDIENCE — Authifi catalog API resource id (FLOH_RESOURCE_ID locally). Operator setup still uses it. The API does not accept it as JWT aud.
FLOH_CONSOLE_AUDIENCE, FLOH_PORTAL_AUDIENCE, FLOH_MCP_AUDIENCE — channel catalog RSIDs. Console BFF AUTH_RESOURCE is FLOH_CONSOLE_AUDIENCE (also written to env/console.env); portal BFF uses FLOH_PORTAL_AUDIENCE (and env/portal.env). The API verifies JWT aud against these identifiers. The deploy workflow copies them from ci.env into the host env files. They remain pairwise distinct from each other and from OIDC_AUDIENCE.
OIDC_JWKS_URI — JWKS from the issuer's discovery document (jwks_uri). Used for BFF step-up and the console Authifi admin token slot. Optional when OIDC_ISSUER is set: Compose and the host-Node launchers derive {OIDC_ISSUER}/.well-known/jwks.json. PORTAL_OIDC_JWKS_URI remains a one-release alias. Deploy fetches discovery and fails if the committed value disagrees.
OIDC_CLAIM_UPSTREAM_ISSUER, OIDC_CLAIM_UPSTREAM_ID — claim names the IdP uses for upstream (federated) identity. For Authifi these are identityIssuer and email.
CADDY_ADMIN_URL — enables automated TLS cert recovery via Caddy's admin API

Why some ci.env keys are also copied into ~/floh/.env: the server container mounts config/public and reads ci.env directly, but the BFF services have no env_file — they interpolate from host --env-file flags (.env, env/console.env, env/portal.env) plus their compose environment: block. The workflow therefore resolves OIDC_ISSUER, PORTAL_OIDC_CLIENT_ID, OIDC_AUDIENCE, FLOH_CONSOLE_AUDIENCE, FLOH_PORTAL_AUDIENCE, FLOH_MCP_AUDIENCE, and OIDC_JWKS_URI out of ci.env and writes them into ~/floh/.env as well. Adding a new ${VAR} reference to a BFF service without extending that list (or the matching process file) yields an empty value, because Compose substitutes unset variables silently. packages/portal-bff/test/config-invariants.test.ts and console-config-invariants.test.ts fail the build when the lists drift.

Config precedence: When public config files are loaded (APP_ENV is set and PUBLIC_CONFIG_DIR points to the config directory), the server reads non-secret settings only from base.env / ci.env. Values in process.env (including those from Docker Compose env_file) are not consulted for non-secret keys unless ALLOW_LEGACY_ENV_NON_SECRET=true is set, which re-enables process.env as a fallback for migration or back-compat. See readNonSecret in packages/server/src/config/index.ts for the implementation. In normal operation, all non-secret OIDC and runtime settings should go in the public config files.

TRUST_PROXY (in config/public/base.env) should only be enabled when the proxy layer strips/overwrites untrusted X-Forwarded-* headers. If misconfigured, clients can spoof source IPs and impact rate-limiting and audit/access-log attribution.

GitHub Actions Node Version¶

CI and release automation resolve Node from the repository root .nvmrc via actions/setup-node node-version-file, so local nvm use, PR CI, and release tagging all use the same toolchain selection. The package compatibility floor remains node >=24.0.0; do not introduce Node 26-only package APIs just because GitHub Actions currently runs Node 26.

Rollback. If Node 26 exposes a tooling regression, restore .nvmrc to the previous Node 24 value and rerun CI or release. No workflow YAML edit is needed for that rollback path because the workflows already read .nvmrc.

Authifi client registration¶

Before client registration, reconcile the Authifi authorization catalog for all three channels. The deploy workflow never runs this command automatically:

pnpm run sync-authifi-rbac -- --dry-run
pnpm run sync-authifi-rbac -- --apply
pnpm run sync-authifi-rbac -- --dry-run

The command rebuilds @floh/shared before planning so a leftover gitignored packages/shared/dist cannot apply an older catalog. It loads repo .env when it exists, with direct shell values taking precedence. Paste the admin JWT into gitignored .authifi-admin-token (not .env). The CLI derives AUTHIFI_BASE_URL, numeric AUTHIFI_TENANT_ID, and AUTHIFI_ADMIN_RESOURCE from that token when they are unset. Set all three FLOH_*_AUDIENCE identifiers in .env or the shell. OIDC client setup uses the same token file (SERVICES_AUTH_URL / AUTH_CLI_ACCESS_TOKEN remain deprecated aliases of the derived base URL and in-memory token copy).

--apply additionally requires the token file to hold an MFA / AAL2 session. Authifi gates group writes behind step-up authentication and answers a password-only token with 401 and WWW-Authenticate: … error="insufficient_user_authentication", acr_values="mod-mf". The command inspects the token before contacting Authifi and refuses to apply when the token carries no MFA, when its auth_time is older than the max_age Authifi enforces on those writes, or when its exp / step-up window has fewer than 30 seconds left before mutation, so a partial apply is not started. Re-authenticate with MFA, paste the fresh token, and run the apply promptly; resource-server, permission, and access-role writes are unaffected, so an interrupted run resumes on the next apply.

STEP_UP_ACR_ALIASES controls which JWT acr values satisfy that acr_values=mod-mf challenge. The sync parser accepts only a JSON object whose keys are safe non-empty challenge strings and whose values are non-empty arrays of safe ACR strings. Invalid JSON, empty keys, empty arrays, unsafe characters, or prototype-pollution keys fail closed before client construction or network I/O.

That preflight is intentionally limited to readable JWTs. If the admin token file is opaque rather than JWT-shaped, the CLI cannot inspect acr, auth_time, or exp locally; Authifi's server-side authorization response remains authoritative for whether the token satisfies the step-up requirement.

The first dry-run must show the expected console, portal, and MCP changes and perform no writes. Apply requires confirmation. The final dry-run should report No changes. For removal of stale Floh-owned roles, groups, relationships, or managed permissions, first preview with --dry-run --prune, then run --apply --prune; prune requires an additional confirmation and never deletes resource servers, client assignments, or unrelated tenant objects. Do not use --yes for production maintenance unless an approved non-interactive procedure has captured and reviewed the exact plan.

Duplicate access-role names on different resource servers are safe, but prune is still authoritative for Floh-owned leftovers. The sync planner and apply path qualify each managed role by both resource-server identifier and role name. A leftover floh-* role on another resource server remains a prune target even when a desired role with the same name exists elsewhere, while the concrete desired same-named live role is preserved because prune only unlinks a group when that exact linked role id is scheduled for deletion.

On Authifi HTTP failures, CLI output is permanently redacted down to the allowlisted top-level WWW-Authenticate fields error, acr_values, and numeric max_age. The command never prints the admin token, any HTTP response body, error_description, arbitrary challenge parameters, excluded quoted challenge text, or URL userinfo/query secrets from credential-bearing Authifi base URLs.

Two OIDC clients back a deploy, and both must exist in the Authifi tenant before the first login attempt:

Client	Callback	Notes
`floh-client`	`https://<DEPLOY_DOMAIN>/bff/callback`	Console. The console BFF is the relying party.
`floh-portal-client`	`https://<DEPLOY_PORTAL_DOMAIN>/bff/callback`	Portal. The portal BFF is the relying party.

Both callbacks are /bff/callback, not /api/auth/callback. The portal gateway derives its callback from AUTH_BASE_URL and its bff.pathPrefix, so it cannot be overridden through PORTAL_OIDC_REDIRECT_URI.

The same command also registers floh-mcp-client, a confidential client with no browser origins. It backs the MCP server rather than a deployed web tier, so it is not required for login and its minted secret belongs in env/mcp.env, not in a deploy secret. See MCP Server Setup.

A client that already exists is never re-created, so setup checks the live one instead: if floh-mcp-client is missing the authorization_code or refresh_token grant, or authenticates with anything other than client_secret_post, the run stops before its first write and names the field to fix. Correct it in the Authifi admin UI, or delete the client and re-run.

Each managed client must be authorized for its channel resource server: floh-client uses FLOH_CONSOLE_AUDIENCE, floh-portal-client uses FLOH_PORTAL_AUDIENCE, and floh-mcp-client uses FLOH_MCP_AUDIENCE. The console BFF requests FLOH_CONSOLE_AUDIENCE on authorize; the portal BFF requests FLOH_PORTAL_AUDIENCE. Console, portal, MCP, and API identifiers must be pairwise distinct. Setup removes each managed client's sibling-channel assignments before reasserting the three channel links and their API-audience links. It does not remove unrelated client relations. FLOH_RESOURCE_ID (OIDC_AUDIENCE alias) never selects or falls back for a channel binding.

pnpm run setup-authifi-oidc-clients reconciles all of this idempotently. Reconciliation is authoritative: it replaces callbackUrls and postLogoutRedirectUris with exactly the origins you pass, so every registered URI you omit is deleted. On a tenant shared with local development that includes the developer localhost entries. Always preview first:

pnpm run setup-authifi-oidc-clients -- --dry-run \
  --identity-provider <your-authifi-identity-provider> \
  --console-origin https://floh.authilize.com --console-origin https://localhost:7072 \
  --portal-origin https://myfloh.authilize.com --portal-origin https://localhost:7073

--identity-provider is repeatable and is required whenever the plan includes creating the console client — the run aborts before the dry run prints anything without it. Drop the flag only once that client exists, since reconciling copies its providers. AUTHIFI_IDENTITY_PROVIDERS=nih,google in the environment is equivalent.

--vault does not exempt you. It is additive rather than exclusive: computePlan receives no vault flag and queues create-console whenever the console client is absent, so on a genuinely new tenant even setup-authifi-oidc-clients -- --vault --dry-run aborts on the same guard. Supply a provider on the first run against a tenant, whatever else the run is for.

The dry run prints every URI it would add or remove and closes with a removal count. Read that list before dropping --dry-run. See the portal guide for the full flag reference.

The OIDC dry-run does not preview numeric client-resource bindings before the clients exist, but it does resolve the tenant and validate all three resource servers before reporting success. Setup derives the unique tenant id from the initial all-clients listing; an entirely empty tenant requires AUTHIFI_TENANT_ID, and a configured value must match listed rows. Run the RBAC sync first, then verify in Authifi that each client is linked to its matching resource server. If apply fails, restore the previous complete origin list and rerun OIDC setup. For catalog rollback, restore the prior catalog revision, preview with pnpm run sync-authifi-rbac -- --dry-run --prune, then apply with --apply --prune. A normal dry-run/apply only reports extras; deletions and unlinks of catalog removals require prune. Relationship removals are planned before object deletion.

This rollout does not add runtime Authifi grant rechecks on delayed steps (LSA-9940).

Upgrading past LSA-9939 (legacy authorization removal)¶

The 20260901130000_drop_legacy_authorization_rbac migration drops the api_token, permission_override, role_permission, user_role, permission, and role tables. Two things must be true before you deploy it, or the release will fail closed:

A default approver group must exist. Approval fallback and default escalation used to scan user_role for admins; they now resolve settings.system.defaultApproverGroupId only. Migration 20260901125000_backfill_default_approver_group runs immediately before the drop. If the setting already names a user_group with at least one live member, it is a no-op. Otherwise it copies every live user (deleted = false and active = true) who currently has the legacy DB admin role — excluding the reserved system user — into a group named Default Approvers (reused if that name already exists) and writes the setting. If that live-admin set is empty, or the legacy tables are already gone and the setting is still missing, migrate throws; it will not invent approvers. After deploy you can change the group in Settings → System, but you cannot clear it: PUT /api/admin/settings rejects an explicit null, and the group routes already refuse to delete the selected group or remove its last member. The drop's up() still refuses any DDL unless the setting is a UUID naming a user_group with at least one non-deleted, active member.
Authifi must already be issuing scopes. Every request's permissions come from the access token's scope claim after this release. Re-run pnpm run sync-authifi-rbac -- --dry-run and then --apply so the catalog's roles — including the new console-only system:administer — exist in Authifi, and confirm your admin users hold them. A user whose token lacks system:administer loses admin-only console behavior (permanent user delete, project-authority bypass, scope=all listings, template/scheduler/email-log administration).

Also remove ALLOW_PERMISSION_OVERRIDES and any OIDC_ROLE_* variables from your environment; the server no longer reads them. floh_-prefixed API tokens are rejected with 401, so migrate any script or MCP client still using one to an OIDC refresh token (see MCP Setup).

Platform access rollout¶

This rollout assumes the console BFF's Authifi admin proxy is configured only by committed NODE_CONFIG plus OIDC_ISSUER (admin slot resource / Authifi API RSID), OIDC_JWKS_URI (discovery jwks_uri), and deploy-time AUTHIFI_ADMIN_TARGET. AUTHIFI_ADMIN_TARGET must stay the fixed tenant-scoped <authifi-base>/auth/admin/tenants/<positive-tenant-id> URL for the deploy target. Never derive it from browser input or request parameters.

Apply/deploy in this order:

apply Authifi desired state
deploy BFF image/config
deploy API runtime feature-flag support
flip the UI flag only after rollout evidence exists

Do not enable the Authifi proxy or the UI surface until LSA-9824 demonstrates that effective scopes equal requested ∩ client/resource allowed ∩ user-granted for both the authorization-code exchange and refresh. No client-credential fallback.

Live non-production verification matrix:

Floh console admin succeeds for assigned-namespace membership reads and writes.
Ordinary non-admin access fails.
Outside-namespace operations, privileged-group operations, and wrong-tenant operations fail.
Group CRUD and role CRUD fail.
Ordinary console login and Floh /api WebSocket traffic remain healthy.
forged browser Authorization is stripped before proxying, and CSRF-less mutation fails.
Authifi audit attributes the mutation to the human administrator without Floh minting or logging credentials.

Removal semantics:

Access removal becomes authoritative on the next token refresh/login, not immediately.
Worst-case exposure is exactly 3600 s, the validated access-token lifetime of the Authifi admin resource server identified by AUTHIFI_ADMIN_RESOURCE. The sync CLI resolves this value from the live resource-server list and fails closed if it exceeds one hour. RESOURCE_SERVER_CONFIG.accessToken.lifetime in scripts/sync-authifi-rbac.mjs owns only the Floh channel resource servers (console, portal, MCP) and must not be used for the admin proxy removal window.

Tenant-admin recovery for administrator self-removal:

Sign in as an Authifi tenant administrator, or use another console admin that still retains namespace access.
Restore the affected administrator's membership in the predefined floh-console:admin group through the supported Authifi recovery path.
Re-authenticate in Floh so the refreshed session picks up the restored grant.

Rollback paths:

Set PLATFORM_ACCESS_ENABLED=false.
Restore the prior console BFF image and NODE_CONFIG.
have an Authifi tenant administrator remove the namespace grant through Authifi's administrative surface. The Floh sync CLI intentionally exposes no grant-revocation method.

Authifi tenant-secrets vault¶

By default the server reads its secrets from the environment, which means every value in the Secrets table lives in GitHub. Setting the SECRETS_BACKEND variable to authifi switches the server to fetching them from the Authifi tenant-secrets vault at startup instead. See secrets management for the provider design.

Three properties of this switch drive everything below:

It is a total replacement, not a fallback. AuthifiSecretProvider serves only what the vault returns, and loadConfig resolves secrets through the provider alone — it does not consult process.env behind it. This matters because the deploy still writes the GitHub secret copies into ~/floh/.env: without that guarantee, a vault missing a key would boot on the stale environment value, cutover verification would falsely succeed, and a later rotation in the vault alone would keep using the old credential. A missing required secret instead resolves to its default, which validateProductionSecrets rejects under NODE_ENV=production, so a partial migration fails the boot rather than degrading quietly.
It covers the server only. postgres, redis, and portal-bff read ~/floh/.env directly and have no vault client, so their secrets stay in GitHub regardless of this setting.
It is a runtime switch, not a build-time one. Flipping the variable and re-running the deploy is the whole cutover, and flipping it back is the whole rollback.

Cutover¶

Create the tenant secrets. Because the vault is the only source the server consults, every secret it reads must exist there before the switch, prefixed with FLOH_. The full set is SECRET_KEYS in packages/shared; copy each value from the existing GitHub secret rather than generating a new one.

Tenant secret	Required
`FLOH_DB_PASSWORD`	yes
`FLOH_JWT_SECRET`	yes
`FLOH_SESSION_SECRET`	yes
`FLOH_SESSION_ENCRYPTION_KEY`	yes
`FLOH_AUDIT_CHECKPOINT_KEY`	yes
`FLOH_CONNECTOR_ENCRYPTION_KEY`	yes
`FLOH_OIDC_CLIENT_SECRET`	yes
`FLOH_PORTAL_OIDC_CLIENT_SECRET`	yes
`FLOH_REDIS_PASSWORD`	when Redis auth is enabled
`FLOH_SMTP_USER`	when SMTP auth is in use
`FLOH_SMTP_PASS`	when SMTP auth is in use
`FLOH_CONNECTOR_ENCRYPTION_KEY_PREVIOUS`	whenever the GitHub copy is set
`FLOH_AUDIT_CHECKPOINT_KEY_PREVIOUS`	whenever the GitHub copy is set

Re-generating rather than copying CONNECTOR_ENCRYPTION_KEY leaves stored connector credentials undecryptable, and re-generating AUDIT_CHECKPOINT_KEY breaks audit-chain verification.

The two _PREVIOUS keys are the easiest to miss and the quietest to get wrong. They exist only during a key rotation, they are optional, and nothing validates them at startup — so a cutover performed mid-rotation that omits them boots cleanly and then cannot decrypt connector credentials written under the old key, or verify historical audit checkpoints. If either GitHub secret is currently set, carry it across before flipping the switch.

Register the vault client. It authenticates with private_key_jwt, so it needs a key pair rather than a client secret. The setup script handles the whole exchange — do not generate the key by hand:

Origins are authoritative on every run, including this one. The script reconciles the console and portal clients whether or not you are touching them, and any registered URI you do not pass is deleted — so a bare --vault on a tenant shared with local development silently removes the localhost callbacks and breaks local login. Pass the full origin set, add --local-dev if that tenant also serves local development, and preview with --dry-run first, which is the only mode that reports removals:

pnpm run setup-authifi-oidc-clients -- --vault --dry-run \
  --console-origin https://floh.authilize.com \
  --portal-origin https://myfloh.authilize.com

# same command without --dry-run once the diff shows no unexpected removals

This registers floh-vault-client with private_key_jwt and client_credentials, has Authifi mint the JWK and register the public half, and writes the private key to ~/.floh/vault-key.pem at mode 0600 (override with --vault-key-file; the script refuses any path inside the repo).

Two things that look like they should be steps here and are not. --vault is required — without it the script plans no vault client at all. And --print-commands returns before the key exchange, so it can show you the client-creation recipe but can never register a JWK; use it to preview, not to execute. The SECRETS_MANAGER.LIST and SECRETS_MANAGER.PLAIN_SECRET scopes are not granted on the client either — they are requested at token time and already default correctly in vault-config.ts.

Store the private key, and keep the local copy:

gh secret set AUTHIFI_VAULT_PRIVATE_KEY --repo Authifi/floh --env dev < ~/.floh/vault-key.pem

Do not delete ~/.floh/vault-key.pem afterwards. Its existence is the script's entire rotation guard: ensureVaultKey decides whether a run would be a rotation by checking whether that path exists, and POST .../jwk replaces the registered key immediately. Delete the file and the next --vault run mints a replacement without warning, while GitHub still holds the superseded key — the server then cannot authenticate to the vault on its next restart. Keep it in a password manager or another encrypted store, and treat a deliberate rotation as "move the file aside, re-run, upload the new key" in one sitting.

Flip the switch. Set the SECRETS_BACKEND variable to authifi and re-run the deploy.
Confirm the source. The server logs the backend it resolved as its first startup line, and a vault failure aborts the boot with Fatal: Floh server failed to start rather than falling back:

docker compose -p floh -f docker-compose.deploy.yml \
  --env-file .env --env-file env/console.env --env-file env/portal.env \
  logs server | grep -i 'Loading secrets from'

The worker entrypoint reports the same line, but docker-compose.deploy.yml defines no worker service (only the dev docker-compose.yml does), so there is nothing to query for it here.

The deploy rejects SECRETS_BACKEND=authifi when AUTHIFI_VAULT_PRIVATE_KEY is unset, and rejects any value other than env or authifi — an unrecognized value would otherwise fall back to env at runtime and leave the vault quietly disabled. The non-secret locators (AUTHIFI_VAULT_URL, _TENANT, _TENANT_ID, _CLIENT_ID, _KEY_FILE) live in config/public/ci.env; the private key is written to ~/floh/vault-key.pem at mode 0600 and bind-mounted, never into .env.

DB_PASSWORD must be kept in both places — and neither is where the database actually keeps it. The postgres container gets its password from ~/floh/.env, so FLOH_DB_PASSWORD in the vault and DB_PASSWORD in GitHub have to hold the same value. Rotating one without the other leaves the server unable to authenticate against a database it can still see. Rotating both is still not enough on an existing volume, because POSTGRES_PASSWORD is honored only at initialization and the live floh role keeps its original password; follow Rotating DB_PASSWORD, which adds the ALTER ROLE. PORTAL_OIDC_CLIENT_SECRET is not a second instance of this rule: the API server never reads it, so the vault copy is inert. It reaches the portal BFF from the GitHub secret via ~/floh/env/portal.env as AUTH_CLIENT_SECRET, and rotating the GitHub secret is what takes effect. Nothing detects a DB_PASSWORD divergence at deploy time — it surfaces as an authentication failure at startup.

Rollback¶

Set SECRETS_BACKEND back to env (or unset it) and re-run the deploy. This works only while the GitHub secrets are still populated, which is why the cutover copies values into the vault rather than moving them. Retiring the GitHub copies is a separate decision to make after the vault path has held.

Deploying¶

Go to Actions → Deploy
Click Run workflow
Optionally specify a branch or tag (defaults to main)
The workflow builds the 5 Floh images on GitHub-hosted runners, pushes to GHCR, then the self-hosted runner pulls them plus the upstream BFF gateway, syncs config, and starts services

The build matrix produces:

Image	Role
`ghcr.io/.../floh/server`	Floh API (Node).
`ghcr.io/.../floh/web`	Floh admin SPA (nginx). `FORM_BUILDER_EMBED_URL` is baked in at build time so the Workflow Designer's Visual editor toggle is enabled.
`ghcr.io/.../floh/portal-web`	Self-service portal SPA (nginx).
`ghcr.io/.../floh/form-builder-app`	Standalone Form Builder SPA (nginx). Iframed by the admin SPA from `DEPLOY_FORM_BUILDER_DOMAIN`.
`ghcr.io/.../floh/mcp`	Floh MCP server (stdio). Pull-only artifact — operators / MCP clients run it via `docker run --rm -i …`. Not registered as a service in `docker-compose.deploy.yml`.

The portal BFF is not a Floh-built image. Since LSA-9825 it is the upstream Authifi gateway, pinned in docker/docker-compose.deploy.yml:

Image	Role
`ghcr.io/authifi/idbroker-tools/bff-gateway:3.3.0`	Portal OIDC relying party and API proxy. Configured by `docker/bff/portal.json` (mounted at `/app/config/local.json`) plus compose env vars.

The deploy runner therefore needs pull access to the Authifi/idbroker-tools package scope, not just Authifi/floh. The version pin is deliberate and is asserted by packages/portal-bff/test/config-invariants.test.ts; read the comment above BFF_VERSION there before bumping it.

First deploy takes ~5 minutes (image pulls). Subsequent deploys are faster due to layer caching.

Running the MCP server¶

The MCP server speaks stdio, so MCP clients invoke it on demand rather than pinning it to a port. After the deploy publishes the image, an MCP-aware client (e.g. Claude Desktop, Cursor) can pull and run it directly. Deploy derives the GHCR prefix from GITHUB_REPOSITORY (lowercased); the compose default and the example below match the current Authifi org:

docker run --rm -i \
  -e FLOH_API_URL="https://floh.authilize.com/api" \
  -e OIDC_ISSUER="..." -e OIDC_CLIENT_ID="floh-mcp-client" \
  -e OIDC_CLIENT_SECRET="..." \
  -e FLOH_MCP_AUDIENCE="..." \
  -e FLOH_REFRESH_TOKEN="..." \
  ghcr.io/authifi/floh/mcp:latest

floh-mcp-client is registered with client_secret_post, so the secret is not optional: without it Authifi rejects the refresh request before any tool runs. Use the value setup captured into env/mcp.env as MCP_OIDC_CLIENT_SECRET (either env key works in the MCP process; MCP_OIDC_CLIENT_SECRET wins).

Compose services pull ${IMAGE_PREFIX:-ghcr.io/authifi/floh}/<name>:latest. The Deploy workflow writes IMAGE_PREFIX into ~/floh/.env so host pulls track the publishing org without editing compose. MCP authentication is OIDC refresh-token exchange only (FLOH_REFRESH_TOKEN + OIDC_ISSUER + OIDC_CLIENT_ID=floh-mcp-client or MCP_OIDC_CLIENT_ID). Do not put the MCP secret in env/console.env. See packages/mcp/src/index.ts for the full env-var contract.

Operations¶

Running compose commands on the host¶

Every compose command on the deployed host needs the project name and all three env files. Define this helper once per shell session and use it for the commands below:

ssh -i your-key.pem ubuntu@<ELASTIC_IP>
cd ~/floh
compose() {
  docker compose -p floh -f docker-compose.deploy.yml \
    --env-file .env --env-file env/console.env --env-file env/portal.env "$@"
}

Omitting the --env-file flags is not cosmetic. Compose interpolates every unresolved ${VAR} to an empty string, so:

compose up without them recreates containers with empty configuration — including empty SERVER_CERT / SERVER_KEY on the BFFs, which reproduces the ERR_OSSL_PEM_NO_START_LINE crash loop on a stack that was healthy a moment earlier. This is the dangerous case, and it extends to any other command that creates or recreates containers (run, create).
restart and stop are safe but noisy. Neither recreates a container, so the empty interpolation never reaches the running config — they act on the containers exactly as up last created them. You still get the warnings below, which is why the habit is worth keeping.
Read-only commands (ps, logs) emit misleading warnings, e.g. The "AUTHIFI_ADMIN_TARGET" variable is not set. Defaulting to a blank string. Those warnings say nothing about the running containers, which were created with the values intact. This one wasted real debugging time; see the failure reference below.

Omitting -p floh is equally load-bearing: compose derives the project name from the directory otherwise, and commands then target a different (usually empty) set of containers than the deploy created.

Service healthchecks¶

The deploy polls every compose service that defines a HEALTHCHECK (GATED in .github/workflows/deploy.yml). form-builder — and any future nginxinc/nginx-unprivileged service — must probe http://127.0.0.1:8080/, not localhost. BusyBox wget (the alpine image's wget, BusyBox v1.37+) prefers IPv6 for localhost, and the nginx configs listen on IPv4 :8080 only, so a localhost probe stays unhealthy while Caddy is happily serving the SPA. See floh#1109.

View logs¶

ssh -i your-key.pem ubuntu@<ELASTIC_IP>
cd ~/floh
compose logs -f          # all services
compose logs -f server   # single service

Restart a service¶

compose restart server

Update secrets or deploy variables¶

Secrets and deploy variables are managed in the GitHub dev environment. Update values in Settings → Environments → dev and re-run the deploy workflow — the workflow writes ~/floh/.env, ~/floh/env/console.env, and ~/floh/env/portal.env from environment secrets/vars on every deploy.

That is the whole procedure for most secrets, but four of them break something that the deploy will still report as successful — DB_PASSWORD, CONNECTOR_ENCRYPTION_KEY, AUDIT_CHECKPOINT_KEY, and the OIDC client secrets. Check Rotating an existing secret before changing any of those.

Update non-sensitive runtime config¶

Edit config/public/ci.env (or base.env) in the repo, merge to the deploy branch, and re-run the deploy workflow. The workflow copies these files to ~/floh/public/, and docker/bff/portal.json to ~/floh/bff/.

Editing a key that a BFF service reads is a two-place change: add it to ci.env and to the read-env-values.mjs invocation (or the matching process file write) in .github/workflows/deploy.yml, since those containers do not mount config/public.

Non-secret keys (OIDC settings, URLs, feature flags) should be added to the public config files rather than GitHub environment variables or .env. When public config is loaded, process.env is only consulted as a fallback if ALLOW_LEGACY_ENV_NON_SECRET=true is set (see packages/server/src/config/index.ts). The NON_SECRET_KEYS list in that file defines which keys follow this rule.

Connector key rotation runbook¶

Use this when changing CONNECTOR_ENCRYPTION_KEY without breaking existing connector secrets.

Generate a new 64-char hex key:

openssl rand -hex 32

In the GitHub dev environment secrets, set:
CONNECTOR_ENCRYPTION_KEY → the new key
CONNECTOR_ENCRYPTION_KEY_PREVIOUS → the old key
Run the deploy workflow to apply the change.
Rotate connector secrets:
UI: Connectors -> Rotate Keys
API:

curl -X POST https://<DEPLOY_DOMAIN>/api/connectors/rotate-keys \
  -H "Authorization: Bearer <admin-token>"

Verify the summary response has failed: [].
Clear CONNECTOR_ENCRYPTION_KEY_PREVIOUS from the dev environment secrets and re-deploy.

Check service status¶

compose ps

Access MailHog¶

Open http://<ELASTIC_IP>:8025 in your browser (if port 8025 is open in the security group).

Database access¶

compose exec postgres psql -U floh -d floh

Failure reference¶

Deploy and first-login failures, keyed on the error text you actually see. Search this page for the literal string.

Error text	Where it appears	Cause
`ERR_OSSL_PEM_NO_START_LINE`	BFF container logs	Malformed TLS PEM secret
`<service> did not become healthy after startup`	Deploy job	Container crash-looping; the cause is in its logs
`invalid_redirect_uri`	Browser, after login	Callback URL not registered on the OIDC client
`invalid_client (client authentication failed: invalid secret provided)`	Browser, after login	`OIDC_CLIENT_SECRET` does not match the tenant
`The "AUTHIFI_ADMIN_TARGET" variable is not set`	Deploy job	Benign — a compose call missing `--env-file`

`ERR_OSSL_PEM_NO_START_LINE`¶

The BFF crash-loops at startup inside configSecureContext → setCerts. Node could not parse the PEM it was handed, which means the secret's content is wrong — not the file plumbing.

An empty value does not produce this error, so emptiness is ruled out when you see it. The three shapes that do produce it are a PEM whose newlines became literal \n escapes, a base64 encoding of the PEM, and a PEM wrapped in quote characters. All three come from capturing the secret with anything other than gh secret set <NAME> … < <file>.

Fix by re-setting the secret from the file and re-running the deploy:

gh secret set CONSOLE_BFF_TLS_CERT --repo Authifi/floh --env dev < console.crt
gh secret set CONSOLE_BFF_TLS_KEY  --repo Authifi/floh --env dev < console.key

The deploy preflight now rejects all three shapes before writing any host file, so this should surface as a named preflight error rather than a crash loop. See Deploy preflight for TLS secrets.

A second cause with the same signature: running compose up on the host without the --env-file flags, which recreates the container with an empty certificate. (restart cannot cause this — it does not recreate the container.) See Running compose commands on the host.

`<service> did not become healthy after startup`¶

The health gate polled the service for 120s and it never reported healthy or running. A crash-looping container sits in restarting and never reaches a terminal state, so this timeout — not a terminal-state error — is what reports it. The deploy prints the last 50 log lines for the failing service immediately after this message; the real cause is there, not in the gate message.

`invalid_redirect_uri`¶

redirect_uri did not match any of the client's registered redirect_uris. The browser reaches the IdP, so client ID and issuer are correct; only the callback URL is unregistered.

The console's RP is the console BFF, so the callback is https://<DEPLOY_DOMAIN>/bff/callback — not the SPA origin. The portal's is https://<DEPLOY_PORTAL_DOMAIN>/bff/callback. Re-register with the canonical client definitions rather than hand-editing:

pnpm run setup-authifi-oidc-clients

See Authifi client registration. Any domain change requires this step; it is easy to miss because nothing else fails.

`invalid_client (client authentication failed: invalid secret provided)`¶

The redirect worked and the BFF is exchanging the code, so this is the client secret alone: the value in OIDC_CLIENT_SECRET (or PORTAL_OIDC_CLIENT_SECRET) is not the secret the tenant holds for that client.

The usual cause is drift — the tenant's client was recreated or its secret rotated, while the GitHub environment kept the old value. Authifi cannot re-read an existing client secret, so there is no way to look up the current value: you either hold a copy, or you rotate and capture the new one.

# Check when the secret was last touched — a date long before the tenant's
# client was created is the tell.
env -u GITHUB_TOKEN gh secret list --repo Authifi/floh --env dev

# If you hold a verified copy (e.g. a working local env/console.env):
env -u GITHUB_TOKEN gh secret set OIDC_CLIENT_SECRET --repo Authifi/floh --env dev < secret.txt

# Otherwise rotate and capture the minted value:
pnpm run setup-authifi-oidc-clients -- --rotate-secrets

Rotating invalidates any other copy of that secret. See Rotating an existing secret.

The "AUTHIFI_ADMIN_TARGET" variable is not set¶

level=warning msg="The \"AUTHIFI_ADMIN_TARGET\" variable is not set. Defaulting to a blank string."

Benign. This is compose interpolation warning about its own invocation, not a statement about the running containers. It appears when a ps or logs call omits the --env-file flags that the up call used, so the variable really was set when the containers were created.

It is worth knowing because the warning repeats once per poll during a health gate, which makes it look like the cause of whatever is failing. It is not. The deploy workflow's read-only compose calls now pass the env files, so these warnings should no longer appear; if you see them from a manual command, add the flags.

Changing Domains Later¶

Update DNS A records for the new domain
Update DEPLOY_DOMAIN and/or DEPLOY_PORTAL_DOMAIN in the GitHub dev environment variables, and/or DEPLOY_FORM_BUILDER_DOMAIN in the repository-level variables (Settings → Variables → Actions)
Update FRONTEND_URL, PORTAL_FRONTEND_URL, OIDC_REDIRECT_URI, ALLOWED_ORIGINS, and ALLOWED_PORTAL_ORIGINS in config/public/ci.env and merge
Re-register the OIDC clients against the new origins (see Authifi client registration below) — the console callback and the portal /bff/callback both carry the domain
Run the deploy workflow
Caddy auto-provisions new Let's Encrypt certificates

Changing DEPLOY_FORM_BUILDER_DOMAIN requires a fresh web image build because the embed URL is baked in at build time. The deploy workflow rebuilds on every run, so re-running the workflow is sufficient.

Cost Estimate¶

Resource	Monthly Cost
EC2 t3.medium	~$33
EBS 30 GB gp3	~$2.50
Elastic IP (attached)	Free
Data transfer	~$1-5
GHCR	Free (included in GitHub plan)
Total	~$37-41