Observability (OpenTelemetry Phase 1)¶

Floh can bootstrap the OpenTelemetry Node SDK in the HTTP server and BullMQ worker. The feature is off by default. When enabled, auto-instrumentation emits HTTP, database, Redis, BullMQ, and Node runtime telemetry to an OTLP-compatible collector.

Enable Locally¶

OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

With the bundled compose stack, the collector is available as http://otel-collector:4318 from the server and worker containers. Set OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 in .env when enabling OTel for compose. The collector exports to the debug exporter so local operators can verify telemetry without a vendor account:

docker compose -f docker/docker-compose.yml up otel-collector server
docker compose -f docker/docker-compose.yml logs -f otel-collector

The server and worker do not depend on the collector by default because OTel is off unless OTEL_ENABLED=true. When testing OTel locally, start the collector first (or include it in the same compose command) before starting server/worker.

Configuration¶

Variable	Required	Notes
`OTEL_ENABLED`	no	Truthy values are `true`, `1`, `yes`; unset is disabled.
`OTEL_EXPORTER_OTLP_ENDPOINT`	yes when enabled	OTLP/HTTP base URL, for example `http://otel-collector:4318`. Embedded credentials are rejected.
`OTEL_EXPORTER_OTLP_HEADERS`	no	Comma-separated `key=value` auth headers, for example `x-honeycomb-team=...`.
`OTEL_SERVICE_NAME`	no	Defaults to `floh-server` or `floh-worker` by entry point.
`OTEL_SERVICE_VERSION`	no	Defaults to package version when available.
`OTEL_RESOURCE_ATTRIBUTES`	no	Comma-separated resource tags. Floh's canonical service keys win over spoofed values.
`OTEL_NODE_RESOURCE_DETECTORS`	ignored	Floh constructs `new NodeSDK({ autoDetectResources: false, resourceDetectors: [] })`. Setting this (including `all`) has no effect and does not enable cloud/container metadata lookups.
`OTEL_PROPAGATORS`	no	If set, only `tracecontext` and `baggage` are accepted.
`OTEL_METRIC_EXPORT_INTERVAL_MS`	no	Defaults to `60000`; minimum `1000`.

Fail-Closed Behavior¶

OTEL_ENABLED=true without a valid OTEL_EXPORTER_OTLP_ENDPOINT throws during startup. Malformed header/resource tokens also throw. This is deliberate: silently starting without telemetry would hide production observability outages.

The endpoint must use http:// or https:// and must not contain embedded credentials. Put backend credentials in OTEL_EXPORTER_OTLP_HEADERS or in the collector's exporter configuration instead.

Trace ID Alignment¶

When an OTel span is active, Floh seeds RequestContext.traceId from the active span's trace id. That keeps system_log.trace_id aligned with the trace id seen in the collector. When OTel is disabled or no span is active, Floh preserves the existing fallback: parse the inbound W3C traceparent header.

Resource Attributes¶

Floh disables automatic cloud/container/Kubernetes resource detectors so local and self-hosted deployments do not trigger metadata lookups. Operators who want deployment metadata in traces should set OTEL_RESOURCE_ATTRIBUTES, for example cloud.provider=aws,k8s.cluster.name=prod.

Floh does not use the SDK's startNodeSDK() helper (file- or environment-driven declarative config). @opentelemetry/sdk-node@0.222.0 fail-fasts that helper when OTEL_NODE_RESOURCE_DETECTORS=all includes unknown experimental detector names such as "container", and then leaves a no-op SDK. Floh's new NodeSDK() path is unaffected; do not introduce startNodeSDK() or an OTel config file for the server/worker process.

Production Hardening¶

The bundled collector config is a development baseline. Before production:

Enable TLS on the OTLP receiver or place the collector behind a trusted TLS terminator.
Require bearer-token auth or mTLS on the collector receiver.
Bind collector ports to loopback/private interfaces on multi-tenant hosts.
Replace the debug exporter with your vendor/exporter pipeline.

Phase 2 — Domain Metric Inventory¶

Phase 2 (LSA-8854) adds a small, well-typed Floh domain metric surface on top of Phase 1's bootstrap. When OTEL_ENABLED=true, the workflow engine emits two metrics per terminal run. The inventory is the contract Phase 3 will extend; no new env vars are required.

Metric inventory¶

Name	Kind	Unit	Status	Emitted from
`floh.workflow.run.terminated`	counter	`1`	registered	`WorkflowEngine.executeRun` (engine-driven)
`floh.workflow.run.duration`	histogram	`s`	registered	`WorkflowEngine.executeRun` (engine-driven)
`floh.workflow.step.executed`	counter	`1`	registered (Phase 3)	`WorkflowEngine.walkFrom` (per terminal step)
`floh.connector.request.duration`	histogram	`s`	registered (Phase 3)	`executeConnector` wrapper
`floh.auth.session.created`	counter	`1`	registered (Phase 3c)	`/api/auth/callback` (4 branches)
`floh.queue.depth`	gauge	`1`	registered (Phase 3c)	BullMQ observable callback (HTTP + worker)

All 6 inventory entries are now registered: true. The inventory contract is structurally complete; further metrics would extend (not flip) this list.

Attribute vocabulary¶

Phase 2 attributes are intentionally low-cardinality. The maximum number of unique time series per registered metric is 18 (6 categories × 3 outcomes).

Attribute key	Allowed values
`floh.workflow.category`	`user`, `user_self_service`, `group`, `project`, `general`, `pam`
`floh.run.outcome`	`completed`, `failed`, `cancelled`

The vocabulary is defined as compile-time exhaustive over WorkflowCategory (from @floh/shared) and over the RunOutcome literal union; adding a new category in shared makes the server fail typecheck until the vocabulary is updated.

Cardinality budget¶

Every emission is validated by assertAttributesWithinBudget(name, attrs) before being sent to the meter. The validator enforces both:

Allowed keyset — the attribute keys must be a subset of the metric's declared budget (e.g. only floh.workflow.category and floh.run.outcome for both floh.workflow.run.* metrics).
Allowed values — each known key's value must be one of the enumerated low-cardinality strings.

Behavior:

In test and development (NODE_ENV !== "production") the validator throws a MetricBudgetViolationError so cardinality regressions surface immediately in CI.
In production it logs a diag.warn and returns false, letting the recorder drop the emission rather than degrade the request.

Redaction / PII policy¶

Phase 2 attributes never contain workflow IDs, run IDs, user IDs, tenant IDs, emails, display names, or any other identifier-like payload. The typed attribute factory's parameters are unions, not string; an as-cast bypass is defended in depth by the budget validator above. A unit test (recorders/workflow-run.test.ts) asserts the emitted attribute keyset is exactly { "floh.workflow.category", "floh.run.outcome" }.

What Phase 2 does NOT emit (all metric placeholders now resolved)¶

Phase 3 (LSA-8855 + LSA-8873) shipped step + connector metrics, manual workflow / step / connector spans, BullMQ cross-process trace propagation, and the route-driven termination emissions. See the Phase 3 section below.
Phase 3c (LSA-8874) shipped the floh.auth.session.created counter and the floh.queue.depth observable gauge — closing the last two registered: false placeholders. See the Phase 3c section.

Still deferred:

Per-workflow-definition or per-tenant attributes on the run-level metrics (typed barrier; no follow-up ticket).
pino / system_log → OTLP log bridge (Phase 4 — LSA-8856).

Phase 3 — Manual Spans + BullMQ Propagation + Step/Connector Metrics + Route Emissions¶

Phase 3 (LSA-8855 + sub-task LSA-8873) wires manual workflow / step / connector spans, BullMQ cross-process trace propagation, two new registered metrics, and route-driven termination emissions. With Phase 3 deployed, a workflow run that crosses HTTP → BullMQ → worker → connector produces a single connected trace AND the four floh.workflow.run.* / floh.workflow.step.* / floh.connector.request.* time series.

Span inventory¶

Span name	Wrapper	Attributes
`floh.workflow.run`	`withWorkflowRunSpan`	`floh.workflow.id`, `floh.run.id`, `floh.workflow.category`
`floh.workflow.step`	`withWorkflowStepSpan`	`floh.step.id`, `floh.step.type`, `floh.step.outcome`
`floh.connector.request`	`withConnectorRequestSpan`	`floh.connector.kind`, `floh.connector.outcome`

All three spans are created inside packages/server/src/observability/tracing/spans/**. An architectural test (packages/server/test/unit/architecture/otel-span-naming.test.ts) enforces that no startActiveSpan / startSpan call exists elsewhere in packages/server/src/ and that every span name matches ^floh\.[a-z][a-z0-9_]*(\.[a-z][a-z0-9_]*)+$ — the same shape as the metric naming pattern.

BullMQ cross-process trace propagation¶

The producer (SchedulerService.addJob / addDelayedJob) injects the current OTel trace context into the BullMQ payload under the namespaced key __floh_otel as a W3C traceparent carrier. The worker handler extracts via extractTraceContext(job.data) and runs the handler inside context.with(...) so the engine's spans become children of the producer's span.

Failure modes that all fall back to ROOT_CONTEXT (the handler still runs):

The carrier key is absent (job enqueued before Phase 3 deployment).
The carrier value is malformed (not a plain object).
The W3C propagator throws on the extract call.

On the malformed-carrier path the engine emits exactly one diag.warn per call so operators can see the regression without flooding logs.

This PR ships downstream-only propagation (server → worker). The worker-initiated direction (worker enqueues a job the server later resumes) has no current call site; routing it through injectTraceContext is a single-line change if a future use case appears.

Phase 3 metric inventory¶

Name	Kind	Unit	Attributes	Cardinality (max)
`floh.workflow.step.executed`	counter	`1`	`floh.workflow.category` × `floh.step.type` × `floh.step.outcome`	6 × 24 × 2 = 288
`floh.connector.request.duration`	histogram	`s`	`floh.connector.kind` × `floh.connector.outcome`	~10 × 4 = 40

Phase 2's two run-level metrics gain a new floh.run.terminator attribute (enum: engine, route_skipped, route_failed) so operator dashboards can split engine-driven from route-driven terminations.

Attribute vocabulary additions (Phase 3)¶

Attribute key	Allowed values
`floh.run.terminator`	`engine` (default), `route_skipped`, `route_failed`
`floh.step.type`	The 24-value `StepType` union from `@floh/shared` (compile-time exhaustive via `Record<StepType, true>`)
`floh.step.outcome` (metric)	`success`, `failed` (terminal only; paused steps live on the span attribute, not the metric)
`floh.step.outcome` (span)	`success`, `failed`, `paused`
`floh.connector.kind`	Any non-empty string matching `^[a-z][a-z0-9_-]*$` (bounded by the registered-connector list at boot)
`floh.connector.outcome`	`success`, `failed`, `timeout`, `validation_error` (last two reserved for Phase 4+ richer error mapping)

Span PII / cardinality discipline¶

Spans tolerate higher cardinality than metrics (one-shot per request rather than a continuous time series), so workflow / run / step IDs are acceptable as span attributes — they would NOT be acceptable on a metric label. Span attributes still never carry:

Workflow variable VALUES (variable names are also not on spans today — span discipline matches the access-log redaction contract).
Authorization headers, OIDC tokens, decrypted secrets.
PII-flagged custom user attributes.

Enforcement: the typed attribute factories in packages/server/src/observability/tracing/span-attributes.ts accept only union / id-shape values. A setAttribute call outside the factory module is caught by the architectural span-naming test (which inverts the metrics-side "no spans outside metrics dir" pattern: "no spans created outside tracing/spans/").

What Phase 3 does NOT emit¶

These remain deferred:

floh.auth.session.created and floh.queue.depth (Cluster C — LSA-8874).
Bidirectional BullMQ propagation (worker-initiated context flow). No follow-up ticket — open when a use case appears.
Per-step-executor manual spans (the step span wraps the executor call as a unit; finer-grained spans inside document-submission / user-prompt executors are a future LSA-8855 follow-up).
Span-level sampling configuration. SDK default sampler is AlwaysOn; per-span sampling needs operator input.
pino → OTLP log bridge (Phase 4 — LSA-8856).

Reviewer requests to add behavior from this list during PR review are scope expansion and tracked separately.

Phase 3c — Auth Session Counter + Queue Depth Gauge¶

Phase 3c (LSA-8874) closes the last two registered: false placeholders from the Phase 2 inventory. After this PR, every metric in METRIC_INVENTORY has a covering emission site.

`floh.auth.session.created` (counter, unit `1`)¶

Emitted exactly once per /api/auth/callback invocation, from any of four terminal branches:

Branch (in routes.ts)	Outcome bucket
IdP returned no `code` (user cancel / `access_denied`)	`denied`
OIDC succeeded but the Floh account is soft-deleted	`denied`
Session minted	`success`
Catch-all error (token exchange / userinfo / upsert threw)	`error`

Attributes (cardinality budget ≈ N providers × 3 outcomes, typically < 10):

Key	Values
`floh.auth.provider`	Allowlisted alias from `OIDC_PROVIDER_ALIASES` env var, OR `"unknown"` sentinel for any unmatched `iss`.
`floh.auth.outcome`	Closed enum: `success` / `denied` / `error`.

The success / denied split is user-attributable (no operator action needed); error is server-attributable and is the right alert axis.

`floh.queue.depth` (observable gauge, unit `1`)¶

Polled at every metric reader tick (default 60s) on BOTH the HTTP and worker processes. Reads BullMQ Queue#getJobCounts("waiting","active","delayed","failed") for each registered queue. Emissions are tagged with the existing service.name resource attribute (floh-server vs floh-worker) so dashboards can split or sum across the two processes.

Attributes (cardinality budget = 4 queues × 4 states = 16 series per process):

Key	Values
`floh.queue.name`	Closed enum: `workflow-execution`, `escalation`, `lifecycle`, `integrations`.
`floh.queue.state`	Closed enum: `waiting`, `active`, `delayed`, `failed`. (Subset of BullMQ's 8 states — we skip `completed` / `paused` / `prioritized` / `waiting-children` because they don't actionably represent backlog.)

Robustness:

A getJobCounts failure on ONE queue (e.g. transient Redis disconnect) is captured per-queue. The failed queue's observations are SKIPPED for that tick (NOT zeroed — zero is reserved for "queue is genuinely empty"); the other queues' observations are still recorded.
The observable callback is unregistered BEFORE schedulerService.close() in the shutdown hook so the OTel reader cannot poll a closed Redis connection.

`OIDC_PROVIDER_ALIASES` env var¶

JSON map of {iss-url: short-alias} consumed at server bootstrap. Example:

OIDC_PROVIDER_ALIASES='{"https://auth.example.com":"authifi","https://kc.internal":"keycloak-prod"}'

Deploy-day note: until the operator populates this map, every auth session reports provider="unknown" (the counter still emits; only the split is collapsed). The variable is fail-closed: malformed JSON, non-object payloads, or non-string values cause a single boot console.warn and an empty allowlist — never propagated as label values.

Hot-reload is out of scope; the alias map is captured at bootstrap and a config change requires a process restart.

Operator-visible deltas at deploy¶

Two new metric series families appear in dashboards.
Until OIDC_PROVIDER_ALIASES is populated, every auth session series is tagged provider="unknown". A single diag.warn per unique unknown iss surfaces in operator logs.
floh.queue.depth series for queues that are genuinely empty emit 0 — this is intentional, so empty-vs-missing is distinguishable.

Phase 4 — pino → OTLP log bridge¶

Phase 4 (LSA-8856) ships the optional pino → OTLP log exporter that closes the LSA-8852 OTel epic. With Phase 4 enabled, the same collector that receives traces (Phase 1/3) and metrics (Phase 2/3/3c) also receives the Fastify HTTP server's pino lines — carrying the active trace_id, the request id, and the user-supplied severity. Operators get a single- pane view: a 5xx span links to the request's pino error line; a workflow run span links to the run's app.log.warn lines.

Enabling the bridge¶

# Required (already in place if you enabled Phase 1):
export OTEL_ENABLED=true
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
# Phase 4 opt-in:
export OTEL_LOGS_ENABLED=true

OTEL_LOGS_ENABLED accepts case-insensitive true / 1 / yes (with whitespace trimmed). Anything else — including unset — resolves to false and the bridge stays off. The log endpoint is derived from the existing OTEL_EXPORTER_OTLP_ENDPOINT (specifically ${endpoint}/v1/logs); there is no log-specific endpoint or header override.

What ships to OTLP¶

Every Fastify-emitted pino line — request-completion logs, ad-hoc app.log.* / request.log.*, the error handler's 5xx structured payload, boot-time app.log.warns — becomes an OTLP log record on the same resource as traces + metrics (service.name, service.version, deployment.environment, service.instance.id). Records carry:

Architecture note — single log pipeline. Phase 4 enables ONE log path to your collector: the worker-thread pino-opentelemetry-transport that wraps Fastify's pino instance. The main-thread NodeSDK is also constructed with an OTLP BatchLogRecordProcessor, but it serves OTel-native log events from auto-instrumentations only — and the pino auto-instrumentation (@opentelemetry/instrumentation-pino) is explicitly DISABLED so pino records never flow through it (which would otherwise produce duplicate log lines on the collector). Both pipelines are configured with an identical Resource via a shared buildResourceAttributes helper so the operator sees a single consistent service entity.

trace_id — set via TWO independent paths for resilience: a pino child binding in the onRequest hook (covers async hops that the transport's active-span lookup might miss) AND the transport's own trace.getActiveSpan() call at emit time.
reqId — Fastify's per-request id (UUID, or a validated upstream X-Request-Id).
Severity number + text (pino's level mapped via the transport's defaults).

Redaction (always on, regardless of OTLP)¶

Pino redaction is configured at Fastify init and scrubs the following credential-shaped paths from every emitted record (stdout AND OTLP). The censor value is the literal string "[REDACTED]":

Header surfaces: authorization, cookie, set-cookie (top-level and nested under req.headers / res.headers).
Body / claim surfaces: password, token, secret, apiKey, accessToken, refreshToken, clientSecret (top-level and nested under req.body).

The path list is locked by an architectural source-scan test (no-pino-secrets-in-logs.test.ts). Adding a credential-shaped path requires updating both the Fastify logger options AND the test in the same commit. The redaction is unconditional — even with OTEL_LOGS_ENABLED=false, stdout aggregators (Datadog Agent, Loki Promtail, Vector, etc.) benefit.

Ownership boundary — `system_log` vs OTLP¶

system_log (DB) and the System Logs UI remain Floh-internal. Operators with admin access read structured workflow / step / connector logs there. LogService.log() writes here.
OTLP logs are operator-side only. Phase 4 does NOT bridge LogService calls. The reasoning is threat-model alignment: system_log rows carry Floh-internal IDs (workflow IDs, run IDs, step IDs) that the UI gates with its own access controls; exporting them to an operator's collector would propagate those IDs past the UI's boundary. If operators ask for a LogService → OTLP exporter in the future, it would land as a separate story with its own design + access-control review.

What's NOT bridged¶

Worker console.* calls. The Floh worker process uses console.log / console.warn / console.error (not pino). Phase 4 hooks pino only; bridging the worker is a separate follow-up story that would require migrating the worker to pino first.
LogService / system_log writes (see ownership boundary above).
Anything outside the Fastify HTTP process (CLI scripts, migrations, cron-style scripts).

Operator deploy-day note¶

Flip OTEL_LOGS_ENABLED=true in your environment.
Restart the Floh server.
Send a few HTTP requests; verify a corresponding log record appears in your OTLP backend with a trace_id that matches the trace span for the same request.
If records do not appear: check your collector's logs pipeline is configured, the OTLP endpoint accepts /v1/logs, and the OTEL_EXPORTER_OTLP_HEADERS (if any) include any required auth.
Setting OTEL_LOGS_ENABLED=false and restarting reverts to Phase 3c-identical behavior (no LoggerProvider, no transport).

Known footgun — `OTEL_LOGS_EXPORTER`¶

Do NOT set the standalone OTel SDK env var OTEL_LOGS_EXPORTER=otlp in combination with OTEL_LOGS_ENABLED=true. The two flags wire two independent log pipelines: OTEL_LOGS_ENABLED=true activates Floh's worker-thread pino transport (the canonical Phase 4 path); OTEL_LOGS_EXPORTER=otlp activates NodeSDK's own env-driven LoggerProvider on the main thread. With both set, every pino line double-emits — once via the worker-thread transport (correctly formatted with trace_id + reqId + redact.paths applied) and once via the main-thread SDK fallback (raw pino object, BYPASSES redact.paths).

For the same reason, do NOT set OTEL_LOGS_EXPORTER=otlp when OTEL_LOGS_ENABLED=false: the SDK env-fallback would activate a LoggerProvider even though you explicitly disabled the bridge. Either both flags off (no log export), or OTEL_LOGS_ENABLED=true

OTEL_LOGS_EXPORTER unset (canonical Phase 4 pipeline).

Backpressure / resource notes¶

The pino transport runs in a worker thread (default behavior of pino-opentelemetry-transport@1.1.x). Backpressure from the OTLP collector cannot block the main Node event loop.
The BatchLogRecordProcessor queue caps at the SDK default (2048 records) and drops oldest on overflow. Operators running very high request volume with a slow collector should size the collector ingest accordingly.

Phase 4 invariants (locked by tests)¶

#	Invariant	Test
1	`OTEL_LOGS_ENABLED=false` (default) → no `LoggerProvider`, no transport — byte-identical to Phase 3c	`otel-sdk-logs-init.test.ts`, `pino-otel-transport.test.ts`
2	Every `request.log.*` line carries `trace_id` AND `reqId`	`app-pino-trace-id-binding.test.ts`
3	Two independent trace-id paths (child binding + transport active-span)	`pino-otel-transport.test.ts` + `app-pino-trace-id-binding.test.ts`
4	Redact paths cover all required credential substrings	`no-pino-secrets-in-logs.test.ts`
5	`OTEL_LOGS_ENABLED` fail-closed parse	`otel-config-logs.test.ts`
6	OTLP log exporter inherits the same Resource as traces + metrics (single source of truth: `resource-attributes.ts`)	`resource-attributes.test.ts`, `otel-sdk-logs-init.test.ts`, `pino-otel-transport.test.ts` (dual-pipeline parity assertion)
7	`OtelHandle.shutdown()` flushes pending log records via NodeSDK's cascade	`otel-sdk-logs-init.test.ts` (wires log-record processor into NodeSDK so `sdk.shutdown()` cascades)
8	Transport runs in worker thread (no main-loop backpressure)	Documented; relies on `pino-opentelemetry-transport` default + the `target` string literal in `pino-otel-transport.ts`

Epic LSA-8852 — complete¶

All four phases of the Floh OTel rollout are now shipped:

Phase	Surface	Story
1	OTel SDK + tracing skeleton + RequestContext alignment	LSA-8853
2	Domain metric inventory + cardinality budget + first 2 metrics	LSA-8854
3	Manual spans + BullMQ propagation + step / connector metrics + route emissions	LSA-8855 + LSA-8873
3c	Auth session counter + queue depth observable gauge	LSA-8874
4	pino → OTLP log bridge	LSA-8856

No further OTel work is currently scoped in the epic.