Operations

How to run Aletheia with traces, metrics and correlated logs, what the API, the worker and the app runner emit, how to follow one workflow run end to end, what the worker deletes once the deployment's retention has passed, and the operator's commands (pnpm ops). Containers and deployment are covered in Deployment.

Telemetry environment

The API, the worker and the app runner read the standard OpenTelemetry variables (packages/core/src/env.ts):

Variable	Default	Effect
OTEL_EXPORTER_OTLP_ENDPOINT	unset	OTLP/HTTP base URL (/v1/traces and /v1/metrics are appended). Unset: telemetry is off entirely.
OTEL_SDK_DISABLED	false	true keeps telemetry off even when an endpoint is set.
OTEL_SERVICE_NAME	aletheia-api / aletheia-worker / aletheia-app-runner	Overrides service.name.
OTEL_TRACES_SAMPLER	parentbased_always_on	One of always_on, always_off, traceidratio, parentbased_always_on, parentbased_always_off, parentbased_traceidratio.
OTEL_TRACES_SAMPLER_ARG	1	Ratio in [0, 1] for the traceidratio samplers.
OTEL_PG_STATEMENTS	false	Puts SQL text on database spans. Off by default: statements can carry personal data.
TRACE_URL_TEMPLATE	unset	API only: a link to one trace in your backend with {traceId} in it. A run stores the id of the sampled trace that started it, and its page links there for admins (ops:read).
METRICS_PORT	unset	Serves Prometheus /metrics on 0.0.0.0:<port>; the worker also binds Temporal core metrics on port + 1. See Metrics.
WORKER_HEALTH_PORT	unset	Worker only: GET /healthz on this port. See Worker health.
NODE_ENV	development	Becomes deployment.environment.name on the resource and env on log lines.
LOG_LEVEL	info	pino level; the worker forwards Temporal core logs at the same level.

service.version is the app's package.json version. Nothing is exported until an endpoint or METRICS_PORT is set, so unit tests and the default dev stack are unaffected.

Startup order matters: apps/api/src/main.ts and apps/worker/src/main.ts call bootstrap(), which loads the environment, creates the logger, calls startTelemetry and only then imports server.ts dynamically. The instrumentations patch http, fastify and undici as those modules load; a static import of the server would hoist them above the SDK and produce no spans.

ESM note: the apps are ES modules, and the package instrumentation (fastify) patches through require-in-the-middle, which an ESM import never passes through; only the http and undici built-ins are traced without help. So bootstrap(), when telemetry is on, also calls module.register('@opentelemetry/instrumentation/hook.mjs') (OpenTelemetry's import-in-the-middle loader hook) before importing server.ts; the hook rewrites every module loaded from then on so the instrumentations see them. Because the registration lives in bootstrap.ts rather than in a node --import flag, pnpm start (node dist/main.js), CI and pnpm dev (tsx watch) all get the same spans; @opentelemetry/instrumentation is a direct dependency of both apps so the hook resolves from their dist/. Without an OTLP endpoint nothing is registered and modules load exactly as before.

What is traced

Source	Spans
API requests (http, fastify)	One server span per request; /health* and /metrics are ignored. The auth hook adds the tenant and principal.
Temporal client calls (workflow-engine)	temporal.<operation> client spans (startRun, startBacktest, startDocumentScan, signal*, queryRunState) with the tenant, run and workflow ids. createTemporalClient adds OpenTelemetryWorkflowClientInterceptor, which nests StartWorkflow:<type> (and SignalWorkflow:*, QueryWorkflow:*) under them and injects the span into the workflow headers, so the worker's RunWorkflow:<type> continues the API request's trace.
Workflows and activities (worker)	@temporalio/interceptors-opentelemetry's OpenTelemetryPlugin: RunWorkflow:<type> in the sandbox, StartActivity:<name>/RunActivity:<name> on the worker, linked through the Temporal headers the client interceptor sets. Each activity body runs in its own activity.<name> span with aletheia.run_id, aletheia.step_id, the tenant, the workflow id and the definition key.
App calls (worker)	app.invoke per attempt with the app's name, the action, the attempt, the call id, the idempotency key and the outcome; the request to the runner nests below it.
The app runner	Its server span per call continues the caller's trace (service aletheia-app-runner), with a client span for each module and document it fetches from the API. A module's call runs untraced (apps/app-runner/src/tracing.ts): the requests the host sends to vendors reach no span and carry no trace context or baggage, since their URLs, the hostname included, can hold an install's secrets; only its model requests are counted, by provider and without content (aletheia.app.ai.*). The instances it makes ahead of calls are made untraced too, and a call's host functions run in that call's context.
Outbound webhooks (worker, API)	Nothing: a delivery and the test ping (POST /webhook-endpoints/:id/test) run untraced (untraced of @aletheia-dev/telemetry). An endpoint's URL may carry its token in the path or the query, so the request reaches no span and no metric (http.client.request.duration included) and the endpoint receives no trace context or baggage.
Rules (rule-engine)	One span per rule set evaluation and one per rule with key, type and outcome.
Postgres (@aletheia-dev/db)	db.query spans from the postgres.js wrapper in @aletheia-dev/db (operation, table, optional statement text with OTEL_PG_STATEMENTS=1); db.transaction around each scoped repository call.

The worker passes the interceptors' Temporal plugin to every Worker.create only when telemetry is enabled, so a worker without an endpoint runs exactly as before. Workflow spans leave the sandbox through a sink built in @aletheia-dev/telemetry (workflowSpanSink); apps/worker/src/temporal-runtime.ts bridges the Temporal package's SDK 1.x span shape to the 2.x exporter.

Attribute glossary (aletheia.*)

Attribute	Set by	Meaning
aletheia.request_id	API request-id hook	The x-request-id of the request (generated or caller-supplied).
aletheia.tenant_id	API auth hook, activities	Tenant the span belongs to. On spans only, never on the resource.
aletheia.principal.kind	API auth hook	human, machine or applicant.
aletheia.principal.id	API auth hook	Subject of humans and machines; applicants carry a sha256 prefix of the submission id.
aletheia.run_id	activities	Aletheia workflow run id (search key for one run).
aletheia.step_id	activities	Step of the definition the activity executes.
aletheia.workflow_id	activities	Temporal workflow id.
aletheia.definition.key	activities	Workflow definition key.
aletheia.app.name, .install, .action, .attempt, .call_id, .outcome, .idempotency_key	app invoker (worker)	One app.invoke span per attempt: name is the catalogue app, install the install the call went through, by its current name (a call through a former name too); the call id is the invocation record's id.
aletheia.rule_set.key, .band, aletheia.rule.count, aletheia.rules.outcome	rule engine	Per rule set evaluation.
aletheia.rule.key, .type, .outcome	rule engine	Per rule.

Temporal's own interceptors add run_id, temporalWorkflowId and temporalActivityId.

Logs and correlation

Output is pino JSON (pino-pretty outside production). Every line carries service, version, env, pid and hostname; when a span is recording, the mixin adds trace_id and span_id, which is what links a log line to its trace in Grafana.

Field	Where	Notes
requestId	every API line of a request	Same value as the x-request-id response header and error.requestId in error bodies.
tenantId, principalKind	API lines after authentication	Bound on request.log by the auth hook; the request completed line always has them.
trace_id, span_id	any line inside a span	Hex ids of the active span.
taskQueue	worker start-up lines	One Temporal worker per queue.
tenantId, runId, stepId, workflowId	activity lines	Bound on the activity logger from its input.
tenantId, app, install, action, attempt	worker, app invoker	One line per attempt, app invocation <status>: warn for an error or a timeout. app is the catalogue app, install the install.
tenantId, appName, installName, callId	app runner	Every line a module logs through log, the host's app http_request lines (debug; reused says whether the request went out on a connection the tenant's earlier request kept alive), and app log refused (warn, a call's first log request outside the contract, with its error).
component: temporal-sdk	worker	Temporal TypeScript and (forwarded) Rust core logs.

Request ids: the API accepts a caller's x-request-id when it is 1 to 128 characters of [A-Za-z0-9._-] and generates a UUID otherwise; the id is echoed on every response, 4xx and 5xx included, and repeated in every error body ({ "error": { "code", "message", "details"?, "requestId" } }), so a client can quote it in a support request. ApiError in @aletheia-dev/api-client exposes it as requestId.

Redaction

createLogger censors these keys at the top level and one level down ([redacted]): authorization (also under req.headers and headers), token, secrets, apiKey, secretKey, webhookSecret, password, submissionUrl, link. Deeper paths need an explicit entry (redact option); log secrets as a top-level key or not at all.

Sampling

The default is parentbased_always_on: everything is recorded, and a span joins its parent's decision. For production set

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

so 10 % of API requests start a trace and the workflow, activity, app and rule spans of those requests follow the parent decision, keeping each sampled trace complete. Errors are not force-sampled yet: a failed request outside the 10 % leaves logs (with requestId) but no trace. Tail-based sampling for errors belongs in the collector (an item for docs/plans/deferred.md).

Local stack: Grafana LGTM

pnpm infra:observability          # grafana/otel-lgtm behind the `observability` compose profile
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
pnpm dev                          # or start the API and worker however you normally do
open http://localhost:3000        # Grafana (no login)

See docker/otel/README.md for ports and volumes. To follow one run end to end:

1. Take the run id from POST /workflow-runs (or the runId on a worker log line).
2. In Grafana choose Explore, data source Tempo, query type Search, and filter on the span attribute aletheia.run_id = the id. Service aletheia-api shows the request that started the run; aletheia-worker the RunWorkflow, RunActivity, app.invoke and rule spans, and aletheia-app-runner the calls it ran for them.
3. Open the API trace: the root server span carries aletheia.request_id, the tenant and the principal, and the worker's RunWorkflow span continues the same trace under StartWorkflow (see What is traced).
4. A vendor callback is its own trace (it is a new request to POST /webhooks/app-installs/<install id>); the session it completed, with its external id and status, is in GET /app-callbacks?workflowRunId=<run>.
5. For logs, copy trace_id from any span into Loki ({service="aletheia-worker"} |= "<trace_id>") or search by requestId for the API request.

CI: the OTLP sink

The e2e job does not run the LGTM image. It starts scripts/otlp-sink.mjs (an HTTP server that answers 200 {} to every POST and appends <time> POST <path> to a log file), points the API and the worker at it with OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4319, and sets SMOKE_OTLP_SINK_LOG=otlp.log. Smoke step 21 (scripts/smoke-operations.sh) checks the x-request-id behaviour and then polls that file for up to 20 s for a /v1/traces line (the batch processor exports every 5 s). Without SMOKE_OTLP_SINK_LOG the export assertion is skipped with a warning, so pnpm smoke against a stack without telemetry still passes.

To reproduce locally:

node scripts/otlp-sink.mjs 4319 /tmp/otlp.log &
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4319 pnpm --filter @aletheia-dev/api start &
SMOKE_OTLP_SINK_LOG=/tmp/otlp.log pnpm smoke

Metrics

Every instrument is defined once in @aletheia-dev/telemetry (packages/telemetry/src/meters.ts: names in METRIC, instruments in meters) and recorded by the code path that owns it. The rule engine and the workflow engine do not depend on that package (it carries the SDK), so each mirrors the few instruments it records with the same names, units and bucket advice (metrics.ts next to the recording site). Instruments are created lazily on whichever meter provider is global, so without startTelemetry every record is a no-op.

Instrument	Type	Unit	Labels	Recorded by
aletheia.rule.evaluation.duration	histogram	ms	tenant, rule_type, outcome	evaluateRule (rule engine), one point per rule; outcome pass/fail/error/skipped
aletheia.rule_set.evaluation.duration	histogram	ms	tenant, rule_set_key, band	the evaluateRules activity, rules plus aggregation; band min..max or none
aletheia.decision.count	counter	{decision}	tenant, outcome, source, definition_key	emitDecision (automated), POST /cases/:id/decide (manual)
aletheia.workflow.run.duration	histogram	s	tenant, definition_key, status	recordRunState when the run finishes (completed/failed/cancelled), from startedAt
aletheia.workflow.run.active	gauge	{run}	tenant, status	gauge collector (worker), non-terminal statuses
aletheia.app.invocation.duration	histogram	ms	tenant, app, install, action, status	the app invoker (worker), one point per attempt, the runner's round trip included; status success/error/timeout/pending
aletheia.app.callback.wait	histogram	s	tenant, app, action, result	the app webhook route (completed/failed), timeoutAppCallback (timed_out); first completion wins
aletheia.app.call.duration	histogram	ms	app, export, outcome, start	the app runner, one point per call of a module's export; outcome ok/pending/error/timeout/trapped/bad_output; start warm for an instance made ahead of the call, cold for one made for it, absent when none ran
aletheia.app.calls.in_flight	up-down counter	{call}	none	the app runner, calls running at the moment
aletheia.app.cache.operations	counter	{operation}	app, op	the app runner, per operation of a call on its install's cached values the host took: op hit or miss for a get that found a live value or none, set, delete; never a name or a value
aletheia.app.ai.requests	counter	{request}	app, provider, op, outcome	the app runner, per ai request of a call: op generate/decide (unknown for one the host could not read), outcome ok or the failure's kind (request/vendor/network/timeout/refusal/output/unsupported); never a model, a prompt or an answer
aletheia.app.ai.tokens	counter	{token}	app, provider, kind	the app runner, the tokens a call's model requests used as their vendors counted them, kind input/output
aletheia.app.ai.duration	histogram	ms	app, provider, op, outcome	the app runner, one point per ai request of a call, its vendor's answer included
aletheia.app.auto_upgrade.count	counter	{upgrade}	tenant, app, outcome	the API's automatic upgrade sweep, one per move of an install; outcome applied, or held for an approval
aletheia.app.fallback.count	counter	{call}	tenant, app, install, reason	the app invoker (worker), per call a fallback took over from the install named (install); reason circuit_open/outage
aletheia.app.quota.refused	counter	{call}	tenant, scope, metric, period	the app invoker (worker), per call of a run refused by an operator's quota (scope tenant) or a budget (install)
aletheia.app.webhooks	counter	{webhook}	tenant, app, result	the app webhook route, per webhook to a known session; refused/rejected/error/ignored/duplicate/accepted/stale/completed/failed
aletheia.app.circuit.open	gauge	{circuit}	tenant, state	gauge collector (worker), installs' circuits not closed; state open/half_open
aletheia.app.quota.used	gauge	1	tenant, metric, period	gauge collector (worker), the share of each of the operator's quotas its UTC period used
aletheia.case.open	gauge	{case}	tenant, priority, sla_state	gauge collector (worker), open and in_review cases
aletheia.case.time_to_decision	histogram	s	tenant, type	POST /cases/:id/decide, decidedAt - createdAt
aletheia.approval.pending	gauge	{approval}	tenant, kind	gauge collector (worker), undecided requests; for apps, kind app_version, app_install_change or app_version_review
aletheia.document.pending	gauge	{document}	tenant, status	gauge collector (worker), pending/uploaded/scanning
aletheia.document.processed.count	counter	{document}	tenant, status	processDocument, final status clean/infected/rejected
aletheia.audit.export.rows	counter	{row}	tenant, format	GET /audit-events/export after the last row is written
aletheia.retention.deleted	counter	{row}	tenant, table	the retention sweep (worker), per batch; table app_callbacks/app_webhook_events/submission_events
http.server.request.duration	histogram	s	HTTP semantic conventions	@opentelemetry/instrumentation-http (the API; /health* and /metrics excluded)

Buckets: millisecond histograms advise 1, 2, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000 (an app call may take two minutes); second histograms 1, 5, 15, 60, 300, 900, 3600, 21600, 86400. Keys and ids are never labels: a rule key is unbounded per tenant, so the per-rule histogram carries the rule type, and the dashboards drill into traces (aletheia.rule.key on rule.evaluate spans) for a specific rule.

App labels. app is the catalogue app, the module's name, so every install of one app counts under it. The worker's invocation histogram adds install, the install's name (a tenant has at most 50); the callback wait, the webhook counter and the runner's instruments carry app only, and the runner logs the install of each call. A rename moves an install's series, and the aletheia.app.install of its spans, to its new name, calls through a former name included, so dashboards and alerts keyed by the label follow the new name; the invocation records keep the name each call used, and their installId ties them to the install.

Tenant label. tenant is on every instrument but the app runner's (which serves every tenant from one process and labels by app): the product is multi-tenant by design and cardinality is bounded by the tenant count, which an operator controls. A deployment that runs one tenant per installation drops it in the dashboard with a recording rule or a sum without (tenant) view; the shipped Grafana dashboards aggregate across tenants by default and offer tenant as a variable.

OTLP and /metrics. With OTEL_EXPORTER_OTLP_ENDPOINT set, metrics are pushed over OTLP/HTTP every 15 s next to the traces. With METRICS_PORT set, the same meter provider also serves Prometheus text exposition on 0.0.0.0:<port>/metrics, with or without an endpoint, so a deployment without a collector scrapes directly. The two paths render names differently: the direct exporter turns dots into underscores, suffixes counters with _total and puts the unit on a # UNIT line only (aletheia_rule_evaluation_duration_bucket, aletheia_decision_count_total, aletheia_workflow_run_active, scope as otel_scope_name="aletheia"); the OTLP receivers of Prometheus, Mimir and the LGTM image add the unit to the name (aletheia_rule_evaluation_duration_milliseconds_bucket, aletheia_workflow_run_duration_seconds_bucket) and drop brace units. The dashboards target the OTLP spelling; smoke step 22 asserts the direct one. A metrics-only process (METRICS_PORT without an endpoint) registers no tracer provider: spans stay non-recording and nothing is sent.

The worker's two ports. The API and the worker are separate processes with their own environment, so each reads METRICS_PORT itself and the deployment sets different values (CI: API 9464, worker 9466). The worker binds a second server on METRICS_PORT + 1 for Temporal core's own metrics (temporal_*: task slots, poll latency, workflow task failures) through Runtime.install({ telemetryOptions: { metrics: { prometheus } } }); core's OTLP option is gRPC only, so it is not wired to the OTLP/HTTP endpoint. Both ports are logged at startup.

Gauge collector. The four observable gauges are grouped counts from MetricsRepo (packages/db/src/repos/metrics.ts) over the worker's single-connection owner handle (the app role sees no rows across tenants). registerGaugeCollector runs the four queries on collection, never per request, and caches the result for 30 s, so a process exporting over OTLP every 15 s and being scraped at the same time still issues one query set per 30 s. A failing query logs a warning and leaves the gauges out of that export rather than reporting stale counts. The gauges are only registered when telemetry is on.

Worker health. With WORKER_HEALTH_PORT set, the worker answers GET /healthz with { status, workers: [{ taskQueue, state }], temporal }: 200 when every Temporal worker is RUNNING, 503 otherwise (status and temporal are then unavailable; a running worker holds a live connection and polls, which is the connectivity signal the SDK exposes). The server closes with the workers. The worker image's health check and the chart's probes call it.

App runner health. The runner answers GET /health and GET /health/ready with { status, version, inFlight, concurrency, cachedModules, warmPlugins }: the calls running, the process limit, the compiled modules it holds and the instances it made ahead of calls that wait for one (APP_RUNNER_WARM_PLUGINS). An instance serves one call and is discarded; after each call of a module the runner makes it another, up to two per module within the budget, so warmPlugins sits at 0 until modules are called and falls back to 0 once they go uncalled for APP_RUNNER_WARM_IDLE_MS. A module whose instance could not be made, its start code failing or running past 5 s, is logged as app instance not prepared (warn, with the app and the module's hash) and gets no more until one of its calls runs on an instance made for it and answers. ready answers 503 when the cache directory cannot be written or the process is at its limit. What each call used comes back in its usage, kept on the invocation record: reusedConnections counts the requests that went out on a connection the same tenant's earlier request to that host kept alive (APP_RUNNER_KEEPALIVE_MS), so a vendor answering slowly on new connections shows as calls whose requests all opened their own.

To reproduce smoke step 22 locally:

METRICS_PORT=9466 WORKER_HEALTH_PORT=9470 pnpm --filter @aletheia-dev/worker start &
METRICS_PORT=9464 pnpm --filter @aletheia-dev/api start &
SMOKE_METRICS_API=http://localhost:9464 SMOKE_METRICS_WORKER=http://localhost:9466 \
SMOKE_METRICS_WORKER_CORE=http://localhost:9467 SMOKE_WORKER_HEALTH=http://localhost:9470 pnpm smoke
curl -s localhost:9464/metrics | grep '^aletheia_'

Without the SMOKE_* variables the step skips its assertions with a warning.

Retention

Every worker deletes, pass after pass, what the deployment keeps no longer, and times out the vendor sessions no run waits for any more. The settings hold for the whole deployment (Environment variables):

Data	Deleted once it was	Setting (default)
Settled vendor sessions of asynchronous app actions (app_callbacks)	created more than the setting plus 7 days ago: a session waits at most 7 days, so it is kept that long after it could last settle	RETENTION_APP_CALLBACK_DAYS (90)
The webhook events received for them (app_webhook_events), kept so a vendor's replay counts once	received more than the same setting plus 7 days ago	RETENTION_APP_CALLBACK_DAYS (90)
The drop-off events the collection terminal reports (submission_events), which the flow funnels read	reported more than the setting ago	RETENTION_SUBMISSION_EVENT_DAYS (90)

0 keeps that data forever. Otherwise the drop-off events are kept at least 90 days, the longest window GET /stats/flows/:key/funnel reads, so a funnel never reads part of its window. A session still pending a day after the longest wait (8 days after it started) has no run waiting for it any more: its run ended outside the platform, terminated in Temporal. Whatever the retention, the sweep times it out (timed_out, error orphaned), so it no longer holds its install, whose removal answers 409 install_in_use while sessions wait; it then ages out like the others. A run cancelled through the platform fails its waiting sessions at once (run_cancelled).

Once deleted, a session is gone from the run page, the install page and GET /app-callbacks (the invocations of the call stay), and a webhook for it answers 404 not_found, like one for an unknown id.

The API reports the two retention settings on the tenant profile (GET /tenants/me, retention), which the console shows under Settings › Tenant › Data & retention: give the API the same values as the workers.

How the sweep runs. A worker's first pass starts up to five minutes after the worker, so workers started together do not sweep together, and each next pass RETENTION_INTERVAL_MINUTES (60) after the last one ended. A pass goes tenant by tenant, oldest rows first, in batches of 5,000 rows, each in a short transaction of its own that takes its rows FOR UPDATE SKIP LOCKED: workers sweeping at once share the work and never wait on each other, and a row a request holds is left for the next pass. After 50 batches of one table for one tenant (250,000 rows) the pass moves on and the next one carries on with the rest, so a large backlog is spread over several passes. The sweep holds an owner connection of its own (DATABASE_URL) in every worker, and its statements are not traced.

Each pass logs retention sweep done at info with the tenants it swept, the rows it deleted per table (deleted), the sessions it timed out (orphaned), the tenant and table pairs it left rows in for the next pass (capped) and those that failed (failed). A table that fails logs retention sweep of a table failed; the next pass tries again at error with the tenant; timed-out sessions log app sessions no run waits for timed out at warn with the tenant and their count. aletheia.retention.deleted counts the deleted rows by tenant and table.

Operator commands

The deployment's operator acts through pnpm ops, on the deployment's own database, never through a tenant's API: there are no instance administrators. It lists every tenant's apps, blocks modules, publishes platform versions outside a release, reviews the versions of the tenants under review, caps tenants' app calls and reads what they used. A command that changes something names its operator with --by <name> (recorded as given, 1 to 100 characters: nothing checks it) and records what it did in the instance's own trail (operator_audit_events) in the transaction of the change. No tenant reads that trail; the tenants an action concerns find it in their own audit trail too, as actor type operator.

pnpm ops --help                    # every command
pnpm ops apps block --help         # one command and what it does
pnpm -s ops apps list --json       # -s keeps pnpm's banner out of what a script reads

--json prints what a command lists or did as JSON. The exit code is 0 when it is done, 1 when it failed (the error goes to standard error with its code: app_blocked, conflict, not_found) and 2 on a usage error, which is reported with the command's usage before the database is opened. What a tenant wrote, such as its name or a manifest's config pointers, prints with any control or bidirectional character written out (\u001b), so it cannot rewrite or reorder what the terminal shows.

Where it runs. The commands use the API's environment and code: the owner connection (DATABASE_URL) and, for platform publish, the object storage (STORAGE_*) and the app runner (APP_RUNNER_URL, APP_RUNNER_TOKEN). They refuse a database whose schema is behind their migrations. From a checkout, pnpm ops reads .env as pnpm tenant:add does, and a path is relative to the repository's root. In a deployment, the API's image (and the migrate image, built from it) carries the same commands as node dist/ops.js, where the credentials already are; - reads a module from standard input, so it never needs copying into the container:

kubectl -n aletheia exec deployment/aletheia-api -- node dist/ops.js apps blocks
kubectl -n aletheia exec -i deployment/aletheia-api -- \
  node dist/ops.js platform publish - --by alice < catalog/http-api/dist/app.wasm
docker compose --env-file .env -f docker-compose.prod.yml exec -T api node dist/ops.js apps list

Command	What it does
apps list	Every app version of the instance with its publisher (a tenant or the platform), status, hash, the hosts it calls, what it needs, its latest review, the block that stops it, its provenance under the trust policy, and every install that runs it with the hosts the install approved; never a config, a secret or a call. --tenant <id or name>, --platform, --app and --status narrow it, and --unsigned, --untrusted or --signer <name> by provenance (Module provenance).
apps blocks, apps block, apps unblock	The blocks, and blocking or unblocking a module: Blocking a module.
apps reviews, apps review approve, apps review reject, tenants review	The operator's review of tenants' versions: Reviewing tenants' versions.
apps versions withdraw <versionId> --reason <text>	Withdraws a tenant's version, whatever its status: it can no longer be installed or published, its installs keep running it, its open approval and pending review close as withdrawn, and the tenant's trail records app.withdrawn with the reason and via: operator.
platform list, platform publish, platform withdraw, platform restore	The platform's versions: Platform versions outside a release; platform publish --bundle records a signature with one (Module provenance).
tenants quota, tenants usage	A tenant's quotas of app calls, and what tenants used of apps over a UTC month: App quotas and use.
audit	The instance's trail, newest first: --since <date>, --action (an action or its start, app.module.), --tenant, --limit (50 by default, at most 500).

Reviewing tenants' versions

A tenant under review publishes none of its own versions directly: once its own approval is given (when the tenant requires one), a publish opens the operator's review, answered 202 with the review, and the version stays uploaded, shown to the tenant as waiting for the operator's review. A new tenant starts with APP_REVIEW_DEFAULT (off or required, Environment variables), whether signup or pnpm tenant:add created it (pnpm tenant:add --app-review off|required names it for the tenant it creates); ops tenants review <tenant> on|off --by <name> sets one tenant's. Turning it off closes the reviews pending as withdrawn, so the tenant can publish those versions itself. Platform versions are never reviewed.

pnpm ops apps reviews                                  # oldest first, with what each asks for
pnpm ops apps review approve <versionId> --by alice    # publishes it
pnpm ops apps review reject <versionId> --by alice --comment 'Calls a host it does not declare'

Each pending version shows its provenance under the trust policy: who signed it and whether the policy trusts that signer (Module provenance). Approving publishes the version in the same transaction, unless it was blocked or is no longer uploaded meanwhile, or, under APP_SIGNATURES=required, the policy no longer trusts who signed it (provenance_required). Rejecting leaves it uploaded, so the tenant can fix it and publish again, and needs a comment, which the tenant reads. Either way the tenant's trail records app.review.decided, which raises the webhook event of that name and the app_review_decided notification to holders of apps:publish, and an approval also app.published with via: review. Reviews waiting count in aletheia.approval.pending as kind app_version_review, and the alert App reviews waiting fires after ten minutes.

Platform versions outside a release

ops platform publish <module.wasm | app directory | -> --by <name> publishes a platform version without a release, for a fix that cannot wait for one. The module passes the checks of a tenant's upload (parsed, then its manifest read by the app runner) and must carry the memory maximum its manifest names, as an app's build writes it (dist/app.wasm of a catalog/<name> package, which a directory argument stands for): nothing rewrites the bytes the operator publishes. It is refused when a block stops it or the platform has that app and version already, in any status. Only apps the platform has get new versions this way: a new platform app comes with a release, and --new-app publishes one anyway, unless a tenant has an app of that name. The module is stored under its hash and the version published as the operator's (platformSource operator, created by operator:<name>); the command lists the installs of the app's other versions, whose tenants move them to it.

A release leaves the operator's versions to the operator: its start-up never withdraws one, nor publishes again one the operator withdrew. A release whose catalogue carries an operator's version with the same manifest adopts it, with the operator's module; with another manifest the operator's version stays and the release's build of it is skipped, logged at error as catalogue: the operator published this version with another manifest and counted in conflicts of the catalogue reconciled line: the app needs a new version number.

ops platform withdraw <name>@<version> withdraws a published platform version, the release's or the operator's, for good: its installs keep running it, nothing can install it, and no start-up publishes it again. ops platform restore <name>@<version> publishes a withdrawn one again, unless a block stops it: the release's when the running release's catalogue (APP_CATALOG_DIR) carries it with the same manifest, otherwise the operator's. ops platform list lists the platform's versions with their source, status and installs.

A block does not keep a release from publishing the versions its catalogue carries: a new version of a blocked platform app is published at start-up and listed to the tenants, and nothing of it runs until the block is lifted (Blocking a module). Withdraw it with ops platform withdraw if the tenants should not see it meanwhile.

Module provenance

A tenant can attach a Sigstore bundle to a version it uploads, a signature over the module or an in-toto attestation that names it, as cosign and GitHub's artifact attestations write them (PUT /apps/:name/versions/:id/provenance, Apps: authoring). The API verifies it against the module it stores and records what it proves: the signer (a keyless certificate's identity and OIDC issuer, or a pinned key's fingerprint), the source the certificate or the attestation names, the transparency log entry and when it was signed. The bundle is not kept, and neither is whether the signer is trusted: the API and pnpm ops compute it against the trust policy, as the process read it at start, wherever a version is shown, published, approved, installed or upgraded to.

The policy is in the environment (Environment variables):

• APP_SIGNATURES: off, the default, takes no bundle and reaches no Sigstore service. optional verifies and shows what a tenant attaches. required publishes, approves, installs and upgrades to a tenant's version only once a signer the policy trusts has signed it, and answers 409 provenance_required otherwise, before any approval or review opens. Platform versions are never gated.
• APP_TRUSTED_SIGNERS: the signers, a JSON array. A keyless entry names the certificate's OIDC issuer exactly and a glob over its subject alternative name, in which * stands for any run of characters within a path segment and ** for any run across them, every other character for itself: pin the organisation, the repository, the workflow and the tag ref. A key is an ECDSA P-256 public key in PEM, as cosign generate-key-pair writes cosign.pub. Each entry's name is what a trusted version shows.
• APP_SIGSTORE_TRUSTED_ROOTS: what keyless signatures, and the log entries and timestamps of any bundle, verify against. public-good, the default, is Sigstore's public instance: the API fetches its trusted root over TUF from tuf-repo-cdn.sigstore.dev when a bundle first needs it, and again after an hour, caching the metadata in APP_SIGSTORE_TUF_CACHE (under /tmp by default), which serves while it is unexpired when the CDN is out of reach. The cache holds the root that metadata is checked against: the API makes a missing one a directory only its user may use, and refuses one that is a symbolic link, belongs to another user or others may write (public-good is then unavailable). Where the API runs outside a container on a host other users share, give it a directory of its own. Absolute paths of trusted_root.json files add other instances, or replace public-good where the API has no egress: GitHub's, which signs the attestations of private repositories (gh attestation trusted-root prints its root and public-good's, one per line, and a file may hold several that way), a private Sigstore, or a copy of public-good's root, which goes stale when Sigstore rotates its keys. Under a root that lists transparency and CT logs, a signature needs a log entry and a certificate timestamp (SCT); under every root, a timestamp.

[
  {
    "name": "acme-release",
    "issuer": "https://token.actions.githubusercontent.com",
    "identity": "https://github.com/acme/screening-app/.github/workflows/release.yml@refs/tags/v*"
  },
  {
    "name": "acme-key",
    "publicKey": "-----BEGIN PUBLIC KEY-----\nMFkw...\n-----END PUBLIC KEY-----\n"
  }
]

How to read what the commands and the console show:

• A signature the policy does not trust is recorded all the same (trusted: false, trustedSigner: null; signature, not trusted by the policy in ops apps list): the module verifies, and a signer the policy does not name signed it. Add the signer to the policy, or ask the tenant to sign with one it names.
• Removing a signer from APP_TRUSTED_SIGNERS and restarting revokes it everywhere: the versions it signed show as not trusted, and under required they no longer publish, install or upgrade to, while their installs keep running. Block a module that must stop running (Blocking a module).
• A publish request or the operator's review fixes a version's provenance while it is open (409 provenance_locked), so an approver decides on the signer they see; under required the approval checks the policy again.
• 503 provenance_unavailable: a trusted root could not be loaded, such as the TUF CDN out of reach with no unexpired cache, or a trusted_root.json the API cannot read. The API logs sigstore: the TUF repository is out of reach at warn.

Before switching to required, list the tenants' versions it would stop, and block what must stop running:

pnpm ops apps list --untrusted --status published   # published; no signer of the policy signed them
pnpm ops apps list --signer acme-release            # what one signer of the policy signed

ops platform publish <module> --bundle <file> --by <name> takes a bundle with a platform version the operator publishes, verified the same way, refused unless it proves the module, and recorded with the version, trusted or not; the deployment must take provenance (APP_SIGNATURES other than off). A platform version a release built records the release's source instead (release: the commit the image was built from, and its repository when the build names it), trusted as the deployment trusts the image (Releasing); a build without REVISION records none. scripts/sign-module.mjs signs with a development key committed for the tests, which the isolated e2e pins as smoke-dev (node scripts/sign-module.mjs --trusted-signers prints its entry): never pin it in a deployment.

App quotas and use

A tenant's app calls are capped in two layers, both weighed before each call goes out:

• The operator's quotas of a tenant: the most calls of actions (calls), or HTTP requests those calls make (http_requests), per UTC day or month, over all the tenant's installs. Only the operator sets them, with ops tenants quota: the tenant reads them (GET /apps/metering, the Use panel of Connect › Apps) and cannot change them, since the API's database role cannot write them.
• An install's budgets, which the tenant's administrators set (apps:write: PUT /app-installs/<name>/limits, or Edit limits… under Budgets and use on the install's page): its calls, HTTP requests or spend per UTC day or month, the spend at the price of each action they enter, in the install's currency, which changes only from a month that spent nothing.

pnpm ops tenants quota acme calls day 10000 --by alice           # at most 10000 calls a UTC day
pnpm ops tenants quota acme http_requests month 500000 --by alice
pnpm ops tenants quota acme calls day none --by alice            # removes that quota
pnpm ops tenants usage acme                    # this month: its installs, quotas and budgets
pnpm ops tenants usage --month 2026-09         # every tenant's month, and every tenant's quotas
pnpm -s ops tenants usage --month 2026-09 --json > usage-2026-09.json   # for billing

What counts. Every attempt of an action that reached the app runner is a call of its install, with the HTTP requests it made: a run's steps and app rules, each retry an attempt of its own, and the test calls of an install's page. A call the platform refused before the runner (invalid input, a disabled install, a blocked module, an open circuit, a quota) or that ran nothing (a busy runner, a call no runner took, a module the runner could not load) counts for nothing; a call that timed out, trapped or answered outside its contract counts, with the HTTP requests the runner counted before it failed (none when the runner never answered). A call that succeeded or started its asynchronous work spends its action's price, kept in the currency it was charged in. The vendors' webhooks the installs' modules handled and the applicants they handed off are counted apart, for billing, and never limited: a vendor's retries or a replayed webhook cannot use up a tenant's calls. A day starts at 00:00 UTC and a month on its first day; what each install used is kept per day and month, a removed install's under the name its last call used.

What a refusal does. A call goes out while every quota and budget it meets has some left, and a priced call while its price fits its spend budgets; the operator's quotas are weighed first. The check is soft: each call reads its period's use as it stands, so calls in flight on several workers can pass a quota by as many as run at once. A call refused fails at once with 429 app_quota_exceeded, naming the quota or budget (scope tenant or install), its metric, period, max, what was used and when the period ends (resetsAt): a step fails without a retry, an app rule ends in error, a test call answers 429. A refused call is recorded nowhere, counts for nothing, never counts against the install's circuit and never goes to a fallback. The first refusal of each quota and period, and of each install's budget and period, appends app.quota.exceeded to the tenant's audit trail against the install, which raises the webhook event of that name and the app_quota_exceeded notification to holders of apps:write. aletheia.app.quota.refused counts the refusals of runs' calls.

Setting a quota.ops tenants quota <tenant> calls|http_requests day|month <max>|none --by <name> sets one of a tenant's quotas (the tenant by its id, or by a name only one tenant has), or removes it with none; it applies from the tenant's next call. A quota set below what its period used already refuses the tenant's calls at once, until the period ends, which the command says. The tenant's audit trail records tenant.app_quota.set or tenant.app_quota.removed with the metric, period and maximum, as actor type operator, and so does the instance's trail. The command prints the tenant's quotas with what their current periods used, what is left and when each resets.

Reading the use. ops tenants usage <tenant> prints one tenant's month, this UTC month unless --month <YYYY-MM> names another, as its administrators read it in GET /apps/metering: each install's calls, HTTP requests, webhooks and hand-offs, and its spend in the currency it was charged in (removed installs included), then the operator's quotas and the installs' budgets with what their current periods used and left; --json adds each day. Without a tenant it prints every tenant's month, one line per tenant, and every tenant's quotas; its --json lists each install's month with its tenant and currency, the rows of a billing export. Spend is what the tenant's own prices say its vendors charge, not anything the deployment bills.

Runbooks

The provisioned alert rules (deploy/grafana/provisioning/alerting/aletheia.yaml) link here. Each entry says what the alert measures, what to look at first, and the usual causes.

App failures

Fires when more than 5 % of the invocations through one install of a tenant end in error or timeout over ten minutes; the alert names the install and the app it runs. Open the Apps row of the Decisions dashboard for that tenant, then the traces for app.invoke spans with aletheia.app.outcome in error|timeout and aletheia.app.install the install. From the API, with a token of that tenant, GET /apps/health?windowMinutes=60 gives each install's calls, failures, timeouts, p95 duration and last failure over the window (5 to 1440 minutes), with the app it runs, the open circuits and the installs on a blocked module; GET /app-invocations?install=<name>&status=error&status=timeout lists the failed calls newest first, each with the version it ran on and what it used at the runner, and with the error text for a caller holding runs:context (an admin). Connect › Apps shows the same per install. Usual causes: a vendor outage or rate limit (every install of that vendor's app fails at once), a rotated secret (store the new value with POST /app-installs/<name>/secrets/<secret>), an upgrade to a version that calls the vendor differently (move the install back to the previous version), or the runner itself: a runner that is unreachable or out of call slots fails every install at once, which the runner's own aletheia.app.call.duration and aletheia.app.calls.in_flight show. A module the operator blocked (Blocking a module) fails every call with app_blocked, and GET /apps/health lists its installs under blocked with the reason. Through an install of http-api the error names the vendor's answer, HTTP <status> <METHOD> <path>: … with the start of its body: a 401 or 403 points at the install's credentials, another 4xx at the request a step or rule builds (its path, its variables, a status it should accept), and a 5xx, 408, 425 or 429 at the vendor (App catalogue). Disabling the install (PUT /app-installs/<name> with enabled: false) stops new invocations; runs already waiting on a callback keep waiting until its timeout, unless a forced removal (DELETE /app-installs/<name>?force=true) fails them at once. A vendor's webhooks are not invocations: aletheia.app.webhooks counts them by what came of each, so rejected or error for one app while its sessions stay pending points at the webhook's key, rotated at the vendor or stored in another encoding than the install's configuration says, and refused at a webhook secret never stored (409 install_not_configured) or a module the operator blocked (409 app_blocked).

An install stops calling its vendor by itself once the vendor's outages in a row reach its circuit's threshold, five unless set otherwise: its circuit opens, and calls through it fail at once with app_circuit_open, without reaching the runner, for its open time, 30 seconds unless set otherwise; then one trial call goes out, whose success closes the circuit and whose failure opens it again. The circuit is the install's, shared by every worker through the database: every worker's outages count toward it, it opens for all of them at once, and one worker of the fleet makes the trial (another install of the same app keeps calling). Only outages count: a timeout, and an error the app calls retryable (a vendor's 5xx, a 429, a failed connection). An error the app calls final (a 4xx from a misconfigured step), a module that traps, a busy or unreachable runner and a call a quota refuses do not open an install's circuit. GET /apps/health lists the open and half-open circuits as circuits, each with the threshold that applies to it, aletheia.app.circuit.open counts them by tenant and state, the audit trail has app.circuit.opened and app.circuit.closed, and Connect › Apps and Home name the install. Check the vendor with the install page's test call (POST /app-installs/<name>/test), which bypasses the circuit.

The deployment's threshold and open time are APP_CIRCUIT_FAILURE_THRESHOLD and APP_CIRCUIT_OPEN_SECONDS (Environment variables): give the API and the workers the same values. A tenant's administrator (apps:write) sets an install's own, the threshold 1 to 100 outages and the open time 5 to 3600 seconds, null for the deployment's (PUT /app-installs/<name>/circuit, or Settings… under Circuit breaker on the install's page), and closes an open circuit once the vendor is back, rather than wait for its trial (POST /app-installs/<name>/circuit/reset, or Reset circuit there). Both apply at once on every worker. The tenant's audit trail records app.circuit.configured with the settings before and after, and app.circuit.reset with the state and the count of outages it cleared; the API logs a reset as app circuit reset.

A workflow step or an app rule can name a fallback, another install with its action and input, reviewed and published with the definition. It takes the call when the install the step names refuses it with an open circuit, or once the vendor's outage outlasted the action's retries; never after a timeout or a failure of the runner on an action that is not idempotent, whose requests may have reached the vendor, nor after a refusal of the platform (a blocked module, invalid input, a quota) or an error the app calls final. The fallback's attempts are recorded under its own install, with fallbackOf naming the install the call was meant for; when the fallback fails too, the step or rule ends with the first install's error, the fallback's in its details.fallback. aletheia.app.fallback.count counts the calls fallbacks took over, by the install they were meant for and the reason (circuit_open, outage).

An install that upgrades automatically may have just moved: the audit trail has app.install.upgraded with fromVersionId, and the install's upgradedFromVersionId is the version it left. To roll one install back, an administrator saves it with that version and autoUpgrade: 'off' in one save (PUT /app-installs/<name>, or Move back on its page), which works even once that version is withdrawn. When the new version is bad for everyone, the operator withdraws it (pnpm ops platform withdraw) or blocks it: within one sweep (five minutes, or at the install's next save) every install that opted in moves back to the highest version of its range still published and unblocked (rollback: true), and no install moves onto it again. aletheia.app.auto_upgrade.count counts the moves by outcome.

Blocking a module

There is no alert rule: the operator blocks a module that must stop running everywhere (it leaks data, calls what it should not, breaks under a vendor's change), by its hash (that build, whoever uploaded it) or by its publisher's app (every version of it, those uploaded later too):

pnpm ops apps list --app acme-screening       # its versions, hashes and installs
pnpm ops apps block <sha256> --reason 'Leaks applicant data' --by alice
pnpm ops apps block --app acme-screening --publisher <tenant> \
  --reason 'Leaks applicant data' --by alice
pnpm ops apps blocks                          # every block and the installs it stops
pnpm ops apps unblock <sha256> --by alice --note 'Fixed in 0.2.1'

--publisher is platform or a tenant (its id or its name); it is needed only when several publishers have an app of that name. A hash stops one build: a re-upload of the app under a new version has another hash, so block the app to stop every version. The reason, at most 500 characters, is shown to every tenant that runs the module.

The block holds from its commit, with no restart: every place that runs a module reads the blocks as it does. Calls through the module (steps and rules, each attempt), test calls, installs and upgrades onto it, and uploads and publishes of it answer 409 app_blocked with the reason; the webhooks of its sessions answer 409 app_blocked without it, since nothing has checked who sent them yet; a hand-off finds none of its sessions (409 handoff_unavailable); and the runner can no longer fetch it (the API's module route answers 404, which the runner reports as module_not_found). Calls already on the runner finish, within the action's timeout (two minutes at most). Runs waiting on a session of the module are left to time out, unless the tenant removes the install with force=true; the vendor's webhooks and the applicant's hand-off for those sessions are refused at once.

Each install the block stops gets app.module.blocked in its tenant's audit trail, with the reason and actor type operator, which raises the webhook event of that name and the app_blocked notification to holders of apps:write; GET /apps/health lists the install under blocked with the reason and when it was blocked. A tenant whose install moved to another version while sessions of the blocked one still wait is not told. Unblocking writes app.module.unblocked, with the note, to the same tenants. Blocking what is blocked already changes nothing, and unblocking what no block names fails.

Calls refused by a quota

Fires when one of the operator's quotas of a tenant has had more than 80 % of its UTC day or month used for five minutes (aletheia.app.quota.used), before its calls are refused; the alert names the tenant, the metric and the period. A call that one of the operator's quotas or an install's budget refuses fails at once with 429 app_quota_exceeded, and the tenant is told once per limit and period (app.quota.exceeded, its webhook event and the app_quota_exceeded notification); aletheia.app.quota.refused counts the refusals of runs' calls by tenant, scope, metric and period. The error's details name the limit: scope tenant is one of the operator's quotas, install a budget of that install, which the tenant's administrators change themselves. pnpm ops tenants usage <tenant> shows both, with what their periods used and when each resets. Raise the tenant's quota, or remove it, with pnpm ops tenants quota (App quotas and use): it applies from the tenant's next call. Otherwise a quota reached refuses the tenant's calls until its UTC day or month ends; the steps and rules it refused meanwhile are not retried.

Webhook deliveries

There is no alert rule: a delivery given up after five attempts raises a notification for the tenant's admins and shows on Home under Needs your attention, and the worker logs it at warn as webhook delivery given up with the tenant, endpoint and delivery ids. Connect › Webhooks, or GET /webhook-deliveries?status=failed&status=exhausted, lists what failed with the response code or error; Send test ping checks the endpoint now. Usual causes: the receiver is down or answers non-2xx (its own logs say why), a name that resolves to a private address, a receiver that takes longer than 10 seconds, or a receiver still on the secret before a rotation more than 24 hours ago (it answers 401 or 403). Deliveries wait while every worker lacks SECRET_STORE_KEY (webhook deliveries paused at start-up) and stay queued for a disabled endpoint. The worker reads new events every two seconds and logs outbound event dispatch failed when it cannot; events are not lost while it fails, it carries on from where it stopped. Deliveries are not traced (an endpoint's URL may hold its token), so the delivery list and these log lines are their record.

Scheduled audit exports

There is no alert rule either: a failed run is recorded with its error, Home lists it for a day and the worker logs scheduled audit export failed at warn. Audit › Exports › Runs shows the error. Usual causes: object storage refusing the write (credentials, bucket, quota), a filter so wide its file passes 256 MiB (narrow it or schedule it more often), or the worker without STORAGE_ENDPOINT (the run never starts: the Temporal UI shows the auditExportSchedule workflow failing on an unregistered activity). The next run covers the failed run's window again.

Workflow task failures

Fires when the worker reports any workflow task failure in five minutes. These are not business failures: a workflow task fails when the interpreter throws outside an activity, which after a deploy almost always means a non-deterministic change replayed against an in-flight run. Check the worker log for DeterminismViolationError and the workflow:incompatible label on the released change; roll the worker back or let the affected runs be reset from the Temporal UI. Activity failures are a separate metric and retry on their own.

HTTP errors

Fires when more than 1 % of responses on one route are 5xx over five minutes. Find the route in the Operations dashboard, then search traces by http.route and status 500; the server span's aletheia.request_id matches the requestId in the API log line, which carries the error. Usual causes: the database or Temporal being unreachable (/health/ready reports which), or an object-storage error on document routes.

Rate limits

Each API instance gives every caller API_RATE_LIMIT_PER_MINUTE requests a minute (600 by default, 0 turns it off) and then answers 429 rate_limited with a Retry-After header and details.retryAfterSeconds; responses carry x-ratelimit-limit, x-ratelimit-remaining and x-ratelimit-reset. A caller is a tenant's user or service user, an applicant's submission (every tab and device on one link share it), or on the public routes (vendor webhooks, flow previews) the client address; health checks are not counted. The collection terminal waits as long as Retry-After says before it saves or polls again. The count is kept per instance, so with several replicas a caller's effective budget is the limit times the replicas it reaches. A caller that keeps hitting it is usually a polling loop: raise the limit only after checking the caller's request pattern in the API logs (principalKind, url).

Every API response carries Cache-Control: no-store unless the route sets its own: responses hold personal data and links that no browser or proxy cache should keep.

SLA breaches

Fires when the number of open cases in breached state for a tenant rises over fifteen minutes. This is a staffing signal, not a system fault: open the admin console's Operate › Cases on the "Breached SLA" queue, assign or close. If every tenant breaches at once, check that the worker is running (the SLA timer lives in the run) and that the aletheia.case.open gauge is fresh.

Stale approvals

Informational: a request has waited more than a day for approval, a definition version's, an app version's (kind app_version) or a save of an install that adds hosts (app_install_change). Open the admin console's Operate › Approvals, "Stale · over 1 day" view; the requester is in the alert labels. The operator's reviews (kind app_version_review) are left out: they alert on their own, sooner (App reviews waiting). A request nobody can approve usually means the tenant enabled four-eyes with a single admin; either add an approver or have the admin reject and re-request with approvals turned off. An install change keeps the four-eyes setting it was requested under, so turning four-eyes off does not unblock it: once four-eyes is off, save the install again, which withdraws the old request and opens one its requester may approve. Until a change is decided, the install serves on its approved hosts and config.

App reviews waiting

Fires when one of a tenant's app versions has waited ten minutes for the operator's review (aletheia.approval.pending with kind app_version_review); the alert names the tenant. The tenant cannot publish the version meanwhile, and only the operator decides it: list the reviews with pnpm ops apps reviews (oldest first, with what each version asks for: the hosts it calls, what it needs, its actions and its secrets) and approve or reject each, a rejection with the comment the tenant reads (Reviewing tenants' versions). When the tenant should no longer be reviewed, pnpm ops tenants review <tenant> off --by <name> closes its pending reviews as withdrawn, and the tenant publishes those versions itself.