Observability

NodePilot provides metrics, traces and an observability API. OpenTelemetry is disabled by default and has to be configured explicitly.

Live-Ops: inspect short executions

Over the execution bars, the mouse wheel zooms the timeline around the pointer. This makes short runs wider and easier to click; narrow bars also accept clicks immediately beside them. Over the workflow list, the wheel continues to scroll vertically.

Zoom reaches a five-second detail view within the loaded 30- or 60-minute window. The selected time range stays fixed. Back to live view resets it; changing the time window also resets zoom.

Multiple rows for one workflow are reusable parallel tracks. An empty space is not an error status. Shared child workflows appear independently with their visible caller count; only groups belonging exclusively to one caller are indented beneath it. The ID beside the name identifies the active or most recently started run on that track. Each bar opens its own execution.

OpenTelemetry (opt-in)

OpenTelemetry is opt-in. The setup is in NodePilot.Telemetry — constants, options, PrometheusClient.

Prometheus scraping

OpenTelemetry:Exporters:PrometheusScrapeAllowAnonymous (default false) — /metrics is not reachable anonymously. Set it to true if the scraper should access it without authentication (a relaxation; set it deliberately).

Host-name redaction

OpenTelemetry:RedactHostnames (default true) keeps the host name out of the telemetry: service.instance.id is a process-stable random ID instead of hostname:pid, the resource attribute host.name is omitted, and the Serilog bridge suppresses the field. Anyone who needs host attribution in Tempo/Grafana sets the value to false deliberately — otherwise the attribution disappears after an upgrade without further notice.

The observability API

CLI: np observability summary|query|query-range.

SIEM logging

For structured log ingest into Elastic, Sentinel or Splunk, set Logging:Format=ecs-json. Details: SIEM logging.

Support diagnostics

For operations and ticket diagnosis, GET /api/diagnostics/support-log|support-log/download|support-events|support-events/export|app-log|app-log/download are available to admins. Details: Logging. Which log files exist, where they live and which one helps with which failure mode: Logs & diagnostics.

Metrics (excerpt)

• nodepilot.database.requests_rejected — requests the open database breaker terminated immediately with 503.
• nodepilot.database.outages — confirmed outage episodes.
• nodepilot.database.probe_cleanup_timeouts — aborted cleanup steps of the recovery probe.
• nodepilot.scheduler.triggers.admission_deferred — trigger fires whose admission was deferred because the database was unavailable; the source keeps the signal and retries it after recovery.
• nodepilot.audit_archive.hash_drift — audit archive drift.
• nodepilot_credential_crypto_calls{operation,result} — encrypt/decrypt × success/failure.
• nodepilot_credential_crypto_legacy_reads — decrypts from the legacy provider (migration window).

For monitoring: /healthz/ready is the traffic gate and returns 503 on Armed or Unavailable. /healthz/database always answers 200 and reports ok, armed or unavailable, including a rough cause. Each recovery produces exactly one DATABASE_RECOVERED audit entry; a trip audit cannot be written reliably during an outage.