Observability

This page covers the signals a production Lahijan stack gives you today: container logs, health checks and probe endpoints, and the observability services in the compose file. It also says plainly which parts of the telemetry pipeline are not wired up yet, so you do not spend time looking for data that is not there.

Logs

The Lahijan process writes its logs to standard output and standard error. They come from three places:

• Application messages through Go's log/slog default handler and GoFiber's logger, as text lines with a level (such as INFO, WARN or ERROR) and key/value details. The output is plain text, not JSON.
• Fatal startup errors through Go's log package, followed by the process exiting.
• One line per HTTP request from GoFiber's request logger, when http.server.logger.enabled is true (the default).

Read them with Docker:

docker compose --env-file deployments/.env.prod -f deployments/docker-compose.prod.yml logs -f lahijandocker compose --env-file deployments/.env.prod -f deployments/docker-compose.prod.yml logs --since 1h lahijan | grep -E "WARN|ERROR"

Every service uses the json-file log driver, rotated at 10 MB with three files kept, so older lines are lost. Ship container logs to a central store with your own log shipper if you need longer retention.

Set LAHIJAN_DEBUG=true (in a compose override) for more verbose GoFiber logging while you investigate a problem, and turn it off again afterwards.

Health checks

Probe endpoints

With http.server.healthcheck.enabled (on by default and in production), Lahijan answers two probe paths:

Both return 200 whenever the HTTP server is running. They do not check PostgreSQL, Incus, PowerDNS or SeaweedFS. Caddy forwards /healthcheck/*, so you can probe from outside:

curl -fsS https://cloud.example.com/healthcheck/liveness

An external probe of this URL checks DNS, the TLS certificate, Caddy and the Lahijan process in one request.

GET /health returns {"status":"ok","version":"..."}. Caddy does not forward it, so call it from inside the container (see Upgrades and migrations).

Container health

Every long-running service in the compose file has a health check, for example pg_isready for PostgreSQL, pdns_control rping for PowerDNS, incus list for Incus and the liveness path for Lahijan. See them with:

docker compose --env-file deployments/.env.prod -f deployments/docker-compose.prod.yml psdocker inspect --format '{{json .State.Health}}' lahijan-prod-app

Telemetry services in the stack

The compose file includes an OpenTelemetry collector and four backends. They are optional: they belong to the monitoring compose profile and do not start unless you enable it.

Enable the monitoring stack

Set COMPOSE_PROFILES in deployments/.env.prod:

COMPOSE_PROFILES=monitoring

Then start the new services:

docker compose --env-file deployments/.env.prod -f deployments/docker-compose.prod.yml up -d

To combine profiles, separate them with commas (COMPOSE_PROFILES=monitoring,dns-full). To turn monitoring off again, remove monitoring from the variable and stop the services by name: docker compose --env-file deployments/.env.prod -f deployments/docker-compose.prod.yml stop otel-collector jaeger loki prometheus grafana. Their volumes (prometheus_data, grafana_data) are kept.

With the profile off, Lahijan runs normally and uses less memory (the five services have limits that add up to about 2.2 GB), but there are no Grafana, Jaeger, Loki or Prometheus pages. Caddy still has the /grafana and /jaeger routes and answers them with 502 Bad Gateway. scripts/upgrade.sh and scripts/restore.sh only touch these services when the profile is enabled.

The services are:

Caddy protects /grafana and /jaeger with basic auth (GRAFANA_BASIC_AUTH_USER/GRAFANA_BASIC_AUTH_HASH and JAEGER_BASIC_AUTH_USER/JAEGER_BASIC_AUTH_HASH). The fallback is admin with the password admin, so set real hashes:

docker run --rm -it caddy:2.8-alpine caddy hash-password

Put the hashes in .env.prod in single quotes and recreate Caddy. Grafana then asks for its own login (GRAFANA_ADMIN_USER, GRAFANA_ADMIN_PASSWORD); sign-up is disabled. Comments in the compose file mention a LAHIJAN_OBSERVABILITY_PUBLIC switch, but nothing reads it; to remove these routes, edit the Caddyfile.

Current limits

Be aware of these gaps in the shipped release:

• The Lahijan binary creates trace spans and job metrics through the OpenTelemetry API, but it does not install an OpenTelemetry SDK or OTLP exporter. The OTEL_EXPORTER_OTLP_* variables on the lahijan service are not used, so no traces, metrics or logs from Lahijan reach the collector.
• Lahijan has no /metrics endpoint. The collector's Prometheus receiver targets host.docker.internal:3000, which only exists in the development setup.
• deployments/prometheus/prometheus.yml points Prometheus at otel-colector:8889, but the service is named otel-collector, so that scrape target shows as down.
• Grafana starts without data sources. Add them by hand under Connections > Data sources using http://prometheus:9090, http://loki:3100 and http://jaeger:16686.

Until this is wired up, container logs and health checks are your main signals.

What to alert on

These checks only use what the stack provides:

For security events (privilege changes, sign-in failures, deletions), use the audit log; see Audit log.