Security hardening
A checklist for securing a production Lahijan host, its secrets, network exposure, accounts and backups.
Use this checklist when you put a Lahijan stack on the internet, and go through it again after every major change. Each item says what to do and why, based on how the shipped compose stack works.
Secrets and the environment file
- Replace every
CHANGEMEvalue indeployments/.env.prod. Generate secrets withopenssl rand, as the example file shows. Useopenssl rand -hex 32for database passwords (a/from base64 breaks the migration URL). - Restrict the file:
chmod 600 deployments/.env.prod, owned by root. Never commit it; it is ignored by git. - Store a copy of
.env.prodsomewhere separate from the backup archives. - Treat
LAHIJAN_AUTH_SECRETS_ENCRYPTIONKEYas permanent. It encrypts TOTP secrets, external identity tokens and stored credentials, and there is no rotation command. If it is lost or changed, that data cannot be decrypted. - Know that changing
LAHIJAN_AUTH_SIGNING_KEYsigns every user out and invalidates outstanding email links. Rotate it if you think it leaked, then recreatelahijan. - Keep
SEAWEEDFS_S3_ACCESS_KEYandSEAWEEDFS_S3_SECRET_KEYstable. The same values are used by theseaweed-iam-initjob and by Lahijan, and Lahijan re-publishes the S3 identity document at startup. There is no tested rotation procedure for them.
Lahijan reads secrets from environment variables once at startup. The *_FILE variants mentioned in some comments are not implemented.
Database passwords
Each service has its own PostgreSQL role (lahijan, pdns, seaweed) created on the first start. To change one later, change it in PostgreSQL first, then in .env.prod, then recreate the service that uses it:
NEW=$(openssl rand -hex 32)docker exec -it lahijan-prod-postgres psql -U postgres -d postgres \ -c "ALTER ROLE lahijan PASSWORD '$NEW';"# set LAHIJAN_DATABASE_PASSWORD=$NEW in deployments/.env.prod, then:docker compose --env-file deployments/.env.prod -f deployments/docker-compose.prod.yml up -d --force-recreate lahijanUse powerdns for the pdns role and seaweed-filer for the seaweed role. Replace postgres with your POSTGRES_SUPERUSER if you changed it.
The first admin account
- Sign in with the bootstrap admin and change the password immediately. The generated password is printed to the container log and stays there until the log rotates.
- Remove
LAHIJAN_BOOTSTRAP_ADMIN_PASSWORDfrom.env.prodafter the first start if you set it. The bootstrap only runs on a database without users. - Turn on two-factor authentication for every platform admin. See Sign-in security.
Accounts and roles
- Registration is always open: anyone who reaches the dashboard can create an account, and by default each new account owns a personal tenant (
auth.signup.personalTenant). Decide whether that fits your install. See Sign-in providers and email. - Review the seeded roles in Roles and permissions. In this release
tenant.ownerreceives every permission that does not start withplatform., which includescompute.ip_pool.manageandcompute.cluster.member.evacuate, andtenant.adminalso hascompute.cluster.member.evacuate. Tenant owners, including self-registered users in their personal tenants, therefore pass the permission check for the operator endpoints described in Compute cluster. - If you enable SAML, keep
auth.saml.jit.enabledoff unless you want unknown IdP users to get accounts automatically, and keepallowIdpInitiatedoff unless you need it.
TLS
- Keep TLS on in Caddy. Session cookies are
Securein the production file (LAHIJAN_AUTH_SESSION_SECURE=true), so the dashboard only works over HTTPS. - Serve S3 over HTTPS through a
conf.d/s3.caddysite rather than the plain HTTP port. See TLS and domains.
Network exposure
The compose file publishes only these ports on the host:
| Port | Needed when |
|---|---|
| 80/tcp, 443/tcp, 443/udp (Caddy) | Always |
| 53/udp, 53/tcp (PowerDNS) | You host public DNS zones |
| 8333/tcp (S3 gateway) | Clients use the published S3 port instead of an HTTPS S3 host |
PostgreSQL, the PowerDNS API, the SeaweedFS master, volume and filer, and Lahijan itself are only on the internal backend network.
- Allow only the ports you need in your host or cloud firewall, and SSH from your admin addresses.
- Do not rely on a host firewall such as
ufwto close a published Docker port. Docker's own iptables rules bypass it, and theincuscontainer insertsiptables -I DOCKER-USER -j ACCEPT(SETIPTABLES=true). To close a port, stop publishing it in a compose override (for example remove8333when you serve S3 through Caddy) or use your provider's network firewall. - Tighten
PDNS_WEBSERVER_ALLOW_FROMto the Dockerbackendnetwork instead of all private ranges. Use a long randomPDNS_API_KEY. - The Incus daemon runs with host networking. Lahijan only needs its Unix socket, and
deployments/incus/preseed.yamldoes not setcore.https_address. Check withincus config get core.https_addressinside theincuscontainer, and leave it empty unless you need remote Incus access.
Host access
- The
incuscontainer is privileged with host PID, network and cgroup namespaces and all of/dev. Anyone who can run Docker commands on the host has root on the host. Limit membership of thedockergroup. - Keep
/var/lib/incusreadable only by root.
Observability endpoints
- Replace the default basic-auth hashes for
/grafanaand/jaeger(GRAFANA_BASIC_AUTH_HASH,JAEGER_BASIC_AUTH_HASH). The fallback isadmin/admin. - Set a strong
GRAFANA_ADMIN_PASSWORD. - If you do not use Grafana or Jaeger from the internet, remove their routes from the Caddyfile.
Logs and audit
- There is no log redaction handler in this release. Treat container logs as sensitive and limit who can read them.
- Review the audit log regularly. Every privileged action is recorded, and rows cannot be changed or deleted through Lahijan. See Audit log.
- The River job UI at
/admin/jobs/uirequires theplatform.jobs.readpermission. See Background jobs.
Plugins
- Plugins run in the WASM sandbox, and an admin approves each permission a plugin asks for at install time. Read the requested permissions before approving, especially network access. See Installing plugins.
- If you do not use plugins, set
LAHIJAN_WASM_ENABLED=falseon thelahijanservice in a compose override.
Updates and backups
- Pin
LAHIJAN_IMAGE_TAGto a release and upgrade deliberately. See Upgrades and migrations. - Pull updated images regularly:
docker compose ... pullandup -d. Theincusservice follows theltstag unless you pinINCUS_IMAGE_TAG. - Keep the host OS and Docker patched.
- Run backups daily, copy them off the host, and test a restore. See Backups, including the warning about object data.