Monitoring

Each release ships an optional Prometheus with alert rules for a stopped service, renders that are not starting, an expiring license and more.

Turn on Prometheus

1. Add this line to .env in the installation directory:

   COMPOSE_PROFILES=monitoring

2. Run docker compose up -d.

3. The Prometheus UI listens only on the server’s own 127.0.0.1, port 9090. From your computer, open an SSH tunnel to the server, then browse to http://localhost:9090/alerts:

   ssh -L 9090:127.0.0.1:9090 you@pdf-server

The port and how long metrics are kept are monitoring settings. Prometheus’s memory limit is MONITORING_MEMORY_LIMIT.

Get notified

The bundled Prometheus shows alerts only in its UI. To receive them, add Alertmanager as below, or use your own monitoring stack. Put every file below in the installation directory, next to compose.yaml.

1. Copy the bundled Prometheus configuration out of its container:

   docker compose cp prometheus:/etc/prometheus/prometheus.yml prometheus.yml

2. Add this block to the end of your copy of prometheus.yml:

   alerting:
     alertmanagers:
       - static_configs:
           - targets: [alertmanager:9093]

3. Create alertmanager.yml to say where alerts go (email, chat, webhook). See Alertmanager’s configuration.

4. Add both to compose.override.yaml (create it if you do not have one). Pin the Alertmanager image to its current release, and mount any file alertmanager.yml reads, such as a password file.

   services:
     prometheus:
       volumes:
         - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
   
     alertmanager:
       image: prom/alertmanager:v0.34.1
       profiles: [monitoring]
       restart: unless-stopped
       volumes:
         - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
       mem_limit: 256m

5. Run docker compose up -d, then send a test alert and check that it arrives:

   docker compose exec alertmanager amtool alert add TypefoldTest \
     --alertmanager.url=http://127.0.0.1:9093

Caution

Your prometheus.yml replaces the bundled one. After each upgrade, compare it with the new release’s file and copy over anything new. To print the new file, use the prometheus image named in the new compose.yaml:

docker run --rm --entrypoint cat <the prometheus image in compose.yaml> /etc/prometheus/prometheus.yml

Use your own monitoring stack

If you already run Prometheus elsewhere:

1. Scrape /metrics on port 9464 of every api replica and port 8081 of every worker replica. Name the scrape jobs api and worker: the “down” alerts look for those names.

2. Copy the alert rules out of the prometheus image named in compose.yaml (the bundled Prometheus does not need to run), and load them into your Prometheus:

   docker run --rm --entrypoint cat <the prometheus image in compose.yaml> /etc/prometheus/alerts.yml > alerts.yml

Neither port is published outside the Compose network, so your Prometheus must join that network, or you publish the ports in compose.override.yaml.

Health checks

The bundled Compose file already health-checks the api and worker services. You need these endpoints only for a load balancer or orchestrator of your own. None needs a credential.

Endpoint	Service	Answers 200 when	Use it for
GET /health	api	PostgreSQL and Redis both answer within 2 seconds	Readiness and load balancer checks
GET /health/live	api	The process can answer HTTP	Restart (liveness) checks
GET /health	worker	It is ready to render and Redis is connected	The worker container’s own health check

Use /health/live, not /health, for restarts, or a database outage restarts every API replica at once. The public address serves only the API’s /health. The API’s /health/live (port 3000) and the worker’s /health (port 8081) are reachable only inside the Compose network; publish the ports in compose.override.yaml if your checks run elsewhere.

When the API cannot reach a dependency, /health answers 503 and names it (database or redis):

{ "status": "unavailable", "unavailable": ["database"] }

Alert rules

Alert	Severity	Fires when
TypefoldApiDown	critical	An api replica cannot be scraped for 2 minutes
TypefoldNoApi	critical	No api replica is up for 2 minutes
TypefoldWorkerDown	critical	A worker replica cannot be scraped for 2 minutes
TypefoldNoWorkers	critical	No worker replica is up for 2 minutes
TypefoldDependencyUnavailable	critical	The API cannot read PostgreSQL or Redis for 2 minutes
TypefoldRenderProcessCold	critical	A worker has not been ready to render for 5 minutes
TypefoldWarmUpFailing	warning	A worker’s render process failed to start 3 or more times in an hour
TypefoldApiErrorRate	warning	More than 5% of API requests return 5xx for 10 minutes
TypefoldApiSlow	warning	API p95 latency is above 2 seconds for 15 minutes, not counting requests that wait for a document
TypefoldQueueBacklog	warning	The next render to start has waited more than 5 minutes
TypefoldRendersStuckQueued	critical	A render has been queued for more than 15 minutes
TypefoldRendersStuckProcessing	warning	A render has been processing for more than 15 minutes
TypefoldRenderFailureRate	warning	More than 10% of render attempts fail for 15 minutes with a retryable error or a timeout
TypefoldRenderProcessRestarts	warning	A worker restarted its render process more than 10 times in 15 minutes after failures or timeouts
TypefoldBackgroundJobStale	warning	Render repair, file cleanup, workspace cleanup or webhook delivery has not succeeded for 15 minutes, or render expiry for 3 hours
TypefoldBackgroundJobNeverSucceeded	warning	A background job keeps failing and has not succeeded since the API started
TypefoldWorkspaceCleanupStuck	warning	A deleted workspace’s data has not been purged after an hour
TypefoldLicenseRefusingRenders	critical	The license has refused renders for 5 minutes
TypefoldLicenseCheckInFailing	warning	The installation has not reached its license server for about 2 days; rendering stops within 5 days
TypefoldLicenseExpiring	warning	The license expires within 14 days
TypefoldLicenseInGrace	warning	The license has expired and is in its grace period

What to do when an alert fires

Alert	First things to check
TypefoldApiDown, TypefoldWorkerDown, TypefoldNoApi, TypefoldNoWorkers	docker compose ps, then the logs of the service that is not running
TypefoldQueueBacklog	Workers are up; if they are all busy, add workers
TypefoldRendersStuckQueued	The api logs for Render queue operation failed.
TypefoldRenderProcessCold, TypefoldWarmUpFailing	The worker logs for Render process warm-up failed.
TypefoldBackgroundJobStale, TypefoldBackgroundJobNeverSucceeded	The api logs for the failing job’s error
TypefoldWorkspaceCleanupStuck	Storage permissions, Object Lock or legal holds that block deleting files
License alerts	When rendering stops

Logs

Read a service’s log with docker compose logs, such as docker compose logs api --since 1h. LOG_MAX_SIZE and LOG_MAX_FILE set how much Docker keeps. To send logs elsewhere, set logging for each service in compose.override.yaml.

Lines worth knowing:

Log line	Service	Meaning
Self-hosted setup required.	api	No platform administrator yet; the line carries the setupToken
Made ADMIN_EMAIL the first platform administrator.	api	First start with ADMIN_EMAIL; the line carries the address
DEPLOY_API_KEY created the deploy key.	api	The deploy key was created, or replaced after its value changed
Neither email nor Microsoft sign-in is configured	api	Only password accounts can sign in; see sign-in methods
Self-hosted license loaded.	api	The license state at startup
Self-hosted license state changed.	api	The license state changed after a check-in
License check-in failed	api	The installation could not reach its license server
Webhook secret encryption failed	api	Stored webhook secrets could not be encrypted with the current key
Signing key encryption failed	api	Stored signing certificate keys could not be encrypted with the current key
Render process warm-up failed.	worker	A worker could not get ready to render

Metrics

The API serves GET /metrics on port 9464 and the worker on port 8081. Neither is reachable from outside the Compose network. Both also export the standard Node.js process metrics (process_*, nodejs_*).

API metrics

Metric	Labels	Meaning
typefold_http_request_duration_seconds	method, route, status_code	Request durations. route is the route pattern, such as /v1/renders/:id, or unmatched.
typefold_background_job_runs_total	task, outcome	Background job runs. outcome is succeeded or failed.
typefold_background_job_last_success_timestamp_seconds	task	When each background job last succeeded on this replica.

The background jobs (task) are asset_cleanup, render_repair, render_retention, workspace_cleanup, webhook_dispatch, webhook_retention, webhook_secret_encryption, signing_key_encryption and license_check_in.

These gauges describe the whole installation, so every API replica reports the same value. Combine replicas with max(), not sum().

Metric	Labels	Meaning
typefold_monitoring_source_up	source	1 if the scrape could read the database or the queue (Redis)
typefold_queue_jobs	state	Render jobs in Redis by state: waiting, active, delayed, failed
typefold_queue_oldest_waiting_age_seconds		How long the next render to start has waited
typefold_renders_unfinished	state	Renders that are queued or processing
typefold_render_oldest_queued_age_seconds		Age of the oldest queued render
typefold_render_oldest_processing_age_seconds		Time since the longest-running processing render last changed
typefold_workspace_deletions_pending		Workspace cleanups not yet finished
typefold_workspace_deletion_oldest_age_seconds		Age of the oldest unfinished workspace cleanup
typefold_license_state	state	1 for the current license state
typefold_license_renew_by_timestamp_seconds		When rendering stops unless a license check-in succeeds (0 if unset)
typefold_license_expires_timestamp_seconds		When the license expires (0 if unset)

Worker metrics

Metric	Labels	Meaning
typefold_worker_job_duration_seconds	outcome	Render durations, by outcome (below)
typefold_worker_child_replacements_total	reason	Render process restarts: recycle, failure, timeout
typefold_worker_child_warm_up_failures_total		Render processes that failed to start
typefold_worker_child_warm		1 while the worker is ready to render

outcome is succeeded, ignored (cancelled or already finished), retryable (a failure that may succeed on retry), final (a failure retrying cannot fix, usually an invalid document or a limit) or timeout (longer than RENDER_TIMEOUT_MS).

A recycle restart is routine: the worker restarts its render process after a large document to free memory. failure and timeout restarts point to problems.