Background Worker & Lock Monitoring

TimescaleDB runs every refresh, compression, and retention policy inside a finite pool of background workers, and each of those jobs takes locks on the chunks it touches. When the pool is smaller than the number of jobs that come due at the same instant, work does not fail — it silently queues, and next_start slides forward while dashboards still show “policy: enabled.” When a compression job grabs its ACCESS EXCLUSIVE lock on a chunk that ingestion is still writing to, inserts stall behind it. This page shows how to observe the worker pool and the locks its jobs hold, so a starved scheduler or a lock collision becomes a number you can alert on instead of a mystery latency spike. It sits within the broader monitoring, observability, and alerting practice and assumes you already run automated policies in production.

The two failure modes look identical from the application’s side — writes and refreshes get slow — but they have opposite fixes. A starved pool needs more workers or a staggered schedule; a lock collision needs the compression window moved past the write-hot phase. Telling them apart requires looking at the scheduler, the running workers, and pg_locks together, which is what the steps below do.

# Prerequisites

Worker and lock observability draws on cluster-wide PostgreSQL settings, the TimescaleDB job catalog, and read access to the shared statistics views. Confirm each item before wiring alerts.

• TimescaleDB 2.10+ on PostgreSQL 14+ so the timescaledb_information.jobs / job_stats columns and the backend_type labels used below are present.
• The monitoring role can read pg_stat_activity and pg_locks for all sessions — a non-superuser sees only its own rows unless granted pg_monitor. Run GRANT pg_monitor TO metrics_reader; first.
• You know your configured timescaledb.max_background_workers and the instance-wide max_worker_processes; the ratio between them is the ceiling this page reasons about.
• Automated policies are already registered — at minimum a refresh policy and a compression policy — so there is real demand to observe.
• A scrape or query cadence of 15–60 s; lock waits and worker saturation are transient and a 5-minute poll will miss most of them.
• lock_timeout is set (session or role level) so a blocked job fails fast and visibly rather than hanging indefinitely.

Confirm the pool ceiling and that the scheduler itself is alive in one pass:

-- The two ceilings and a liveness check on the launcher/scheduler.
SHOW max_worker_processes;
SHOW timescaledb.max_background_workers;

SELECT count(*) FILTER (WHERE backend_type = 'TimescaleDB Background Worker Launcher') AS launchers,
       count(*) FILTER (WHERE backend_type = 'TimescaleDB Background Worker Scheduler') AS schedulers
FROM pg_stat_activity;

If schedulers is zero, no policy is running at all — the extension was loaded without shared_preload_libraries, or the pool is fully consumed before a scheduler could start. Everything downstream depends on this returning at least one launcher and one scheduler per database with jobs.

# Step-by-Step Implementation

The four steps map to the diagram: inspect the pool that runs the slots, measure the demand queued in front of it, catch the locks a busy slot holds, then alert on both saturation and lock waits.

# 1. Inspect the scheduler and running workers

The pool’s live occupancy lives in pg_stat_activity, filtered by backend_type. TimescaleDB background jobs run as workers whose type string begins with TimescaleDB Background Worker; the launcher and per-database scheduler are separate, always-on backends. Separate them so you can see how many job workers are actually executing versus how many slots exist:

-- Live worker occupancy: how many slots are busy running jobs right now.
SELECT
    backend_type,
    count(*)                                   AS sessions,
    count(*) FILTER (WHERE wait_event_type = 'Lock') AS waiting_on_lock,
    max(now() - query_start)                   AS longest_running
FROM pg_stat_activity
WHERE backend_type LIKE 'TimescaleDB Background Worker%'
GROUP BY backend_type
ORDER BY backend_type;

The TimescaleDB Background Worker rows (without “Launcher” or “Scheduler”) are your job slots in use. If that count equals timescaledb.max_background_workers, the pool is saturated at the moment of the query and any newly-due job is queuing. A non-zero waiting_on_lock on a job worker is the first hint that step 3 will find contention.

# 2. Compare demand against the worker ceiling

Saturation is not a single snapshot — it is demand clustering in time. A pool of four workers handles forty jobs comfortably if they are spread across the hour, and drowns if all forty share a next_start. Query the schedule to find where jobs bunch up, then compare the worst bucket against the ceiling:

-- Demand clustering: jobs grouped by the minute their next run is due.
-- Any bucket whose count exceeds max_background_workers will queue.
SELECT
    date_trunc('minute', next_start)      AS due_minute,
    count(*)                              AS jobs_due,
    string_agg(DISTINCT proc_name, ', ')  AS job_kinds
FROM timescaledb_information.jobs
WHERE scheduled
  AND next_start IS NOT NULL
GROUP BY 1
HAVING count(*) > 1
ORDER BY jobs_due DESC, due_minute
LIMIT 20;

Any jobs_due greater than your timescaledb.max_background_workers is a queue in the making: those jobs will run serially through the free slots, and the ones at the back of the line have their effective start delayed by the sum of the durations ahead of them. This is the demand-versus-supply comparison the diagram’s queue depicts; sizing the pool against it is covered in depth in sizing max_background_workers for many aggregates.

To quantify the gap in one number, contrast the peak bucket with the ceiling:

-- Peak concurrent demand vs. the pool size.
SELECT
    max(jobs_due)                                   AS peak_jobs_due,
    current_setting('timescaledb.max_background_workers')::int AS worker_ceiling,
    max(jobs_due) - current_setting('timescaledb.max_background_workers')::int AS overflow
FROM (
    SELECT count(*) AS jobs_due
    FROM timescaledb_information.jobs
    WHERE scheduled AND next_start IS NOT NULL
    GROUP BY date_trunc('minute', next_start)
) d;

A positive overflow means at least that many jobs queue at peak. Note the ceiling reserves one slot per database for the scheduler itself, so the effective job capacity is timescaledb.max_background_workers minus one per active database.

# 3. Detect blocked and blocking sessions

When ingestion or a refresh stalls, the question is which session holds the lock the victim is waiting for. PostgreSQL exposes this through pg_locks, but the raw view is unreadable — you join it to itself to pair each waiting (granted = false) lock with the granted lock that conflicts, then to pg_stat_activity for the human-readable query on each side. Focus on AccessExclusiveLock, the mode compress_chunk takes while it seals a chunk:

-- Blocked/blocking pairs, with the compression job called out.
SELECT
    waiter.pid                       AS waiting_pid,
    waiter_act.backend_type          AS waiting_backend,
    left(waiter_act.query, 60)       AS waiting_query,
    now() - waiter_act.query_start   AS waited_for,
    blocker.pid                      AS blocking_pid,
    blocker_act.backend_type         AS blocking_backend,
    blocker.mode                     AS blocking_mode,
    left(blocker_act.query, 60)      AS blocking_query
FROM pg_locks waiter
JOIN pg_locks blocker
  ON  waiter.locktype       = blocker.locktype
  AND waiter.database       IS NOT DISTINCT FROM blocker.database
  AND waiter.relation       IS NOT DISTINCT FROM blocker.relation
  AND waiter.pid           <> blocker.pid
JOIN pg_stat_activity waiter_act  ON waiter_act.pid  = waiter.pid
JOIN pg_stat_activity blocker_act ON blocker_act.pid = blocker.pid
WHERE waiter.granted = false
  AND blocker.granted = true
ORDER BY waited_for DESC NULLS LAST;

PostgreSQL 14+ also ships the helper pg_blocking_pids(), which is cheaper for a quick check but does not tell you the lock mode or the blocker’s query — use it to confirm, then the join above to diagnose:

-- Fast confirmation: any session blocked, and by whom.
SELECT pid,
       left(query, 50)          AS blocked_query,
       pg_blocking_pids(pid)    AS blocked_by,
       now() - query_start      AS waited_for
FROM pg_stat_activity
WHERE cardinality(pg_blocking_pids(pid)) > 0
ORDER BY waited_for DESC;

If the blocking_backend is a TimescaleDB Background Worker running compress_chunk and the waiting_backend is a client running INSERT, you have reproduced the diagram’s right-hand collision: compression is sealing a chunk that ingestion is still writing. The full anatomy of that case, including how to identify the exact chunk, is in diagnosing lock contention during compression.

# 4. Alert on queue depth and lock waits

Snapshots become monitoring when you export two gauges on a schedule: worker-pool saturation (busy slots ÷ ceiling) and longest lock wait. Compute both server-side and let your collector scrape the result, so alert thresholds live in one place:

import psycopg
from psycopg.rows import dict_row

def sample_worker_and_lock_health(conn_str: str) -> dict:
    """Return pool saturation and the worst current lock wait, for a metrics scrape."""
    with psycopg.connect(conn_str, row_factory=dict_row) as conn:
        with conn.cursor() as cur:
            cur.execute("""
                SELECT
                    (SELECT count(*) FROM pg_stat_activity
                       WHERE backend_type = 'TimescaleDB Background Worker')      AS busy_workers,
                    current_setting('timescaledb.max_background_workers')::int    AS worker_ceiling,
                    (SELECT count(*) FROM pg_stat_activity
                       WHERE wait_event_type = 'Lock')                            AS sessions_waiting_on_lock,
                    (SELECT COALESCE(EXTRACT(epoch FROM max(now() - query_start)), 0)
                       FROM pg_stat_activity
                       WHERE wait_event_type = 'Lock')                            AS longest_lock_wait_s
            """)
            row = cur.fetchone()
    ceiling = max(row["worker_ceiling"], 1)
    row["pool_saturation"] = round(row["busy_workers"] / ceiling, 3)
    return row

# Alert rules (evaluate in your collector):
#   pool_saturation      >= 0.9  for 10m  -> pool is a bottleneck; stagger or resize
#   longest_lock_wait_s  >= 30           -> ingestion likely stalled behind a lock
#   sessions_waiting_on_lock > 0 for 5m  -> sustained contention, page on-call

Pair the live gauges with a trailing check on the job catalog: a job whose next_start keeps moving forward without a matching last_run_started_at advancing is queuing every cycle. That signal, and the wider set of scheduler health checks, belongs to job stats and scheduler diagnostics; wire these gauges through the same Prometheus and Grafana integration you use for the rest of the platform.

# Configuration Parameters Reference

The relationship between the two ceilings is the one to internalize. Every TimescaleDB job worker is a PostgreSQL background worker, so the pool cannot exceed what max_worker_processes leaves after parallel query and replication claim their share:

$$\text{effective jobs} \le \min\bigl(\texttt{ts.max\_background\_workers},\; \texttt{max\_worker\_processes} - w_{\text{parallel}} - w_{\text{repl}}\bigr) - d$$

where $w_{\text{parallel}}$ is your max_parallel_workers, $w_{\text{repl}}$ covers logical/physical replication slots, and $d$ is the number of databases each reserving a scheduler. Raising timescaledb.max_background_workers without raising max_worker_processes above it is the single most common misconfiguration — the pool silently caps at the lower number.

# Integration with Adjacent Features

Worker and lock monitoring is the connective tissue between the policies that share the pool.

• Refresh queue. When refresh and compression jobs contend for the same slots, the order and concurrency in which they drain is governed by the asynchronous execution queue management layer. The demand query in step 2 is the input to sizing that queue: if refresh jobs cluster at the top of every hour, either their schedule_interval needs staggering or the pool needs a slot they can always find free.
• Compression scheduling. The lock collision in step 3 originates in chunk compression scheduling and automation. The fix is usually upstream — move compress_after past the write-hot window so the sealed chunk is no longer receiving inserts — not in the monitor itself. The monitor’s job is to prove the collision is happening and how often.
• Retention. A retention drop also takes a brief exclusive lock to remove a chunk, and it competes for the same worker slots. Sequencing retention after compression in the schedule keeps both out of each other’s lock path and spreads their demand across the pool.

# Performance Validation

After tuning the pool or the schedule, confirm two things improved: fewer jobs queue at peak, and lock waits shortened. The job catalog’s own accounting shows whether jobs still slip their schedule:

-- Jobs whose actual start lags their scheduled next_start — evidence of queuing.
SELECT
    j.job_id,
    j.proc_name,
    js.last_run_started_at,
    js.last_run_duration,
    js.total_runs,
    js.total_failures,
    j.next_start,
    j.next_start - js.last_run_started_at AS schedule_slip
FROM timescaledb_information.jobs j
JOIN timescaledb_information.job_stats js USING (job_id)
WHERE j.scheduled
ORDER BY js.last_run_duration DESC NULLS LAST
LIMIT 20;

Compare that against the live pool and lock picture from the shared views. A healthy result: busy_workers peaks below the ceiling, schedule_slip stays near the configured interval rather than growing, and the lock-wait query returns no rows for more than a few seconds at a time.

-- Cumulative lock-wait pressure since stats reset — a trend, not a snapshot.
SELECT
    backend_type,
    count(*)                                          AS sessions,
    count(*) FILTER (WHERE wait_event = 'relation')   AS waiting_relation_lock
FROM pg_stat_activity
WHERE state <> 'idle'
GROUP BY backend_type
ORDER BY waiting_relation_lock DESC;

If last_run_duration for a job approaches its schedule_interval, runs risk overlapping and holding two slots — treat that as a sign to lengthen the interval or shrink each run’s backlog, the same way you would for any long-running policy.

# Troubleshooting

Jobs are registered and scheduled = true, but they never seem to run. The pool is fully consumed. Run the step 1 occupancy query: if busy TimescaleDB Background Worker count equals timescaledb.max_background_workers, every slot is taken and new jobs queue behind them indefinitely. Raise the pool (and max_worker_processes above it), or stagger schedule_interval values so demand de-clusters. Confirm the launcher and scheduler exist — a zero from the prerequisites liveness query means nothing will ever run.

next_start keeps sliding forward without the job ever executing. The scheduler re-computes next_start each cycle it cannot dispatch the job because no slot is free. This is the queue in slow motion: the job is due, gets skipped for lack of a worker, and its next slot is pushed out. Correlate the sliding next_start from step 2 with a saturated pool from step 1 — together they confirm starvation rather than a broken job.

Ingestion latency spikes on a schedule matching the compression window. A compression job is holding ACCESS EXCLUSIVE on a chunk your writers still touch. The step 3 join will show a TimescaleDB Background Worker running compress_chunk as the blocking_pid and client INSERTs as the waiters. Move compress_after later so the chunk is write-cold before compression seals it, and set lock_timeout so a collision fails fast instead of pinning the slot and the writers together.

The worker pool is simply smaller than peak demand. Step 2’s overflow is persistently positive and no staggering closes it — you have more concurrent jobs than the ceiling can serve. Increase timescaledb.max_background_workers, remembering to raise max_worker_processes above it and to leave headroom for parallel query and replication per the formula above. If the host cannot spare the processes, consolidate policies (fewer, coarser aggregates) instead.

pg_locks shows a long-held lock but no query in pg_stat_activity. The holder is likely idle in transaction — an application session that opened a transaction, took a lock, and stalled. Set idle_in_transaction_session_timeout so those sessions are reaped, freeing the lock the background job is waiting on.

Frequently Asked Questions

# Related & Navigation

← Back to Monitoring, Observability & Alerting for TimescaleDB Automation

• Job Stats & Scheduler Diagnostics — the catalog-side view of the jobs competing for these workers.
• Prometheus & Grafana Integration — export the pool-saturation and lock-wait gauges to your dashboards.
• Compression & Retention Observability — track the outcomes of the jobs that take the locks watched here.
• Diagnosing Lock Contention During Compression — the full anatomy of the ACCESS EXCLUSIVE collision from step 3.
• Sizing max_background_workers for Many Aggregates — turn the step 2 demand numbers into a pool size.