PG19 Hacktober: 31 Days of New Features: Automatic Scaling of PostgreSQL I/O Worker Processes

As part of the PostgreSQL 19 development cycle, the asynchronous I/O (AIO) subsystem has taken another major leap forward. In PostgreSQL 18, when using thread/process-based asynchronous I/O (io_method = worker), the system relied on a fixed pool size defined by io_workers (which defaulted to just 3).

Tuning this static worker pool proved to be tricky for DBAs: set it too low, and the submission queue quickly overflows, forcing queries to silently fall back to synchronous I/O; set it too high, and the system suffers from scheduling overhead, latch contention, and unnecessary worker wakeups.

PostgreSQL 19 completely redesigns this mechanism by removing the static io_workers GUC and introducing automatic, demand-driven scaling of the I/O worker pool. The database can now dynamically scale workers up during heavy I/O backlogs and scale them down when traffic subsides.

In this post, we explain how automatic scaling works, cover the new configuration parameters, look under the hood at the scaling heuristics, and share practical tuning recommendations.

The Problem with Static io_workers in PostgreSQL 18

In PostgreSQL 18, io_method = worker uses a dedicated submission queue (fixed at 64 entries) shared across backends. Under a fixed worker count:

Under-provisioning & Queue Overflow:

If io_workers is configured too low during sudden I/O surges, the 64-slot submission queue fills up. When the queue overflows, backends cannot enqueue additional asynchronous operations and must fall back to synchronous I/O. This causes query latency spikes and allows overflowed synchronous requests to jump ahead of queued asynchronous jobs.

Over-provisioning & Wakeup Storms:

If DBAs set io_workers arbitrarily high (e.g., 16 or 32) to prevent queue overflows, low-traffic periods suffer from CPU scheduling overhead, process churn, and spurious latch wakeups.

No Adaptation to Workload Variance:

Real-world database workloads naturally fluctuate between cached OLTP queries and heavy batch analytics or checkpoint bursts. A single static number cannot fit both states.

The Solution in PostgreSQL 19: Four New GUCs

PostgreSQL 19 deprecates io_workers in favor of four granular parameters that govern dynamic scaling: vi /var/lib/pgsql/19/data/postgresql.conf

# - I/O -
#io_min_workers = 2                     # 1-32
#io_max_workers = 8                     # 1-32
#io_worker_idle_timeout = 60s
#io_worker_launch_interval = 100ms

Detailed Parameter Reference

• io_min_workers (integer, default 2):

The floor of the worker pool. Even during prolonged idle periods, the postmaster will never reduce the pool below this threshold, ensuring that initial asynchronous I/O requests always experience zero spawn latency. (Note: While early patches experimented with a minimum of 1, the community settled on 2 to guarantee baseline concurrency for read-intensive queries).

• io_max_workers (integer, default 8):

The upper ceiling of concurrent I/O workers the postmaster is permitted to spawn. PostgreSQL supports an internal maximum of 32 workers per cluster.

• io_worker_idle_timeout (integer, default 60s):

The duration of inactivity after which entirely idle workers exit, gradually shrinking the pool size back to io_min_workers. Setting this to -1 disables shrinking.

• io_worker_launch_interval (integer, default 100ms):

The minimum cooldown interval between launching consecutive workers. This rate-limiting mechanism dampens pool growth, preventing fork storms during micro-bursts of temporary traffic.

How Dynamic Scaling Works Under the Hood

1. Work Concentration (Lowest-Worker Bias)

Rather than distributing I/O requests uniformly or round-robin among all workers, PostgreSQL deliberately concentrates tasks onto the lowest-numbered available workers (Worker 0, then Worker 1, etc.):

I/O Submission ───► [ Worker 0 (Hot) ] ───► [ Worker 1 (Warm) ] ───► [ Worker 2..N (Idle) ]
                     Always preferred         Woken on spillover       Allowed to time out

• Cache Locality & Latch Collapsing: Keeps lower-numbered workers hot and running in CPU caches while allowing latches to collapse when multiple jobs arrive in quick succession.
• Deterministic Scale-Down: High-numbered workers remain idle longer, allowing them to cleanly hit their io_worker_idle_timeout.

2. Backlog Detection & Scale-Up Mechanics

When an I/O request is submitted or when an active worker checks the queue:

1. The worker evaluates whether peers are available. If no idle higher-numbered worker can be found to take up remaining queued work, and the queue depth exceeds the active worker threshold (queue_depth > worker_id), a backlog is detected.
2. The worker sets a shared memory flag io_worker_control->grow = true and sends a PMSIGNAL_IO_WORKER_GROW signal to the postmaster.
3. Suppression & Pacing:
   • If the pool is already at io_max_workers, signals to the postmaster are suppressed.
   • If a launch signal has already been delivered, redundant signals are suppressed via grow_signal_sent until the postmaster processes the request.
   • The postmaster enforces io_worker_launch_interval (default 100ms), ensuring that workers are added at a controlled, linear rate rather than overwhelming the OS process scheduler.

3. Graceful, Serialized Scale-Down

To avoid a stampede where multiple idle workers exit simultaneously:

• Only the highest-numbered worker can time out: Only worker_id == highest_active_worker checks the idle countdown.
• When that worker exits after exceeding io_worker_idle_timeout, it notifies the remaining pool members via pgaio_workerset_wake().
• The newly promoted highest worker then restarts its idle countdown from zero. This creates a staggered, orderly contraction back down to io_min_workers.

4. Spurious Wakeup Suppression

In high-concurrency systems, waking up an idle worker when another fast worker is about to steal the queued task results in wasted CPU cycles. PostgreSQL 19 introduces a moving ratio between wakeups and completed I/O operations (hist_wakeups vs hist_ios). If a worker observes that wakeups are not yielding real I/O completions, it suppresses fan-out wakeups to reduce system overhead.

Monitoring I/O Workers in PostgreSQL 19

Hands-on Test Case (With Real Terminal Outputs)

Below is the complete walkthrough formatted for your blog post or documentation.

Step 1: Checking Baseline Configuration and Idle State

First, verify the current settings in postgresql.conf or inspect them via SHOW:

postgres=# SHOW io_method;
 io_method
-----------
 worker
(1 row)

postgres=# SHOW io_min_workers;
 io_min_workers
----------------
 2
(1 row)

postgres=# SHOW io_max_workers;
 io_max_workers
----------------
 8
(1 row)

Explanation:

• io_method = worker: Uses worker-based asynchronous I/O.
• io_min_workers = 2: Minimum worker pool size.
• io_max_workers = 8: Maximum worker pool size.

PostgreSQL 19 makes it easier to observe the worker pool in action:

Inspecting Active Workers via the OS at Idle

When the server is idle, PostgreSQL maintains strictly io_min_workers (2 workers: Worker 0 and Worker 1):

[postgres@node3 ~]$  ps -ef | grep "postgres: io worker"
postgres    1974    1972  0 Oct09 ?        00:00:01 postgres: io worker 0
postgres    1975    1972  0 Oct09 ?        00:00:01 postgres: io worker 1
postgres    4673    4337  0 09:59 pts/2    00:00:00 grep --color=auto postgres: io worker

Note: Workers 0 and 1 remain running indefinitely to handle baseline I/O without incurring process fork latency.

Step 2: Checking the Submission Status View (pg_aios)

PostgreSQL 19 exposes the internal state of all asynchronous I/O handles through the pg_aios system view:

postgres=# \d pg_aios
                  View "pg_catalog.pg_aios"
     Column      |   Type   | Collation | Nullable | Default
-----------------+----------+-----------+----------+---------
 pid             | integer  |           |          |
 io_id           | integer  |           |          |
 io_generation   | bigint   |           |          |
 state           | text     |           |          |
 operation       | text     |           |          |
 off             | bigint   |           |          |
 length          | bigint   |           |          |
 target          | text     |           |          |
 handle_data_len | smallint |           |          |
 raw_result      | integer  |           |          |
 result          | text     |           |          |
 target_desc     | text     |           |          |
 f_sync          | boolean  |           |          |
 f_localmem      | boolean  |           |          |
 f_buffered      | boolean  |           |          |
When no queries are submitting I/O operations:
postgres=# SELECT
    pid,
    operation,
    target,
    state,
    f_sync
FROM pg_aios;
 pid | operation | target | state | f_sync
-----+-----------+--------+-------+--------
(0 rows)

No I/O handles were visible at that moment. This does not mean there are no I/O worker processes.

f_sync = false: Normal asynchronous operation handled by workers.

f_sync = true: Queue overflowed! The request had to fall back to synchronous execution. If you see f_sync = true regularly, your io_max_workers ceiling is set too low for your hardware throughput.

Step 3: Preparing the Test Database and Table

Use pgbench to create a test database and insert test data:

[postgres@node3 ~]$ pgbench -i -s 50 -p 5419 postgres
dropping old tables...
creating tables...
generating data (client-side)...
vacuuming...
creating primary keys...
done in 8.13 s (drop tables 0.11 s, create tables 0.06 s, client-side generate 5.69 s, vacuum 0.42 s, primary keys 1.86 s).

During the workload, when you check the IO activity using the pg_aios view from another terminal, you might see an output similar to the following:

postgres=# SELECT pid, operation, target, state, f_sync FROM pg_aios;
 pid  | operation | target |      state       | f_sync
------+-----------+--------+------------------+--------
 5455 | readv     | smgr   | COMPLETED_SHARED | f
 5455 | readv     | smgr   | COMPLETED_SHARED | f
 5455 | readv     | smgr   | COMPLETED_SHARED | f
 5455 | readv     | smgr   | COMPLETED_SHARED | f
 5455 | readv     | smgr   | COMPLETED_SHARED | f
 5456 | readv     | smgr   | SUBMITTED        | f
(6 rows)

• readv: Read operation.
• smgr: Relation storage I/O.
• SUBMITTED: Request submitted, not necessarily finished.
• COMPLETED_SHARED: Shared completion processing finished.
• f: false; the operation is not marked as synchronous.

4. In another terminal, trigger aggressive unbuffered reads across multiple sessions using parallel pgbench connections

[postgres@node3 ~]$  pgbench -c 10 -j 4 -T 25 -P 3 -p 5419 -f <(echo "SELECT count(*) FROM pgbench_accounts WHERE abalance > 0;") postgres
pgbench (17.10.1.0, server 19beta4)
starting vacuum...end.
progress: 3.0 s, 1.0 tps, lat 2004.034 ms stddev 57.041, 0 failed
progress: 6.0 s, 3.3 tps, lat 3833.202 ms stddev 1668.034, 0 failed
progress: 9.0 s, 2.7 tps, lat 2652.766 ms stddev 1354.571, 0 failed
progress: 12.0 s, 3.7 tps, lat 4134.428 ms stddev 1917.795, 0 failed
progress: 15.0 s, 1.3 tps, lat 2406.603 ms stddev 413.259, 0 failed
progress: 18.0 s, 3.3 tps, lat 4390.195 ms stddev 1622.151, 0 failed
progress: 21.0 s, 3.0 tps, lat 2543.506 ms stddev 439.779, 0 failed
progress: 24.0 s, 2.7 tps, lat 4233.965 ms stddev 1805.128, 0 failed
progress: 27.0 s, 3.3 tps, lat 2845.316 ms stddev 1157.577, 0 failed
transaction type: /dev/fd/63
scaling factor: 1
query mode: simple
number of clients: 10
number of threads: 4
maximum number of tries: 1
duration: 25 s
number of transactions actually processed: 77
number of failed transactions: 0 (0.000%)
latency average = 3430.774 ms
latency stddev = 1606.671 ms
initial connection time = 16.098 ms
tps = 2.833363 (without initial connection time)

Explanation:

• -c 10: Ten client connections.
• -j 4: Four benchmark threads.
• -T 25: Run for 25 seconds.
• -P 3: Report progress every three seconds.

While the reads are happening, if you check the IO activity using the pg_aios view from another terminal, you might see an output similar to the following:

postgres=# SELECT pid, operation, target, state, f_sync FROM pg_aios;
 pid  | operation | target |      state       | f_sync
------+-----------+--------+------------------+--------
 5464 | readv     | smgr   | COMPLETED_SHARED | f
 5464 | readv     | smgr   | COMPLETED_SHARED | f
 5464 | readv     | smgr   | COMPLETED_SHARED | f
 5466 | readv     | smgr   | COMPLETED_SHARED | f
 5467 | readv     | smgr   | COMPLETED_SHARED | f
 5467 | readv     | smgr   | COMPLETED_SHARED | f
 5469 | readv     | smgr   | COMPLETED_SHARED | f
 5469 | readv     | smgr   | COMPLETED_SHARED | f
 5469 | readv     | smgr   | COMPLETED_SHARED | f
 5469 | readv     | smgr   | COMPLETED_SHARED | f
 5465 | readv     | smgr   | COMPLETED_SHARED | f
 5465 | readv     | smgr   | COMPLETED_SHARED | f
 5468 | readv     | smgr   | SUBMITTED        | f
 5468 | readv     | smgr   | SUBMITTED        | f
 5468 | readv     | smgr   | SUBMITTED        | f
 5468 | readv     | smgr   | SUBMITTED        | f
 5468 | readv     | smgr   | SUBMITTED        | f
 5470 | readv     | smgr   | SUBMITTED        | f
 5470 | readv     | smgr   | SUBMITTED        | f
 5470 | readv     | smgr   | SUBMITTED        | f
 5470 | readv     | smgr   | SUBMITTED        | f
 5470 | readv     | smgr   | SUBMITTED        | f
 5473 | readv     | smgr   | COMPLETED_SHARED | f
 5473 | readv     | smgr   | COMPLETED_SHARED | f
 5473 | readv     | smgr   | COMPLETED_SHARED | f
 5472 | readv     | smgr   | COMPLETED_SHARED | f
 5536 | readv     | smgr   | COMPLETED_SHARED | f
 5536 | readv     | smgr   | COMPLETED_SHARED | f
 5536 | readv     | smgr   | COMPLETED_SHARED | f
 5536 | readv     | smgr   | COMPLETED_SHARED | f
 5534 | readv     | smgr   | SUBMITTED        | f
 5534 | readv     | smgr   | SUBMITTED        | f
 5534 | readv     | smgr   | SUBMITTED        | f
 5534 | readv     | smgr   | SUBMITTED        | f
 5529 | readv     | smgr   | COMPLETED_SHARED | f
 5529 | readv     | smgr   | COMPLETED_SHARED | f
 5529 | readv     | smgr   | COMPLETED_SHARED | f
 5529 | readv     | smgr   | COMPLETED_SHARED | f
 5529 | readv     | smgr   | COMPLETED_SHARED | f
 5529 | readv     | smgr   | COMPLETED_SHARED | f
 5533 | readv     | smgr   | COMPLETED_SHARED | f
 5533 | readv     | smgr   | COMPLETED_SHARED | f
 5533 | readv     | smgr   | COMPLETED_SHARED | f
 5533 | readv     | smgr   | COMPLETED_SHARED | f
 5530 | readv     | smgr   | COMPLETED_SHARED | f
 5530 | readv     | smgr   | SUBMITTED        | f
 5530 | readv     | smgr   | COMPLETED_SHARED | f
 5530 | readv     | smgr   | COMPLETED_SHARED | f
 5530 | readv     | smgr   | COMPLETED_SHARED | f
 5535 | readv     | smgr   | COMPLETED_SHARED | f
 5535 | readv     | smgr   | COMPLETED_SHARED | f
 5535 | readv     | smgr   | COMPLETED_SHARED | f
 5535 | readv     | smgr   | COMPLETED_SHARED | f
 5537 | readv     | smgr   | COMPLETED_SHARED | f
 5537 | readv     | smgr   | COMPLETED_SHARED | f
 5537 | readv     | smgr   | COMPLETED_SHARED | f
 5537 | readv     | smgr   | COMPLETED_SHARED | f
(57 rows)

5. While at least one of the above process is running (i.e., #3 or #4), check the number of active Postgres IO Workers. Depending on the workload assigned, this should have increased from 2 to a value less than ‘io_max_workers’

[postgres@node3 ~]$ ps -ef | grep "[p]ostgres: io worker"
postgres    1974    1972  0 Oct09 ?        00:00:06 postgres: io worker 0
postgres    1975    1972  0 Oct09 ?        00:00:05 postgres: io worker 1
postgres    5345    1972  4 12:12 ?        00:00:02 postgres: io worker 2
postgres    5353    1972  4 12:12 ?        00:00:02 postgres: io worker 3
postgres    5354    1972  4 12:12 ?        00:00:02 postgres: io worker 4
postgres    5355    1972  3 12:12 ?        00:00:02 postgres: io worker 5
postgres    5356    1972  3 12:12 ?        00:00:01 postgres: io worker 6
postgres    5357    1972  2 12:12 ?        00:00:01 postgres: io worker 7

Output: Workers 0–7 are shown.

Explanation: The pool has reached its configured maximum of eight workers, demonstrating that the pool can grow under workload.

6. After the activities #3 and #4 get completed, check the number of active Postgres IO Workers again. This time it should have dropped from the earlier number.

[postgres@node3 ~]$  ps -ef | grep "[p]ostgres: io worker"
postgres    1974    1972  0 Oct09 ?        00:00:08 postgres: io worker 0
postgres    1975    1972  0 Oct09 ?        00:00:08 postgres: io worker 1
postgres    5345    1972  0 12:12 ?        00:00:04 postgres: io worker 2
postgres    5353    1972  0 12:12 ?        00:00:04 postgres: io worker 3
postgres    5354    1972  0 12:12 ?        00:00:04 postgres: io worker 4

Extra workers may remain alive until the idle timeout expires. They do not necessarily disappear immediately when the benchmark ends.

Check the timeout:

postgres=# SHOW io_worker_idle_timeout;
 io_worker_idle_timeout
------------------------
 1min
(1 row)

Wait beyond the configured timeout and check the process list again.

Expected behavior: The pool may shrink toward the minimum of two workers when demand subsides, and extra workers become eligible to exit.

Summary

PostgreSQL 19’s automatic scaling for I/O worker processes solves a critical usability hurdle of the new AIO architecture. By replacing the static io_workers setting with an elastic, self-adjusting pool:

• DBAs spend less time guessing pool sizes.
• Temporary I/O spikes are accommodated without fallback to synchronous execution.
• The system conserves resources and context switches during quiet periods.

If you are running PostgreSQL on systems without native kernel AIO engines like Linux io_uring, or running on platforms like macOS, BSD, or Windows, io_method = worker in PostgreSQL 19 is now faster, more resilient, and self-tuning out of the box.