Patroni-managed PostgreSQL clusters depend on etcd as their distributed consensus store, and when etcd stops accepting writes, the impact reaches PostgreSQL even though the database itself is healthy. This post walks through a real incident where all three members of an etcd cluster reported unhealthy with a NOSPACE alarm after the database hit its default 2 GB backend quota. We cover why Patroni’s constant leader-key refreshes cause etcd to grow, how this can lead a primary to demote itself, and the exact steps to recover: compacting revisions, defragmenting each member one at a time, and disarming the alarm. All of this was done, with no restarts and no data loss.
The symptom
A health check against all three members showed every endpoint unhealthy:
+--------------------------+--------+-----------+--------------------------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+--------------------------+--------+-----------+--------------------------+
| http://<etcd-node-1> | false | ~2ms | Active Alarm(s): NOSPACE |
| http://<etcd-node-2> | false | ~2ms | Active Alarm(s): NOSPACE |
| http://<etcd-node-3> | false | ~166ms | Active Alarm(s): NOSPACE |
+--------------------------+--------+-----------+--------------------------+
Error: unhealthy clusterThe status output showed why:
- Version: 3.5.x on all members
- DB size: 2.1 GB on every member
- Raft term, index, and applied index were identical, so the cluster was consistent, just full
- A leader was still elected
etcd’s default backend quota is 2 GB. Once the database crosses it, etcd raises an NOSPACE alarm and goes read-only for writes. Only reads and deletes are accepted until space is reclaimed and the alarm is cleared.
Why this matters for PostgreSQL
Patroni continuously writes to etcd: the leader key is refreshed every loop, and the member and status keys are updated too. If etcd rejects writes, the primary can’t renew its leader lock. Depending on your Patroni settings (for example, whether failsafe_mode is enabled), the primary may demote itself to protect against split-brain. A full etcd database can therefore turn into a PostgreSQL write outage, even though PostgreSQL itself is perfectly healthy.
Why etcd grew to 2 GB
etcd is a multi-version key-value store. Every write creates a new revision, and old revisions are kept until compacted. Patroni’s constant leader-key refreshes generate a huge number of revisions over time. Without compaction, the database grows until it hits the quota, which here was about 4.5 million raft entries. Compaction frees old revisions logically, but defragmentation is what returns the space to the filesystem and shrinks the DB file.
How to fix this
Set up variables
export ETCDCTL_API=3
ETCDCTL=/path/to/etcdctl
EPS=http://<etcd-node-1>:2389,http://<etcd-node-2>:2389,http://<etcd-node-3>:2389Get the current revision
$ETCDCTL --endpoints=http://<etcd-node-1>:2389 endpoint status --write-out=json \
| grep -o '"revision":[0-9]*'Output in our case: "revision":4366376
Compact up to that revision
Compaction is a cluster-wide operation, so run it once against any member:
$ETCDCTL --endpoints=http://<etcd-node-1>:2389 compact 4366376
# Compacted revision 4366376Compaction is still allowed while the NOSPACE alarm is active, which is why this works. Note that the revision number is a required argument, and omitting it is an easy typo to make when pasting commands.
Defragment each member, one at a time
Defrag is per member, so you have to run it on every endpoint:
$ETCDCTL --endpoints=http://<etcd-node-2>:2389 defrag
$ETCDCTL --endpoints=http://<etcd-node-3>:2389 defrag
$ETCDCTL --endpoints=http://<etcd-node-1>:2389 defrag Best practices here:
- Never defrag all members simultaneously. A member being defragmented blocks reads and writes while it rewrites its DB file.
- Do followers first, leader last. It’s a risk-minimization convention rather than a hard etcd requirement
- Confirm each member’s DB size dropped before moving to the next.
Check and disarm the alarm
Clearing the alarm before defrag is pointless. etcd will re-trigger it if the DB is still over quota. Once space is reclaimed:
$ETCDCTL --endpoints=$EPS alarm list
$ETCDCTL --endpoints=$EPS alarm disarm
$ETCDCTL --endpoints=$EPS endpoint health --write-out=tableVerify
+--------------------------+--------+--------------+-------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+--------------------------+--------+--------------+-------+
| http://<etcd-node-3> | true | ~35ms | |
| http://<etcd-node-1> | true | ~101ms | |
| http://<etcd-node-2> | true | ~102ms | |
+--------------------------+--------+--------------+-------+And the status table:
- DB size: 2.1 GB → 45 kB on every member, which will be captured in endpoint status output.
- Leader unchanged, same Raft term (10)
- Raft index and applied index identical across members, and still advancing
- The
ERRORScolumn empty
The whole recovery is done, with no etcd restarts and no data loss.
Conclusion:
A full etcd database is a quiet failure: the cluster stays consistent, and a leader remains elected, yet writes are rejected, and Patroni can no longer renew its leader lock, which puts PostgreSQL availability at risk. The fix is straightforward once you know the sequence: compact to the current revision, defragment members one at a time, then disarm the alarm and verify health. In this case, the database dropped from 2.1 GB to 45 kB with the leader and raft term unchanged.
The better outcome is to prevent it. If you want us to blog about automating compaction.
Thank you!!