PostgreSQL is widely used for applications where data availability and durability are critical. As databases grow and recovery requirements become stricter, having a reliable backup strategy is essential. pg_hardstorage is an open-source PostgreSQL backup and recovery solution designed around continuous WAL streaming, base backups, deduplication, and point-in-time recovery (PITR).

Developed by CYBERTEC PostgreSQL International and released under the Apache 2.0 license, pg_hardstorage is designed to provide a transparent and practical approach to PostgreSQL backup management.

What is pg_hardstorage?

pg_hardstorage speaks PostgreSQL’s replication protocol, the same one a streaming replica uses. It runs as a single static binary on a backup host, takes a base backup, and streams WAL continuously. Nothing is installed on the database itself.

Backups are stored in a content-addressed format: data is split into chunks, each named by its hash and stored once. Every backup is self-contained and restores on its own

How it works:

1. Data Plane — PostgreSQL Replication Protocol

• pg_hardstorage uses PostgreSQL’s native replication protocol, similar to a streaming replica.
• It can request both a consistent base backup and WAL stream through a normal libpq connection.
• No shell access or backup agent is required on the PostgreSQL server.
• Only a PostgreSQL role with the REPLICATION attribute and a repository URL are required.
• It can work with bare-metal PostgreSQL, containers, Patroni, and Kubernetes-managed PostgreSQL.
• It cannot perform physical base backups on managed DBaaS platforms that don’t expose the required BASE_BACKUP replication functionality.

2. Store — Content-Addressed Repository

• Backup data is divided into variable-sized chunks using FastCDC.
• Each chunk is identified using its SHA-256 hash.
• Identical chunks are stored only once, providing automatic deduplication.
• Each backup is represented by a manifest containing chunk references and the WAL range.
• Every backup is logically independent—there is no traditional incremental backup chain.
• Deleting an older backup does not break newer backups because unused chunks can be removed safely.

3. Three Main Stages

• Base Backup: Creates a consistent copy of the PostgreSQL cluster and records the starting WAL position.
• WAL Streaming: Continuously streams WAL through a replication slot and stores it in the repository.
• Restore: Reconstructs the database from the selected base backup and replays WAL to the required recovery target.
• PITR can target a specific timestamp, named restore point, or latest available state.

4. One Static Binary

• pg_hardstorage is distributed as a single statically linked binary.
• Nothing needs to be installed on the PostgreSQL database host.
• The binary can run on a dedicated backup server, VM, container, or Kubernetes CronJob.
• The same binary provides backup, WAL streaming, restore, verification, and health-check functionality.
• This makes upgrades easier because the database host remains untouched.

5. Key Design Choices

• No incremental backup chain: Deduplication provides storage efficiency without creating fragile backup dependencies.
• Every backup is independently restorable: Removing an older backup does not automatically invalidate newer backups.
• No proprietary repository: The repository consists of files, chunks, manifests, and metadata.
• The repository can be stored on a filesystem or S3-compatible object storage.
• The repository format is documented and the project is open source under Apache 2.0.

Conclusion

pg_hardstorage is a PostgreSQL backup system that uses the native replication protocol to take base backups and continuously stream WAL into a content-addressed, deduplicated repository, enabling independent backups and precise Point-in-Time Recovery without installing software on the database host.