Best Practices for WAL Archiving:  Compression, Cleanup & Retention

Let’s learn how to compress, clean up, and retain PostgreSQL WAL files efficiently for PITR, replication, and backups.

PostgreSQL’s Write-Ahead Logging (WAL) is the foundation of its durability and crash recovery system. Every change made to a PostgreSQL database is first recorded in a WAL file before being applied to the actual data files. While this ensures data integrity and consistency, it also means that WAL files accumulate rapidly, especially in busy systems.

If unmanaged, they can:

• Consume large amounts of disk space
• Break the replication setup

In this blog post, we’ll cover three essential best practices to help you manage WAL archives efficiently:

• Compression
• Cleanup
• Retention

Why WAL Archiving Matters

WAL archiving is crucial for:

• Point-in-Time Recovery (PITR)
• Disaster Recovery
• Physical Replication (Streaming or Archive-based)
• Backup verification and consistency

Let’s break down how to make WAL archiving more efficient and production-ready.

1. Compress Your WAL Files

Why Compress?

WAL files can be large and numerous(typically 16MB per segment by default). Compressing them:

• Saves storage
• Reduces I/O and network transfer costs
• Speeds up archiving to remote/cloud destinations (e.g., S3, GCS)

Implementation

You can use compression in your archive_command like this:

archive_command = 'gzip < %p > /mnt/wal_archive/%f.gz' 

Or, use advanced tools that support compression natively:

pgBackRest :

[global]
compress-type=zst
compress-level=3

Barman:

Compression is enabled by default with gzip.

Choose based on your balance between speed and compression ratio.

Tip:

• gzip: universally available, slower compression, but widely supported.
• zstd: newer, faster, better ratios, but requires installation.

2. Automate Cleanup of Old WALs

 Why It’s Important

WAL files must be removed once:

• They’ve been backed up and archived successfully
• They’re no longer needed for PITR or replicas

Failing to clean them up leads to:

• Disk full errors
• Broken replication
• Slower backups

Implementation

 Using archive_cleanup_command (for replicas):

archive_cleanup_command = 'pg_archivecleanup /mnt/wal_archive %r'

This ensures WAL files no longer needed by replicas are deleted automatically.

Using Backup Tools:

pgBackRest

It automatically tracks which WALs are required for each backup.

You can control retention with:

retention-full=2 #Keep WALs needed for the last 2 full backups

Barman

It uses a retention policy like:

retention_policy = REDUNDANCY 3

WARNING: Avoid manual cleanup unless you know exactly which WALs are still needed.

3. Set a Smart Retention Policy

Retention = How long you keep archived WALs

Your retention strategy should depend on:

• Your PITR window (how far back you might need to recover)
• Backup frequency
• Replication needs
• Audit/logging policies

Best Practices

Align WAL retention with your backup strategy:

Example:

• Nightly full backup
• Keep 7 days of WALs for PITR
• Incremental backups every hour → adjust WAL retention accordingly

For pgBackRest:

retention-full=7         # Keep 7 full backups
retention-diff=6         # Keep diffs for them

For  Barman:

retention_policy = RECOVERY WINDOW OF 7 DAYS

If using native tools, schedule scripts that delete WALs older than the oldest required LSN.

Common Pitfalls to Avoid

• Deleting WALs too early breaks PITR and replication
• Keeping WALs forever leads to disk exhaustion
• Forgetting to test recovery retention settings means nothing without verification
• Not monitoring archive success, missing WAL segments can render backups useless

Conclusion

Efficient WAL management in PostgreSQL comes down to:

• Compressing to save space and transfer time
• Cleaning up to prevent disk issues and broken replication
• Retaining only as long as needed for PITR and compliance

By following these best practices and regularly testing your recovery process, you’ll keep your PostgreSQL backups fast, lean, and reliable.