Streaming PostgreSQL changes as JSON with wal2json

Modern data systems thrive on real-time change streams. Whether it’s auditing user actions, replicating changes across databases, or feeding events into Kafka, the key is turning raw database changes into something useful.

That’s where wal2json  plays a key role.

wal2json is a PostgreSQL logical decoding plugin that allows you to stream database changes(INSERT, UPDATE, DELETE, optionally TRUNCATE) in JSON format. This is useful for auditing, streaming to Kafka, or building event-driven systems.

Why JSON?
JSON is the common language of modern distributed systems. It’s lightweight, human-readable, language-agnostic, and works seamlessly with tools and programming languages like Kafka, Elasticsearch, Node.js, Python, and Go. By turning PostgreSQL changes into JSON, wal2json becomes the perfect bridge between traditional databases and event-driven ecosystems.

This above diagram illustrates how wal2json works in a PostgreSQL logical replication pipe

PostgreSQL:

• The database generates Write-Ahead Log (WAL) records for every change (insert, update, delete).
• WAL ensures durability and replication in PostgreSQL.

wal2json:

• A logical decoding plugin that converts WAL changes into JSON events.
• Instead of raw binary WAL data, you get structured JSON describing the changes.

JSON Events Output:

The output from wal2json can be directed to multiple destinations
 as shown in the diagram, such as:

• App → Send directly to an application or microservice that consumes database events.
• File → Store changes in a JSON file for auditing or offline analysis.
• Kafka → Stream JSON events into a Kafka topic for real-time processing.

Setup

Install wal2json
sudo dnf install postgresql-wal2json

Make sure to have logical replication  enabled
Make sure you changed these parameters in postgresql.conf file
wal_level = logical
max_replication_slots = 10
max_wal_senders = 10

Restart PostgreSQL to apply changes: sudo systemctl restart postgresql

Check it: which pg_recvlogical
If it is not installed: sudo dnf install -y postgresql17-contrib

/usr/pgsql-17/bin/pg_recvlogical

For reference use the following command:
/usr/pgsql-17/bin/pg_recvlogical -d source_rep --slot kslot --start -o pretty-print=1 -o add-msg-prefixes=wal2json -f - | kafka-console-producer.sh --broker-list localhost:9092 --topic db_changes

Create a Logical Replication Slot (SQL way)

You can create and test a slot directly in SQL:
SELECT * FROM pg_create_logical_replication_slot('myslot', 'wal2json');

SELECT * FROM pg_logical_slot_get_changes('myslot', NULL, NULL);

Using Pg_recvlogical:

Create a slot
/usr/pgsql-17/bin/pg_recvlogical -d source_rep -U postgres   --slot kslot --create-slot --plugin wal2json

Start streaming changes
/usr/pgsql-17/bin/pg_recvlogical -d source_rep --slot kslot --start -o pretty-print=1 -o add-msg-prefixes=wal2json -f -

After this command, open another terminal and log in to the same database, and  make some changes into it: Example: insert or update a table

 Go to previous terminal 

Drop a Slot
pg_recvlogical -d source_rep --slot kslot --drop-slot

Store Changes in a File

If you want to store the changes in a file.
You can write the streamed JSON output to a file:

/usr/pgsql-17/bin/pg_recvlogical -d source_rep --slot kslot --start -o pretty-print=1 -o add-msg-prefixes=wal2json -f /var/lib/pgsql/output_file.json

Format Version 2

By default, wal2json groups changes by transaction (format 1).
If you want one JSON object per row change, use format-version=2:

pg_recvlogical -d mydb -U repuser --slot myslot --start  --plugin wal2json --option 'pretty-print=1' --option 'include-xids=1'   --option 'format-version=2' --file -

Most real-time systems need reliability monitoring

Monitoring & fault tolerance tips:

• Use pg_stat_replication to monitor replication slots.
• Persist slot state carefully (don’t lose replication progress).
• Set up Prometheus/Grafana to watch WAL lag and slot status.
• Handle connection retries in pg_recvlogical if running it in a script.

Use Cases of wal2json

Once you have changed events streaming in JSON, there are many ways to use them. Some common use cases include:

 1. Change Data Capture (CDC)

Capture every row-level change in your PostgreSQL database and feed it to downstream systems for analytics, monitoring, or machine learning.

 2. Real-Time Auditing

Keep track of inserts, updates, and deletes for compliance and debugging.
For example, log all user changes into an audit trail service.

 3. Event-Driven Architectures

Stream database changes directly to message brokers (e.g., Kafka, RabbitMQ, AWS Kinesis).
Each JSON message represents a database event, making it easy to trigger a downstream workflow

 4. Data Replication

Send PostgreSQL changes to other databases (Postgres, MySQL, MongoDB, Elasticsearch, etc.) by consuming wal2json output and applying changes on the target system.

 5. Microservices Communication

Instead of services querying the database repeatedly, changes can be pushed out as events. This helps keep microservices loosely coupled and reactive.

 6. Debugging & Testing

During development, you can stream changes to a file or console to see exactly what transactions PostgreSQL is producing.

Comparison table

Feature	wal2json	Debezium	pgoutput
Format	JSON	JSON/Protobuf	Binary-postgres natie
External dependencies	None	Kafka connect	None
Filtering tables	Yes	Yes	Yes
Support	Lightweight	Enterprise-grade	Postgres built-in

Common errors

Some errors and issues I have encountered:

• ERROR: replication slot “kslot” already exists
  Use a unique slot name or drop it with –drop-slot if a slot with the same name already exists, and is no longer needed.
• Permission denied for replication
  Ensured the role has REPLICATION privilege.
• Output is empty
  This can happen if no changes were done on the replicated table(s). I had modified data in a replicated table to trigger an event . Also check that WAL level is set correctly (logical)

Conclusion

wal2json makes it easy to stream PostgreSQL changes as JSON. With just a few setup steps, you can capture inserts, updates, and deletes in real time and send them to files, applications, or event systems like Kafka. It’s a simple but powerful way to turn your database into a source of live data events.