Patroni Cluster Backup

This page walks the complete procedure for backing up a Patroni HA cluster: getting the backup infrastructure up with pg backup setup, taking full/incr/diff backups with pg ha snapshot, and one operational pitfall — when a replica’s Replay LSN stalls and pg ha status shows members disagreeing on LSN, how to pull it back with patronictl reinit. Command and mechanism details live in Patroni HA and Backup; this page strings them into a path you can run end to end.

Model: one stanza per cluster, the backup container finds the primary

A Patroni cluster (scope) maps to one pgBackRest stanza: pgcli_<scope> (matching the namespace-qualified DCS scope, so two pgcli namespaces sharing one S3 bucket never collide). The stanza is cluster-wide — the backup-side config lists every member as a pg*-host, and pgBackRest SSH-probes them, locates the current primary itself, and keeps following it across failovers with no config change. Therefore:

• There is no per-member backup. full / incr / diff all operate on the one cluster stanza; pgBackRest snapshots whatever host is the leader at that moment.
• Backups run from the shared backup container, reaching the primary over SSH. Cross-host members are reachable too (trust is wired up automatically by setup).
• pg ha snapshot commands never need you to name the leader — pass the scope; pgBackRest resolves the primary.

Prerequisite: pg backup setup

One idempotent run brings up the whole backup path and gives the cluster archiving:

# simplest (backup data lands under the default base-dir)
pg backup setup

# S3 object repository (the prerequisite for archiving to MinIO / AWS S3).
# When the store is on another host and you have no ca.crt yet, pull it with
# one TLS handshake first — no scp:
pg backup fetch-ca 10.0.0.9:9000          # prints a SHA-256 fingerprint to cross-check
pg backup setup --s3-endpoint 10.0.0.9:9000 --s3-bucket pgbackrest \
    --s3-access-key admin --s3-ca-file ~/.pgcli/backup/repo-ca/ca-10.0.0.9-9000.crt

What setup does (when an S3 repo is configured, it first runs an endpoint preflight — a plain TCP dial that fails the run immediately on a mistyped host or a store that is down, rather than after the image pull / container start buries the real cause):

1. Build/pull the pgbackrest image, create the network and dirs, generate the backup-container pgbackrest.conf and the member-local archive view pgbackrest-archive.conf.
2. Start the shared backup container, then verify repository connectivity: a pgbackrest repo-ls proves the whole stack (TLS, credentials, bucket) and maps a failure to an actionable hint — a cert error points at pg backup fetch-ca, access-denied at the credentials, a refused/timeout connection at the endpoint.
3. Wire up cross-host backup trust: each host publishes its backup public key into the cluster’s etcd registry at pg ha create; setup merges every member’s key into one per-cluster authorized_keys bind-mounted into each member container. sshd re-reads that file on every login and the merge rewrites it in place, so hosts that join later are trusted without a restart. The S3 repo CA follows the same path: a host with ca_file publishes it into the registry, joiners with it empty pull it. (Details: HA → Backing up a cross-host leader.)
4. Enable WAL archiving: when a member’s archive config is stale, pause the cluster, recreate stale members replicas-first / leader last (a recreate is exactly the postmaster restart archive_mode needs), resume, and render the same archive_command into every member’s patroni.yml.
5. Run stanza-create + check for every cluster stanza.

HTTPS is mandatory. pgBackRest rejects plaintext S3. A MinIO meant to receive cluster archives must serve TLS — install it with pg addon install minio --tls (pgcli’s self-signed CA; point --s3-ca-file at its ca.crt). When the MinIO is on another machine, pull its CA with one TLS handshake — pg backup fetch-ca <store-host>:<port> — no scp needed (see MinIO → Getting the CA onto a remote host).

Confirm after setup:

pg backup status

Once the backup container is Up and the stanzas are ready, start backing up.

Create members after setup, on every host. A member’s archiving is fixed at creation time: the container only mounts pgbackrest-archive.conf if that file already exists, and the rendered patroni.yml only carries the archive GUCs if an S3 repo is configured. A member pg ha created before pg backup setup on its host therefore starts without WAL archiving — pg ha snapshot/pg ha restore cannot cover it until a later setup flags it stale and recreates it inside a pause window. pg ha create warns about this up front; run pg backup setup first on each host for a backup-ready create.

Snapshot operations: pg ha snapshot

These act on a cluster (scope), running pgBackRest against the current primary from the shared backup container. This is the Patroni-cluster counterpart of pg snapshot (single instance).

# full backup (default --type full); --tail-logs streams pgBackRest output
pg ha snapshot create app --tail-logs

# incremental (changes since the last backup)
pg ha snapshot create app --type incr

# differential (changes since the last full)
pg ha snapshot create app --type diff

# list every snapshot for the cluster (Start / Stop / Name / Type)
pg ha snapshot list app
pg ha snapshot list app --limit 5

# delete a specific snapshot (label from list) — prompts unless --force
pg ha snapshot delete app 20260916-140131F_20260917-015154I
pg ha snapshot delete app <label> --force

Notes:

• Before each command, pg ha snapshot re-renders the backup-container pgbackrest.conf from the current topology, so member adds/removes and port changes are reflected in the stanza immediately — the file is bind-mounted, no recreate.
• You can only delete a non-unique full: pgBackRest keeps at least one full, so deleting the only one is refused (take a new full first).
• create connects to whatever host is leader at the time; after a failover it still succeeds — that is the point of the cluster-wide stanza.

Pitfall: a stuck replica LSN → patronictl reinit

Symptom. In pg ha status app one replica’s Replay LSN trails its own Receive LSN (or visibly differs from another replica) and never converges over time. Note Patroni’s Lag column is relative to the leader, so every number moves when the leader’s LSN advances; to spot a real stall compare a replica’s own recv vs replay, or watch whether its Replay LSN stays frozen across checks.

Confirm. Three signals point to the same conclusion — the replica’s WAL replay is at a hole and streaming has actually stopped:

# 1) leader has no replication clients, slots inactive
pg exec --dsn "postgres://<user>@<leader_host>:<port>/postgres" \
  "SELECT client_addr, state FROM pg_stat_replication"
pg exec --dsn "postgres://<user>@<leader_host>:<port>/postgres" \
  "SELECT slot_name, active, confirmed_flush_lsn FROM pg_replication_slots"

# 2) the stuck replica has no walreceiver
pg exec --dsn "postgres://<user>@<replica_host>:<port>/postgres" \
  "SELECT status FROM pg_stat_wal_receiver"

# 3) the replica container log loops the same two lines (pg logs ha fetches by
#    scope + member name; -f follows, -n takes more lines):
pg logs ha <scope> -m <member> -n 50 | grep -iE "prev-link|waiting for WAL"
#   LOG: record with incorrect prev-link ... at 0/20000060
#   LOG: waiting for WAL to become available at 0/20000078

record with incorrect prev-link + a repeating waiting for WAL to become available means the WAL stream has a hole/discontinuity at a segment boundary and the startup process is dead-waiting for the next segment. Patroni often keeps logging “no action. I am … a secondary, and following a leader” each cycle — it believes it is following while the postmaster’s walreceiver has actually stopped, so it will not self-heal. This is a replication runtime issue, unrelated to backups.

Fix: reinit the replica. Have Patroni take a fresh pg_basebackup from the leader and restart replication. This destructively rebuilds that replica’s data directory (the leader and other replicas are untouched), so Patroni requires --force:

# CLUSTER_NAME is the namespace-qualified scope (the name after "Cluster:" in pg ha status)
pg ha ctl <scope> -- reinit <scope>-<ns-suffix> <member> --force
# example (namespace default -> -default suffix):
pg ha ctl app -- reinit app-default node1 --force

-f does not work: patronictl reinit only accepts the full --force (unlike switchover/failover’s --yes). Success: reinitialize for member node1 means it was dispatched.

Verify recovery. reinit wipes and rebuilds the data dir, then basebackups + replays. Poll until it is recv == replay and status=streaming:

pg exec --dsn "postgres://<user>@<replica_host>:<port>/postgres" \
  "SELECT pg_last_wal_receive_lsn() recv, pg_last_wal_replay_lsn() replay,
          (SELECT status FROM pg_stat_wal_receiver) wr"
#   expected: recv == replay, wr = streaming

pg ha status app    # the replica's Replay LSN catches up, Lag converges

After reinit completes, run pg ha snapshot create app --type full once to realign the snapshot baseline with the healthy topology.