Etcd

etcd is a distributed key-value store. pgcli can run one or more etcd members as a standalone, top-level addon — shared infrastructure rather than a per-instance sidecar. This is useful as the DCS (Distributed Concurrent Store) backing a PostgreSQL HA stack, or as a general config/lock service.

Security caveat: pgcli-managed etcd members currently run without TLS/CA certificates and without authentication/RBAC — any client that can reach a client port has full read/write access. Plan a deployment around network isolation (loopback binds by default; keep advertised ports inside a trusted network). CA/TLS and auth/RBAC support is on the roadmap for a later release.

Members are managed one pg addon install etcd at a time: the first member bootstraps a cluster, and later members join the same named cluster on the fly — from the same host or from another machine.

How It Works

• Shared infrastructure: etcd lives in the top-level addons.etcd map in pg.yaml, keyed by member name — not under any single instance.
• Host network, configurable bind: each member runs with --network host. By default its client/peer URLs bind to 127.0.0.1 — etcd ships without authentication, so loopback-only exposure is the intended posture for a single-host cluster. For cross-host clusters each member advertises a reachable address (--advertise-host) while also listening on loopback, so local etcdctl and remote peers both work.
• Dynamic membership: the first member starts with --initial-cluster-state new; each subsequent member is registered with etcdctl member add against a running peer, then started with --initial-cluster-state existing. pgcli does this automatically — same-host via a member container, cross-host via a temporary etcdctl container.
• Cluster identity: members sharing the same --cluster value join the same etcd cluster (it maps to etcd’s --initial-cluster-token). The default is pgcli-etcd.
• Baked-in tuning: every member launches with periodic compaction (--auto-compaction-mode periodic, --auto-compaction-retention 24h) and an 8 GiB backend quota (--quota-backend-bytes 8589934592) — sane defaults for a small HA metadata store.

Deployment Topologies

pgcli supports two cluster shapes. Both use the same install commands — the difference is whether members share a host.

Single host — all members on one machine (testing / dev)

The classic layout for a dev box or CI: every member listens on loopback with a different auto-assigned port. No --advertise-host needed (default 127.0.0.1), no firewall changes, nothing reachable from outside the host.

pg addon install etcd --name m1              # 127.0.0.1:2379/2380
pg addon install etcd --name m2              # 127.0.0.1:2381/2382
pg addon install etcd --name m3              # 127.0.0.1:2383/2384

All three belong to cluster pgcli-etcd; m1 bootstraps, m2/m3 join on the fly. This gives you real 3-node raft semantics but zero host isolation — the whole cluster dies with one machine, which is exactly why it is a testing topology.

Cross host — one member per machine (production HA)

The production shape: spread members across machines (ideally 3 or 5, an odd count — see Topology & Quorum). Each member advertises its host’s LAN address; the first member must bootstrap with --advertise-host so remote peers can dial it. Ports may repeat on every host since each binds its own interface.

# host A (10.0.0.1) — bootstrap
pg addon install etcd --name m1 --cluster prod \
  --advertise-host 10.0.0.1 --client-port 2379 --peer-port 2380

# host B (10.0.0.2) — join
pg addon install etcd --name m2 --cluster prod \
  --advertise-host 10.0.0.2 --client-port 2379 --peer-port 2380 \
  --join http://10.0.0.1:2379

# host C (10.0.0.3) — join
pg addon install etcd --name m3 --cluster prod \
  --advertise-host 10.0.0.3 --client-port 2379 --peer-port 2380 \
  --join http://10.0.0.1:2379

Requirements: every member’s --advertise-host is a mutual-LAN-reachable address, the firewall on each host lets the other members through on every member’s client and peer ports (bidirectional peer traffic — e.g. open 2379-2386/tcp for a 4-member cluster on default ports), and all members share the same --cluster token. Each host survives losing any one member; the cluster only needs a majority of machines.

Commands

Create the first member

pg addon install etcd --name m1

-> Bootstrapping etcd cluster...
  [OK] etcd container started

✓ etcd installed: "m1"
  Container:    pgcli-etcd-m1
  Cluster:      pgcli-etcd
  Data dir:     ~/.pgcli/addon/etcd/m1/data
  Client port:  2379
  Peer port:    2380
  Advertise:    127.0.0.1

  Client URL: http://127.0.0.1:2379
  Connect (etcdctl): ETCDCTL_ENDPOINTS=http://127.0.0.1:2379

The client port starts at 2379 and the peer port takes the next free port (2380), auto-allocated from etcd_start_port.

Add members to the same cluster

With m1 running, installing another member of the same --cluster joins it:

pg addon install etcd --name m2
pg addon install etcd --name m3

pgcli finds a running peer to act as coordinator, registers m2/m3 via etcdctl member add, and starts them with --initial-cluster-state existing. Ports continue to auto-assign without collisions (m2 → 2381/2382, m3 → 2383/2384). A freshly-grown cluster briefly lacks quorum during leader re-election; pgcli retries the membership operations automatically, so you do not need to wait between installs.

Use a different --cluster to keep members in a separate etcd cluster.

Join a member on another machine

Members on different hosts form one cluster too. Each cross-host member must advertise an address the other members can reach, set with --advertise-host. On the machine that already hosts a member, bootstrap as usual, passing its LAN address:

# host A (10.0.0.1)
pg addon install etcd --name m1 --cluster prod \
  --advertise-host 10.0.0.1 --client-port 2379 --peer-port 2380

Then on the new host, point --join at any existing member’s client endpoint. pgcli registers the member through that endpoint (from a temporary etcdctl container — no local member container or image needed; the image is pulled on demand), takes the authoritative ETCD_INITIAL_CLUSTER from the response, and starts the member:

# host B (10.0.0.2)
pg addon install etcd --name m2 --cluster prod \
  --advertise-host 10.0.0.2 --client-port 2379 --peer-port 2380 \
  --join http://10.0.0.1:2379

-> Registering member with the cluster at http://10.0.0.1:2379...
-> Starting etcd container (joining cluster)...
  [OK] etcd container started

Notes for cross-host clusters:

• Every member needs a reachable --advertise-host — including the first. Its advertised peer URL propagates into the cluster’s membership list, so a first member started on the default 127.0.0.1 can never be joined from another host. If you plan a cross-host cluster, pass the LAN address at bootstrap time.
• --cluster is required with --join and must match the remote cluster’s token — membership is checked by that cluster, not by the local config.
• --advertise-host is required with --join; without it the other members could register a peer URL they can’t dial.
• Ports may repeat across hosts (2379/2380 on both) since each host binds its own interfaces.
• A member’s client/peer ports must be reachable between hosts — open your firewall for every member’s ports, in both directions. Peers dial each other’s peer ports bidirectionally, and clients (and --join/pg etcdctl) reach the client ports, so allow both. Auto-assignment means the exact numbers vary per member — read them from the install summary or pg addon list (or pin them with --client-port/--peer-port) and open that range, e.g. 2379-2386/tcp for a 4-member cluster on default ports. Skipping this is the usual cause of a member hanging with etcdserver: no leader or a --join that times out.
• Re-running the same --join install for an already-registered name fails with a clear error; deregister first via pg etcdctl member remove <hex-id> against the cluster.

Inspect with pg etcdctl

pg etcdctl runs etcd’s client from a short-lived container — no need to exec into a member (and it works even on a host with no member installed, pulling the image on demand). The target comes from ETCDCTL_ENDPOINTS, falling back to the first configured member:

export ETCDCTL_ENDPOINTS=http://10.0.0.2:2379
pg etcdctl member list
pg etcdctl endpoint health
pg etcdctl endpoint status -- -w table

etcdctl flags that pg’s own parser would reject (-w table, --hex, …) go after --.

List

pg addon list

etcd members appear under the Infra add-ons (etcd) section:

Infra add-ons (etcd):
  etcd (name: m1)
    Status:      running
    Cluster:     pgcli-etcd
    Client URL:  http://127.0.0.1:2379
    Client port: 2379
    Peer port:   2380
    Image:       quay.io/coreos/etcd:v3.5.30
    Container:   pgcli-etcd-m1

Start and stop

After a host reboot or a manual stop, bring a member back up without re-running install — no member add, the existing config and on-disk data are reused:

pg addon start etcd --name m1
pg addon stop etcd --name m1

start is safe to repeat: an already-running member is a no-op, and a member whose container was removed is recreated from its data with --initial-cluster-state existing. For a cross-host cluster, confirm the member’s peers are reachable (quorum intact) before starting it.

Remove a member

pg addon remove etcd --name m2

Removal first deregisters the member from the cluster (resolving its hex member ID — etcd v3.5 member remove takes an ID, not a name), then deletes the container and the member’s data directory. Remaining members stay healthy as long as quorum holds.

Cross-host members must be deregistered separately. The automatic deregistration above only works when another running member of the same cluster is present in the local pg.yaml — pgcli has no view of members that live on other hosts. So on a cross-host cluster, removing a member from the host that runs it deletes the container and data but leaves its entry in the cluster’s membership list (a “stale” member) — the surviving members keep trying to peer with the now-deleted host.

Remove it in two steps instead:

# 1. on the host that runs the member — stop and clean up local state
pg addon remove etcd --name m4

# 2. from any surviving host's member, deregister it from the cluster
export ETCDCTL_ENDPOINTS=http://10.0.0.1:2379   # a surviving member's client URL
pg etcdctl member list                            # find m4's hex ID
pg etcdctl member remove <hex-id>                 # e.g. 5c7048c8f7521ec7

pg etcdctl needs a reachable peer to talk to — point ETCDCTL_ENDPOINTS at a member of that cluster that is still running (on a host that still has a running member, this falls back automatically; otherwise set it explicitly), then remove by ID (see Inspect with pg etcdctl). Verify with pg etcdctl member list afterward: the removed name should be gone.

Parameters

Container name follows the namespace convention: pgcli-etcd-<namespace>-<name> (namespace omitted when unset).

Data Directory Layout

Data is always laid out as <root>/<name>/data, so multiple members on one host never share a directory:

<base_dir>/addon/etcd/
├── m1/data/     # member m1
├── m2/data/     # member m2
└── m3/data/     # member m3

--data-dir sets the root. A relative value resolves against the config’s base_dir; the resolved root is persisted, so the config stays portable:

# base_dir: /data/pgcli  →  root /data/pgcli/etcd, member d1 → /data/pgcli/etcd/d1/data
pg addon install etcd --name d1 --data-dir ./etcd

# absolute root
pg addon install etcd --name d2 --data-dir /mnt/ssd/etcd

pg addon remove deletes only that member’s <name>/data dir and leaves the shared root in place.

Connecting

Use pg etcdctl, which runs etcd’s v3 client from a short-lived container against a member’s client URL — the target comes from ETCDCTL_ENDPOINTS, falling back to the first configured member. etcdctl’s native flags go after --:

pg etcdctl member list
pg etcdctl endpoint status -- -w table
pg etcdctl endpoint health
pg etcdctl put foo bar
pg etcdctl get foo -- --hex

pg etcdctl always speaks the v3 API (ETCDCTL_API=3, etcdctl’s default); v2 is not supported.

Topology & Quorum

etcd requires a quorum (majority) of members to accept writes:

• Use an odd number of members — 3 or 5 for production HA.
• Removing members can drop the cluster below quorum (e.g. 3 → 1 surviving); it will then be unable to commit writes. Keep a majority running.

Configuration

After install, pg.yaml records each member under the top-level addons.etcd:

addons:
  etcd:
    m1:
      container_name: pgcli-etcd-m1
      name: m1
      cluster_name: pgcli-etcd
      image_tag: quay.io/coreos/etcd:v3.5.30
      data_dir: /home/user/.pgcli/addon/etcd
      client_port: 2379
      peer_port: 2380
      autostart: true
    m2:
      container_name: pgcli-etcd-m2
      name: m2
      cluster_name: pgcli-etcd
      client_port: 2381
      peer_port: 2382

Port base is configurable via top-level etcd_start_port (default 2379). A cross-host member records the address it advertises:

addons:
  etcd:
    m1:
      name: m1
      cluster_name: prod
      advertise_host: 10.0.0.1
      client_port: 2379
      peer_port: 2380

Auto-start on Boot

Containers also carry a --restart unless-stopped policy, which a per-container conmon monitor enforces even without a podman daemon — it covers crashes but not host reboots. To bring members up after a reboot, enable autostart per member:

pg autostart enable --etcd --name m1
pg autostart enable --etcd --name m2
pg autostart enable --etcd --name m3

This sets autostart: true on the member in pg.yaml and installs/refreshes the boot service (see Auto-start on Boot). Boot is start-only: it starts the member’s existing container and never re-runs member add — an initialized member reloads its cluster from disk and rejoins. On a cross-host cluster, run the command on each host for that host’s member(s). pg autostart status lists every member’s state.

Notes

• Single-node vs cluster: --name m1 alone gives a one-member cluster; install more members with the same --cluster to grow it.
• Production-ready defaults: every member already launches with periodic compaction and an 8 GiB backend quota (see How It Works), suitable for a production DCS without further tuning.
• Reinstall is idempotent: re-running pg addon install etcd --name <m> on a running member recreates its container with updated flags; a member already registered in the cluster is not re-added.