Alertmanager

pg addon install alertmanager runs Alertmanager — the alert deduplication, grouping, routing and notification layer that sits downstream of Prometheus — as a managed container from the official docker.io/prom/alertmanager image. Like Prometheus and Grafana it is a top-level addon: it fronts no single PostgreSQL instance, so it is not attached to any one of them. Prometheus and Alertmanager are the natural pair — Prometheus fires alerts, Alertmanager decides who gets notified and how — and this addon is Prometheus’s near-exact twin in shape (single container, host networking on Linux, one web port from its own pool, persistent data dir, autostart, logs, rendered config file + trailing CLI flags, SIGHUP reload, no web login).

The wiring lives on the Prometheus side. pg addon install alertmanager does not touch any prometheus.yml — and it does not need to: Prometheus points at this instance itself, with pg addon install prometheus --alertmanager-url <host:port of this instance>, which injects the alerting: section (and a scrape job watching this instance’s /metrics). On a hand-written Prometheus config (--conf-file), write the alerting: block yourself. Either way, once pointed here, alerts flow.

There is NO web login/password — and the stakes are higher than Prometheus’s. Alertmanager ships no authentication at all, and its unauthenticated API is write-capable in a way Prometheus’s read-mostly one is not: anyone who reaches the port can create silences — i.e. silently mute every notification your on-call expects, with no trace in Prometheus’s own metrics. Its 0.0.0.0 default is therefore guarded only by the network boundary, and that boundary matters more here than anywhere else in the addon family. Treat exposing it as handing out a mute switch for your alerting, not like a password-protected dashboard.

Platform: Linux is the tested platform. The design supports macOS (the image is multi-arch; the container runs under the podman machine VM on the pgcli-net bridge with the port published), but it has not been validated there yet.

Parameters

The data directory holds Alertmanager’s state: silences and notification state. It survives pg addon remove (delete it with --clean-data), so reinstalling under the same name revives them. The image runs as a baked nobody (uid/gid 65534) and creates no writable directory itself, so pgcli chowns the data dir on the host before creating the container — via podman unshare chown 65534:65534 under rootless podman, or a direct recursive chown when pgcli runs as root on a rootful host (exactly Prometheus’s and Grafana’s mechanism, different id).

--webhook-url is not connectivity-probed, on purpose: unlike a Prometheus scrape (GET /metrics, reachable at install time and answerable then), a webhook is a POST that only fires when an alert actually routes through it — dialling it at install would tell you nothing you could rely on. The only early gate is the URL’s shape; whether the endpoint exists, is reachable, and accepts the payload is discovered at alert time (check pg logs addon alertmanager when notifications go missing).

Install

pg addon install alertmanager

The command pulls the image if needed, renders an alertmanager.yml, creates the container, and prints the URL, config path, and what the route renders to:

✓ Alertmanager installed: "alertmanager"
  Container:  pgcli-alertmanager-default-alertmanager
  Image:      docker.io/prom/alertmanager:v0.34.1
  Data:       /home/you/.pgcli/addon/alertmanager/alertmanager/data
  Config:     /home/you/.pgcli/addon/alertmanager/alertmanager/alertmanager.yml (rendered: no-op receiver)
  URL:        http://0.0.0.0:9093/

With no --webhook-url the rendered config is a single route pointing at a name-only no-op receiver: a legal, fully-functional config — the web UI and API are live, silences work, the HA cluster port is disabled — that simply sends nothing. This is the “bring up the endpoint, decide routing later” default. To actually notify something, add a webhook:

# one URL → the route's receiver becomes web.hook with one webhook_configs entry
pg addon install alertmanager --name am1 --webhook-url http://127.0.0.1:5001/

# several URLs → all of them land under ONE web.hook receiver (fan-out to each)
pg addon install alertmanager --name am1 \
  --webhook-url http://127.0.0.1:5001/ \
  --webhook-url https://hooks.example.com/alert?token=abc

Re-running install on a running instance hot-reloads it (SIGHUP) when the rendered/copied config actually changed, and leaves it alone when it didn’t; --force recreates the container instead — to apply a changed --port / --listen (which are container arguments, not config-file contents).

Re-running with a different --webhook-url set replaces the stored list wholesale (passing none clears it back to no-op); the diff is what decides whether to reload.

Custom config file (--conf-file)

Give pgcli your own alertmanager.yml and it skips rendering entirely:

pg addon install alertmanager --name am1 --conf-file /etc/alertmanager/my.yml

pgcli copies your file into its managed path (<base>/addon/alertmanager/<name>/alertmanager.yml) and mounts that copy read-only — never your original. This is deliberate: a single-file bind mount pins the source inode, so an in-place edit of your original file would not propagate into the running container. Because it’s a copy, editing your own file and running pg addon alertmanager reload re-copies it and signals Alertmanager to reread. pgcli does not validate or rewrite the file’s contents (same contract as the nginx/haproxy/prometheus --conf-file). --conf-file is mutually exclusive with --webhook-url (the flag that shapes the rendered file) — a real routing tree (email/Slack/PagerDuty, sub-routes, inhibition) lives in your own file. Omit --conf-file on a reinstall to go back to render mode.

The file is plain Alertmanager YAML — pgcli adds nothing to it and expects nothing from it beyond “parses”. A worked example with a sub-route tree, inhibition, and three different receiver types:

# Complete Alertmanager config for `pg addon install alertmanager
# --name am1 --conf-file /path/to/this.yml` — everything pgcli would
# render is replaced by what follows; only the container flags
# (--web.listen-address, --storage.path, --cluster.listen-address=)
# stay pgcli's job.
global:
  resolve_timeout: 5m
  smtp_smarthost: "smtp.example.com:587"
  smtp_from: "alertmanager@example.com"
  smtp_auth_username: "alertmanager@example.com"
  smtp_auth_password: "CHANGE-ME"
  slack_api_url: "https://hooks.slack.com/services/T00000/B00000/XXXXXXXX"

inhibit_rules:
  # a critical alert mutes warning/info noise from the SAME alert on the SAME instance
  - source_matchers: ['severity = critical']
    target_matchers: ['severity =~ warning|info']
    equal: ['alertname', 'instance']

route:
  receiver: default
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # critical goes to the on-call mailbox, and still falls through to default
    - matchers: ['severity = critical']
      receiver: oncall-email
      continue: true
    # storage team's alerts get their own Slack channel
    - matchers: ['team = storage']
      receiver: storage-slack
      group_by: ['alertname']
      group_wait: 10s

receivers:
  - name: default
    webhook_configs:
      - url: "http://127.0.0.1:5001/"

  - name: oncall-email
    email_configs:
      - to: "oncall@example.com"
        send_resolved: true

  - name: storage-slack
    slack_configs:
      - channel: "#storage-alerts"
        send_resolved: true

pgcli itself never validates this file (no amtool check-config gate at install — see Known limitations), but the image ships amtool, so you can check the managed copy inside the running container before trusting a reload (container name as printed by pg addon list, namespace-scoped — pgcli-alertmanager-default-am1 in the transcript above):

pg addon install alertmanager --name am1 --conf-file /etc/alertmanager/my.yml
# edit /etc/alertmanager/my.yml ...
podman exec pgcli-alertmanager-default-am1 amtool check-config /etc/alertmanager/alertmanager.yml
pg addon alertmanager reload --name am1   # re-copies and SIGHUPs once the file is good

Reload (hot config change)

pg addon alertmanager reload --name am1

This rewrites the managed alertmanager.yml (re-render in render mode, re-copy your file in --conf-file mode) and sends SIGHUP to the running container, which makes Alertmanager reread it in place — no container recreate, no dropped notifications.

Unlike Prometheus, Alertmanager’s POST /-/reload endpoint exists and cannot be disabled (there is no --web.enable-lifecycle-style gate on it). pgcli still reloads via SIGHUP — not because the route is closed, but so it never has to depend on an unauthenticated HTTP endpoint that is, by the image’s design, always open. Anyone who can reach the port can already hit that route directly; pgcli simply doesn’t add to the surface. If you reload a hand-written --conf-file whose config is rejected, Alertmanager keeps its last good config and logs the error, so check pg logs addon alertmanager afterwards. The instance must already be running (pg addon start alertmanager).

Listening and exposure

Alertmanager’s Listen defaults to 0.0.0.0: the container shares the host’s network namespace, so the UI and API are reachable on every interface at the assigned port. There is no password behind it, and — unlike Prometheus — the API lets a caller CREATE SILENCES, i.e. mute every notification your on-call expects, and reload its config. A loopback-only deployment is the safe default: pass --listen 127.0.0.1 (recreate with --force); pg addon list then shows http://127.0.0.1:<port>/. Otherwise restrict access at the firewall, and treat an open 0.0.0.0 as a reachable mute switch.

Ports

Each Alertmanager instance owns a single port: the web UI/API. It is allocated from a pool with base alertmanager_start_port (default 9093 — Alertmanager’s conventional web port), incrementing for every additional instance and skipping ports already taken by any managed container or the used_ports list. The allocation is written into pg.yaml, so a reinstall never re-hands-out a port another addon is using. 9093 shares no number with Grafana (3000), pgAdmin (5050), Prometheus (9090), or nginx (8080); an explicit --port still wins.

9094 — the HA gossip port — is never opened, because it’s disabled by a flag. Alertmanager’s high-availability clustering listens on TCP+UDP 9094 by default and will keep trying to bootstrap peers that never arrive for a single instance. pgcli passes --cluster.listen-address= (empty) to switch it off entirely — a single-instance contract, enforced at every create, so the cluster port never opens and no UDP listener lingers. (This is only about the cluster use of 9094; the number is not reserved, so the web-port pool is free to hand 9094 out as the second instance’s web port — see the example below.) Clustering several pgcli-managed Alertmanagers into one gossip group is not wired up (see Known limitations).

pg addon install alertmanager --name am2          # second instance: next free port after 9093 (e.g. 9094 — the pool is plain numeric assignment; nothing reserves the disabled cluster port's number)
pg addon install alertmanager --name am3 --port 9200  # explicit port still takes precedence

Configuration

addons:
  alertmanager:
    am1:
      name: am1
      image_tag: docker.io/prom/alertmanager:v0.34.1
      host_port: 9093
      listen: 0.0.0.0
      webhook:
        - http://127.0.0.1:5001/
        - https://hooks.example.com/alert?token=abc
      autostart: false
      # conf_file: /etc/alertmanager/my.yml   # operator-file mode, skips rendering
      # data_dir: /srv/alertmanager-data      # optional, overrides the default location

The rendered alertmanager.yml carries a global.resolve_timeout, one root route (group_by: ['alertname'], default group_wait/interval/repeat_interval) pointing at a single receiver, and that receiver’s webhook_configs — one entry per webhook URL under web.hook, or a name-only no-op receiver when the list is empty. The listener bind, storage path, and the disabled cluster address are not in that file — they are passed to the alertmanager binary as CLI flags (--web.listen-address, --storage.path, --cluster.listen-address=), which is why --listen remains settable alongside --conf-file.

List

pg addon list

Web add-ons (alertmanager):
  alertmanager (name: am1)
    Status:      running
    URL:         http://0.0.0.0:9093/
    Config:      /home/you/.pgcli/addon/alertmanager/am1/alertmanager.yml [rendered (2 webhook)]
    Data:        /home/you/.pgcli/addon/alertmanager/am1/data
    Image:       docker.io/prom/alertmanager:v0.34.1
    Container:   pgcli-alertmanager-am1

There are no credentials to reveal — Alertmanager has none — so --show-password shows nothing extra for it, and pg addon password alertmanager resolves nothing.

Start and stop

Containers run with podman’s unless-stopped restart policy, so they come back automatically with the podman service after a host reboot. start/stop manage them explicitly:

pg addon stop alertmanager
pg addon start alertmanager

Auto-start on Boot

To have an Alertmanager instance started automatically on host boot, enable it:

pg autostart enable --alertmanager --name am1

This flags the instance in pg.yaml and installs/refreshes a boot service (a systemd --user unit on Linux, a LaunchAgent on macOS) that runs pg start --autostart. A boot-time start only brings the container up — reading the managed alertmanager.yml already on disk, and the data dir was already chowned at create time, so it needs no host privileges beyond podman itself. In the boot order Alertmanager starts right after Prometheus and before nginx — the natural companion to the collector above, which is also where the two get linked (pg addon install prometheus --alertmanager-url …). See Autostart for the full mechanism, status and disabling.

Remove

pg addon remove alertmanager              # keeps silences + notification state — they survive a reinstall
pg addon remove alertmanager --clean-data # deletes them too

Under rootless podman, --clean-data deletes the data dir through podman unshare rm — Alertmanager writes as its container uid (65534), which maps to a subordinate uid the host user cannot rm directly. --conf-file mode removes only pgcli’s copy; your original file wherever you put it is never touched.

Logs

pg logs addon alertmanager            # show recent console output
pg logs addon alertmanager -f         # follow
pg logs addon alertmanager -n 200     # last 200 lines (also -i / --tail)

Troubleshooting

• install fails with “chowning Alertmanager data dir … to 65534:65534” — the image runs as nobody (uid/gid 65534) and never fixes ownership itself, so pgcli chowns the host data dir before creating the container: through podman unshare chown under rootless podman (where the mapped subordinate uid is the only correct target), or a direct recursive chown when pgcli itself runs as root on a rootful host. The manager shells out to the plain podman CLI, so this only fails outside those two cases — e.g. a non-root pg explicitly pointed at someone else’s rootful daemon. Run the install as root, or switch to rootless podman; a manual sudo chown -R 65534:65534 <data-dir> alone does not clear it, since pgcli re-checks ownership on every install attempt.
• No notifications arrive — the most common cause is a no-op receiver: an install with no --webhook-url renders a config that never sends anything. Check the Config: line in pg addon list (it says no-op receiver vs rendered (N webhooks)). If a real webhook is configured, it is not connectivity-probed at install — reachability is only discovered when an alert fires, so check pg logs addon alertmanager for POST failures and confirm the endpoint accepts Alertmanager’s webhook payload.
• --webhook-url aborted with a parse error — by design: every URL is validated (net/url parse, http/https scheme, non-empty host) before pg.yaml or any file is written, so a bad URL leaves the running instance and config untouched. Fix the URL and re-run.
• A config reload didn’t take effect — pg addon alertmanager reload needs the container running; if the new alertmanager.yml is rejected (a route referencing an undefined receiver, bad YAML) Alertmanager keeps serving the last good config and logs the error. Check pg logs addon alertmanager, fix the file, reload again.
• Port already allocated — the assigned port is held in pg.yaml and skipped for later instances; free it with pg addon remove first, or pick another with --port N.
• Remote API unreachable / unexpectedly reachable — on the default 0.0.0.0 a failure to reach it is host firewall, not bind address; conversely, if you did not want it reachable (and remember: reachability means anyone can create silences), --listen 127.0.0.1 --force.

Known limitations

• One-sided wiring. This addon neither injects an alerting: section into a prometheus.yml nor reads one from Prometheus — it only receives whatever is pushed to its API. The linking is done from the Prometheus side (pg addon install prometheus --alertmanager-url …, render mode; in --conf-file mode write the alerting: block yourself). An alertmanager-aware prometheus.yml is never inspected from here.
• No authentication — and its consequences are sharper than Prometheus’s. The API can create silences (mute on-call) and reload config, and POST /-/reload cannot be disabled in the image (pgcli just doesn’t use it). The only protection for a 0.0.0.0 bind is the network boundary; do not expose it on an untrusted network.
• Single instance / cluster disabled. --cluster.listen-address= is always passed empty, so HA gossip (TCP+UDP 9094) never opens. Clustering several pgcli-managed Alertmanagers is not supported in v1.
• Webhooks are not connectivity-probed at install (only their URL shape is validated). Reachability and payload acceptance surface at alert time, in the logs.
• pgcli renders or copies alertmanager.yml but does not validate it (there is no amtool check-config gate at install). A bad rendered config is impossible, but a bad --conf-file surfaces only in the container logs on reload.
• The host-side ownership chown is automatic under rootless podman (podman unshare chown) and when pgcli runs as root on a rootful host (direct chown). A rootful daemon talked to by a non-root pgcli (only via an explicitly configured remote socket) has no automatic path.
• On macOS the container runs inside the podman machine; it has been validated on Linux only.