Skip to content

BeeGFS

Run a BeeGFS parallel filesystem as a managed addon.

BeeGFS is a parallel filesystem that chose the easy route. It was born in 2004 at Fraunhofer ITWM in Germany — a research group that needed Lustre, found its deployment and operations too heavy, and wrote its own instead, with the goal set from day one: not the biggest scale, but the least human effort to deploy and run. In 2014 the core team spun out as the commercial company ThinkParQ, the product was renamed BeeGFS, and it grew from an institute-internal tool into the shared storage you find on many European universities and regional supercomputers.

It solves the “many machines, one high-throughput dataset” problem: a local filesystem lives on a single host, and NFS-style shares cap out quickly under concurrent large-file IO from many clients. Like Lustre, BeeGFS is a parallel filesystem — metadata separated from data, files striped across storage servers, clients reading data blocks directly once they know where they are — so aggregate bandwidth scales with the number of servers. But at every point where it could be complicated, it chose simple: service roles may share one machine, daemons self-register into the cluster when they start (no hand-synced configuration), growing means adding a box, and plain Ethernet works — InfiniBand is not mandatory. The trade: it does not chase extreme scale or the deepest tuning control; what it buys is that a research group or department cluster without a dedicated storage team can be up within a day or two.

Typical fit: HPC clusters from a few up to a couple hundred nodes, the data supply layer for AI training (GPUDirect Storage since 7.3 — storage-to-GPU direct paths, so large training/inference jobs read shared datasets directly), post-production editing media in film and video (many editors reading and writing large concurrent streams without fighting for bandwidth), and department-scale shared volumes of many large files. One-line selection rule: national-lab scale, thousands of nodes, a storage team on staff → Lustre; “usable today, no storage team” → BeeGFS.

Platform: Linux only. BeeGFS daemons run on host networking (BeeGFS speaks its own TCP/UDP protocol, not HTTP, so there is no container port-mapping to hide behind). macOS has no host networking under podman, so pg addon install beegfs is refused there with a clear error.

The client is out of scope, on purpose. Mounting a BeeGFS filesystem needs the beegfs-client kernel module (DKMS). Kernel modules cannot be containerised, so pgcli neither installs nor mounts it — it manages the server services only (three containers on a converged host, the configured subset on a cross-host member). pg addon beegfs mount-hint prints the exact host-side config a machine with the beegfs-client + beegfs-utils packages needs. See Mounting a client.

Parameters

Flag Default Description
--name beegfs Filesystem name (map key); containers are <prefix>-<role> for each role this host runs
--image 8.4.0 Version suffix — the manager builds ghcr.io/thinkparq/beegfs-<role>:<version> from it (all roles share one tag)
--data-dir <base>/addon/beegfs/<name> One dir for the whole filesystem; its services use <dir>/{mgmtd,meta,storage}
--listen 0.0.0.0 The address clients self-reference as sysMgmtdHost; with host networking the daemons actually bind every interface
--password generated (mgmtd hosts) The connection-auth secret (ConnAuth) shared by all daemons and every client; generated on first install only if this host runs mgmtd — a join host must pass the cluster’s existing value
--roles all three v2: which services THIS host runs — comma list of mgmtd,meta,storage; omitting it is the converged single-host filesystem
--mgmtd-host — v2: address of the mgmtd host this host’s meta/storage register against (required when --roles excludes mgmtd)
--node-id 1 v2: this host’s BeeGFS node ID (1..655); storage target IDs derive from it (2 → 201, 202, …). Must be unique per service type across hosts
--storage-target single dir v2: one host dir per storage target — repeat for multiple disks (one target each)
--license-file — Path to a ThinkParQ license .pem to carry into the mgmtd container (stored in pg.yaml, injected as BEEGFS_LICENSE_FILE_DATA). mgmtd host only — pgcli never generates a license; pg addon beegfs license shows/hot-swaps it later
--beegfs-mgmtd-port 8008 Move mgmtd’s BeeMsg port (the only movable meta/storage registration target); 0 = stock
--beegfs-mgmtd-grpc-port 8010 Move mgmtd’s gRPC port (what the beegfs CLI dials); 0 = stock
--force — Recreate this host’s containers even if they are already running

There is deliberately no --port for meta or storage: see Ports.

Install

pg addon install beegfs              # default filesystem named "beegfs"
pg addon install beegfs --name fs1
pg addon install beegfs --listen 10.0.0.5   # advertise a reachable mgmtd address

The install is fully non-interactive. It pulls the images for the roles this host runs up front (so a mid-cluster network failure never leaves a half-registered filesystem), then brings them up in dependency order: mgmtd first, then meta, then storage — a join host with --roles storage just runs its own step. meta/storage block until they log Registration and management info download complete, the line that proves both “mgmtd is reachable” and “this service joined”. Each container’s first boot runs a beegfs-setup-<service> step (driven by beegfs_setup_N env vars the image’s entrypoint consumes) that initialises its target directory; the marker is idempotent, so a recreated container against a surviving data dir reuses the existing target.

Re-running install is safe and cluster-granular (over this host’s roles):

  • all of them up → skip (use --force to recreate, or pg addon beegfs check);
  • all of them down and present → just start the stored cluster (no pull, no recreate);
  • anything else (partial, or a fresh install) → EnsureCluster recreates the set in order.

The install summary lists this host’s roles and their container names, the images, the mgmtd address and its two ports (marked remote on a join host), the node ID and data dirs, the conn-auth state, and a pointer to pg addon beegfs mount-hint. If --listen is the wildcard 0.0.0.0 it prints an exposure NOTE (a parallel filesystem with TLS off is reachable from any machine that holds the conn-auth secret).

Cross-host clusters (v2)

A BeeGFS filesystem scales by adding hosts, not by hand-syncing config: each service daemon starts with sysMgmtdHost=<mgmtd address> plus the shared conn-auth secret and self-registers into the cluster — mgmtd is the registry. pgcli mirrors that model exactly: there is no central deploy command and pgcli never SSHes anywhere. Every host runs its own pg addon install beegfs --roles <subset> with its own local pg.yaml, and BeeGFS does the rest — the same declarative, per-host pattern Patroni cross-host members use (where etcd, not pgcli, aggregates topology; here the role is mgmtd’s own SQLite registry).

Three hosts, one filesystem:

# vm01 — management + metadata on this host; conn-auth generated here:
pg addon install beegfs --name fs1 --roles mgmtd,meta --listen 192.0.2.10

# render the verbatim join commands (secret included) from vm01:
pg addon beegfs join-hint --name fs1

# vm02 — two data disks, one storage target each:
pg addon install beegfs --name fs1 --roles storage --mgmtd-host 192.0.2.10 \
    --node-id 2 --password <conn-auth> \
    --storage-target /mnt/d1 --storage-target /mnt/d2

# vm03 — another storage node (or --roles meta to move metadata off vm01):
pg addon install beegfs --name fs1 --roles storage --mgmtd-host 192.0.2.10 \
    --node-id 3 --password <conn-auth> --storage-target /mnt/d3

Clients mount from vm01’s address as usual — cross-host changes nothing about the kernel-module client.

Rules and boundaries, honestly stated:

  • --mgmtd-host is required whenever --roles excludes mgmtd (there is no localhost to fall back to), and the join must carry the cluster’s existing conn-auth — pgcli never generates a secret for a member of an existing cluster. A wrong secret fails the install in seconds with a pointer to pg addon password beegfs, not after a registration timeout.
  • --node-id must be unique per service type across hosts (node IDs are per-type namespaces: meta_1 and storage_1 coexist). It seeds the storage target IDs: node 2’s targets are 201, 202, … (target IDs derive as node_id*100 + ordinal, capped at node_id 655).
  • Each pg.yaml entry stores only what its own host runs; check, start/stop, list and autostart are all per-host views. The cluster’s full membership lives in mgmtd (beegfs ... node list).
  • mgmtd remains a single point: if it dies, existing client IO keeps flowing (clients talk to meta/storage directly) but registration and admin stop. A redundant management pair is an enterprise-license feature, not pgcli wiring. The license rides on the mgmtd host alone — join hosts neither store nor need a PEM (join-hint says so), since mgmtd verifies the certificate for the whole filesystem.
  • There is still one filesystem per host — cross-host is the scaling axis, not second-filesystems-per-host (the fixed 8005/8003 ports make those collide).

The data directories and persistence

A converged host owns three host directories, each bind-mounted into its own container at /data/<role>:

<base>/addon/beegfs/<name>/mgmtd     mgmtd.sqlite — the node/target registry
<base>/addon/beegfs/<name>/meta      the metadata target
<base>/addon/beegfs/<name>/storage   the storage target (the actual file data)

--data-dir <dir> lays all three out as <dir>/{mgmtd,meta,storage} in one flag — operators never address the roles separately. On a cross-host member only its own role’s dir is used. Multiple storage targets (the --storage-target repeat) each get their own host dir mounted at /data/storage/<targetID> inside the single storage container; a single target keeps the plain /data/storage path so a v1 store still revives unchanged (a target directory records its own path internally, so revival must mount the same path). The three images run as root and own their data dirs, so unlike Gitea (uid 1000) or Grafana (uid 472) there is no pre-chown dance — the host dirs are writable as-is.

All data survives container removal: pg addon remove beegfs without --clean-data keeps the dirs, and reinstalling the same name revives the filesystem (the install prints a NOTE when it detects an existing mgmtd.sqlite). Revival keeps whatever registry and connection auth the store already had — a new --password only reaches freshly-initialised daemons, not a surviving SQLite registry.

Failure handling: striping for performance, ZFS for reliability

BeeGFS’s cross-node striping is a performance mechanism, not a redundancy one: in the default RAID0 layout, losing ANY one storage target loses the chunks it held — and on a striped file that means the file is corrupt. Node-level data redundancy is buddy mirroring, an enterprise-license feature. So the community edition splits the two concerns, and the storage side needs a local-failure scheme underneath it:

  • Across nodes: striping for throughput. Each file’s chunks spread over many targets on many hosts; aggregate bandwidth scales with the number of servers. This is what --num-targets/--chunk-size tune. It buys speed, never safety.
  • Within each storage server: a resilient volume under every target. The recommended baseline is ZFS, because a BeeGFS storage target is “just a directory” and ZFS turns that directory into a self-healing store for free — checksums against silent corruption, copy-on-write, snapshots, compression. The daemon cannot tell the difference, so this layer needs zero pgcli or BeeGFS involvement: build the pool, make a dataset per target, and point --storage-target at it.
    # two disks → a mirror vdev (ZFS's RAID1; capacity = one disk, survives
    # either disk dying). raidz1/2 need 3+/4+ disks.
    zpool create bgtank mirror /dev/disk1 /dev/disk2
    zfs create -o recordsize=1M -o compression=lz4 bgtank/t1
    # ...then:  pg addon install beegfs --roles storage ... \
    #              --storage-target /bgtank/t1
    (mdadm RAID1/5/6 works too, but ZFS’s checksumming catches the silent bit rot a BeeGFS target would otherwise serve straight to clients.) One target per dataset keeps a disk fault entirely out of the BeeGFS target path.
  • Whole-node loss is still not survivable (community edition): a dead server takes its storage targets down with it — local ZFS/RAID survives a disk, not a burnt-out machine. Mitigations without a license: keep meta/mgmtd services duplicated across hosts (free — service redundancy, so the filesystem stays mounted and admin keeps working), snapshot the metadata, and treat storage data as rebuild-from-upstream where the workload allows (re-generable training datasets — the honest community trade-off). A temporary full-feature certificate (free, time-limited, one per system) or a paid enterprise license is the only way to get real cross-node mirroring.
  • Metadata deserves the same care: losing the meta target makes the whole tree unreadable regardless of data redundancy — keep its dir on a local mirror too (it is small). pgcli takes no BeeGFS backups, so protecting the mgmtd registry and meta target is host-level work: a ZFS snapshot of the volume holding the role dirs, or a quiesced file copy while the daemons are stopped.

Ports

The four BeeGFS listener ports are the stock protocol ports, not a pool:

mgmtd   8008  BeeMsg — meta/storage register here   (movable: --beemsg-port)
mgmtd   8010  gRPC   — the `beegfs` CLI dials here  (movable: --grpc-port)
meta    8005  metadata service                      (FIXED)
storage 8003  storage service                       (FIXED)

Verified on a live host: BeeGFS 8.4’s meta and storage daemons accept no listen-port override (connPortsList/connTcpPort are rejected as invalid, and connMetaPort is accepted but breaks registration). Only mgmtd’s two ports can move, and moving mgmtd’s BeeMsg port re-targets meta/storage registration automatically (the manager emits connMgmtdPort=<port> into both). The direct consequence: exactly one BeeGFS filesystem can exist per host — a second one could never bind 8005/8003 regardless of what pgcli allocated, so EnsureCluster guards it out with an explicit error, and the CLI rejects a hand-pinned non-default meta_port/storage_port in pg.yaml with the real reason. There is therefore no beegfs_start_port pool — unlike every other addon, BeeGFS ports are fixed, not allocated.

Connection auth and TLS

Every daemon in the filesystem and every client share one connection-authentication secret (ConnAuth, delivered as the CONN_AUTH_FILE_DATA env var that the image’s init writes to /etc/beegfs/conn.auth). It is generated on first install (20 random chars) only on a host that runs mgmtd, printed once, and stored in pg.yaml — retrieve it with pg addon password beegfs. A cross-host join host must be given that existing value via --password (see Cross-host clusters); pgcli never generates a secret for a member of an existing cluster. Like every pgcli secret it is never defaulted into the config by ApplyDefaults.

TLS is disabled on mgmtd (--tls-disable true, the explicit v1 decision). This means only clients that pass --tls-disable too can join, and it is a real exposure consideration: a wildcard --listen 0.0.0.0 with TLS off leaves ports 8008/8010/8005/8003 open to any machine on the network that holds the conn-auth secret. For anything not on a private network, either bind loopback (--listen 127.0.0.1) or firewall those four ports.

Licensing: as of BeeGFS 8 a license is expected on every deployment. pgcli never generates a license (same principle as Gitea’s crypto secrets) — but it carries one you obtained: the PEM content is stored in pg.yaml (so it survives and travels with your config backups) and injected into the mgmtd container as BEEGFS_LICENSE_FILE_DATA, which the official image’s init writes to /etc/beegfs/license.pem and mgmtd’s start.sh passes as --license-cert-file. Only the mgmtd host may carry it: its image is the only one shipping the license library, and its daemon verifies the certificate for the whole filesystem (the install refuses --license-file on a join host).

What a license costs, by tier (all three self-serve via ThinkParQ’s portal):

Tier Price What it does
community free Lifts the unlicensed limits (8.4’s five-client mount cap, quota warnings); has a capacity ceiling and does NOT unlock enterprise features
temporary free trial Unlocks everything for a limited period; since 8.4 one system can hold only one
enterprise paid Storage pools, quotas, mirroring (buddy groups), the HA management pair

Mirroring (buddy groups) is an ENTERPRISE feature — it needs a paid license. Without one, mgmtd refuses every mirroring request (beegfs mirror create → “no license loaded” — verified on a live cluster). The free community license does NOT unlock it; only a paid enterprise license (or a temporary full-feature trial certificate) does. Data redundancy in the community edition is limited to running multiple meta/mgmtd daemons — that is service redundancy, not mirrored storage targets.

Obtaining a PEM is out-of-band and takes a minute for a community license: beegfs license --get prints a system-bound URL → open it in a browser → download the .pem. Then hand it to pgcli — at install time (--license-file license.pem) or later with pg addon beegfs license --set (see License status and hot-swap).

The 8.4 realities of running unlicensed: mgmtd logs a quota warning, every beegfs command nags, client mounts are capped at five (a sixth mount is rejected), and enterprise features are refused outright (e.g. beegfs mirror create fails with “no license loaded”). Any valid license — even the free community one — lifts the warnings and the mount cap.

Mounting a client

The client is a host kernel module — pgcli does not install it. To mount a filesystem from a Debian/Ubuntu host, add ThinkParQ’s apt repository and install the client packages yourself. For 8.4 on Debian 12 (bookworm) — the exact commands, verified on a test host:

# 1. the BeeGFS signing key (fingerprint 0683 AF2B 9AC5 BC7D 59B2 D623 40CE 4E24 3CD5 0804)
wget https://www.beegfs.io/release/beegfs_8.4/gpg/GPG-KEY-beegfs -O /etc/apt/trusted.gpg.d/beegfs.asc
# 2. the repo file (bookworm = Debian 12; use beegfs-trixie.list for Debian 13,
#    beegfs-jammy/noble.list for Ubuntu, …)
wget https://www.beegfs.io/release/beegfs_8.4/dists/beegfs-bookworm.list -O /etc/apt/sources.list.d/beegfs.list
apt update
# 3. client + tools; the module is DKMS-built against the kernel headers
apt install beegfs-client beegfs-utils beegfs-tools linux-headers-$(uname -r)

On RHEL-family hosts the same repo lives under https://www.beegfs.io/release/beegfs_8.4/dists/ as an .repo file (rpm --import …/GPG-KEY-beegfs + wget …/beegfs-rhel9.repo -O /etc/yum.repos.d/beegfs.repo). Always take the URLs for your BeeGFS version and distribution from the ThinkParQ download page — pinning the server to 8.4 means clients should mount from the matching 8.4 repo.

beegfs-client depends on build-essential and recommends kernel headers — DKMS compiles the beegfs module against linux-headers-$(uname -r) at install time, so it needs the running kernel’s headers present and will rebuild on every kernel update.

Then write the config pg addon beegfs mount-hint prints — it renders the three files verbatim:

pg addon beegfs mount-hint --name fs1
1. /etc/beegfs/conn.auth (mode 0600) — the SAME secret as the servers
2. /etc/beegfs/beegfs-client.conf — set in the main section:
     sysMgmtdHost=<mgmtd-host>          # + connMgmtdPort=<port> only if mgmtd's BeeMsg port was moved
3. /etc/beegfs/beegfs-mounts.conf (one mount per line, fields:
   <mountpoint> <client-conf> <fstype>):
     /mnt/beegfs /etc/beegfs/beegfs-client.conf beegfs

conn.auth gotcha: the client’s connection-auth file must end in a newline. echo <secret> > /etc/beegfs/conn.auth works; printf <secret> > … (no trailing newline) is rejected with tried to authenticate stream with wrong secret — the trailing byte is part of what both sides compare.

Then mount and verify:

systemctl restart beegfs-client            # reads beegfs-mounts.conf and mounts
df -h /mnt/beegfs
echo hi > /mnt/beegfs/t && cat /mnt/beegfs/t   # end-to-end write/read

Because the cluster runs with TLS disabled, every beegfs CLI call needs the disabled-TLS form pointed at the gRPC port; node list shows all four node types (management, meta, storage, and this client once mounted):

beegfs --tls-disable --mgmtd-addr=<mgmtd-host>:8010 node list

Common client-side operations (all verified against 8.4 — the beegfs CLI is the 8.x rewrite; 7.x beegfs-*-ctl incantations do not apply):

B="beegfs --tls-disable --mgmtd-addr=<mgmtd-host>:8010"

$B node list                          # every node: mg / meta / storage / client
$B target list                        # storage targets, their node and state
$B entry info /mnt/beegfs/big.bin     # one file's stripe pattern and the
                                      # targets its chunks landed on
$B entry set /mnt/beegfs/data --num-targets 3 --chunk-size 4Mi
                                      # new files under data/ stripe across 3
                                      # targets, 4 MiB chunks (dirs set the
                                      # pattern for what is created after)
$B entry set /mnt/beegfs/data --num-targets 0   # back to the cluster default
$B health check                       # cluster-wide health report
$B stats server                       # live IO counters per server (8.x stats)

mount-hint is pure local rendering — it never touches podman, since the client is out of pgcli’s management. It renders the mgmtd address from the config entry: on a cross-host member that is the stored --mgmtd-host; when the local --listen is the wildcard, it prints <mgmtd-host-ip> as a placeholder (0.0.0.0 is not dial-able) rather than a value clients cannot connect to.

Health check

pg addon beegfs check             # default name
pg addon beegfs check --name fs1

Reports the aggregate state of this host’s roles (running / stopped / partial (mgmtd,meta)) and, per service, whether its container is up and whether it logged its readiness marker — mgmtd Waiting for shutdown signal (serving), meta/storage Registration and management info download complete (joined). None of the images ships beegfs-ctl/check-servers, so the daemons’ own logs are the authoritative in-container signal; the cluster-wide membership lives in mgmtd — for the whole picture use the beegfs gRPC CLI above (node list), and for a stopped cluster check still works (it reports state, it does not require running).

Retrieving the connection-auth secret

pg addon password beegfs             # the "beegfs" filesystem
pg addon password beegfs --name fs1

pg addon list shows only a presence marker (Auth: none / set (connection-auth)) and prints the secret itself only with --show-password, so it never leaks to a casual listing.

License status and hot-swap

pg addon beegfs license                     # what does mgmtd think of the cert?
pg addon beegfs license --name fs1 --set /path/license.pem   # load/replace, reloads mgmtd
pg addon beegfs license --name fs1 --clear                    # drop it again

Run on the mgmtd host (the license lives there — see Connection auth and TLS). Without flags it reports both sides: whether pg.yaml carries a PEM, and what mgmtd itself logged when loading it — Successfully loaded license certificate for a valid one, or the verification errors when mgmtd rejects a bad/expired certificate. pgcli only shape-checks the PEM (BEGIN/END block); the real verification is always mgmtd’s.

--set <pem> reads the file (shape-checked for a BEGIN/END PEM block — the cryptographic verification is mgmtd’s, not pgcli’s), stores the content in pg.yaml, and recreates the mgmtd container so the new env lands (podman env only exists from creation). meta/storage and every client re-register on their own — the swap is a few seconds and IO-safe. --clear removes the PEM and reloads unlicensed. --set and --clear are mutually exclusive; passing --license-file at install is the same load without the recreate (the container is being created anyway).

List

pg addon list

The BeeGFS block (“Infra add-ons (beegfs)”) prints one entry per filesystem with its aggregate state over this host’s roles, the role list, the mgmtd address and its two ports (marked remote on a join host), the fixed meta/storage ports, conn-auth presence, image, the data dirs of the roles this host runs (one line per storage target), and the container names. When podman is unavailable it still lists the configured entries (state shown as n/a).

Start and stop

Containers run with podman’s unless-stopped restart policy, so they come back with the podman service after a host reboot. start/stop manage this host’s roles explicitly, respecting dependency order:

pg addon stop beegfs     # storage → meta → mgmtd (reverse order, targets stay GOOD)
pg addon start beegfs    # mgmtd → meta → storage

A cross-host member runs only its own subset (a --roles storage host starts and stops just storage).

Auto-start on Boot

pg autostart enable --beegfs --name fs1

BeeGFS is the only cluster-granular autostart: pg start --autostart brings up the filesystem as this host runs it (mgmtd → meta → storage in order, or the configured subset), not one container. Because the images run as root and own their dirs, an autostart needs no host privileges beyond podman itself (no chown step like Gitea’s). The client on each mounting host re-attaches on its own via beegfs-mounts.conf — pgcli never manages that. See Autostart for the full mechanism, status, and disabling.

Remove

pg addon remove beegfs                # keeps the data dirs — reinstall revives the filesystem
pg addon remove beegfs --clean-data   # deletes them too (the data is gone)

Remove tears down this host’s roles in reverse order (storage → meta → mgmtd). --clean-data refuses to delete a directory that is still a mount point. A client mountpoint pointing at this host must be unmounted either way — pgcli cannot reach the client (it is not pgcli’s), so unmount it on the client host first.

Logs

A filesystem spans up to three containers, so pg logs addon beegfs takes --service to pick one (default mgmtd):

pg logs addon beegfs                       # mgmtd (default)
pg logs addon beegfs --service meta -f     # follow the metadata daemon
pg logs addon beegfs --service storage -n 200

--service is only valid for beegfs (a value like meta against another addon type is rejected).

Troubleshooting

  • install beegfs --name fs2 refuses with “a BeeGFS filesystem already exists on this host” — this is the one-filesystem-per-host rule, not a bug: meta (8005) and storage (8003) cannot move their listen ports in 8.4, so a second filesystem on the same host could never bind them. Grow by adding roles on ANOTHER host (pg addon beegfs join-hint renders the commands), or pg addon remove the first one.
  • meta/storage stuck “not registered” in check — mgmtd must be serving first; if mgmtd’s container died, both block on registration until beegfsSetupTimeout (60 s) fails the install. Check pg logs addon beegfs --service mgmtd for why mgmtd exited (port already in use on the host, or an un-writable data dir).
  • install refuses on the meta_port/storage_port config check — a non-default value was hand-edited into pg.yaml. The daemons cannot bind those, so remove the pinned values and let them fall back to 8005/8003.
  • Client cannot mount / beegfs CLI says TLS error — the cluster runs with mgmtd TLS disabled; every client and every beegfs CLI call needs --tls-disable too, and /etc/beegfs/conn.auth must be the exact ConnAuth from pg addon password beegfs.
  • A join host fails fast with “wrong secret” — the --password passed to its install is not the cluster’s connection-auth value (BeeGFS logs tried to authenticate stream with wrong secret). The install aborts in seconds rather than timing out; re-run with the value from pg addon password beegfs on the mgmtd host.
  • pg addon beegfs license prints a verification error from mgmtd — the stored PEM is not valid for this system (certificates are system-bound — request one with beegfs license --get on the mgmtd host) or has expired. The container still runs and community functionality is unaffected; mgmtd simply behaves as unlicensed. --clear (or --set with the right PEM) fixes the config side.
  • Clone/mount to localhost fails from another machine — a wildcard --listen 0.0.0.0 renders sysMgmtdHost as a placeholder in mount-hint; install with --listen <reachable-ip> (with --force) so clients get a real mgmtd address.

Known limitations

  • Linux only. Host networking is mandatory and podman provides it on Linux only; installation is refused on macOS. No bridge fallback exists because BeeGFS is not HTTP.
  • One filesystem per host, fixed ports. pgcli pins the stock ports (8008/8010/8005/8003); meta and storage have no movable listen port in 8.4, so a single host holds a single filesystem. Scaling out is cross-host (--roles), not a second filesystem.
  • The client is never installed or mounted by pgcli. It is a kernel module (DKMS) that cannot be containerised; pgcli manages only the server containers and documents the mount via mount-hint.
  • TLS off; licenses are carried, not issued. mgmtd runs --tls-disable true; the beegfs gRPC CLI and clients must match. pgcli never generates a license and never handles renewal — it carries a ThinkParQ PEM you obtained out of band (beegfs license --get prints the system-bound URL) to the mgmtd container, and pg addon beegfs license shows/hot-swaps it. Mirroring, pools, and quota stay enterprise-license features you still have to buy; an unlicensed 8.4 caps client mounts at five.
  • Cross-host deployment is per-host and manual by design (v2). Each host runs its own pg addon install beegfs --roles ...; pgcli never SSHes and has no “deploy the whole cluster from one box” command. Growing means running one command on the new host (the conn-auth secret is carried by hand — see join-hint), and every host’s pg.yaml describes only its own slice.
  • mgmtd is a single point. The management daemon is the cluster’s registry; if it stops, new registrations and admin operations stop (existing mounted clients keep doing IO against meta/storage). A redundant management pair is a BeeGFS enterprise-license feature and is outside pgcli’s v1/v2 wiring.