BeeGFS
BeeGFS is a parallel filesystem that chose the easy route. It was born in 2004 at Fraunhofer ITWM in Germany — a research group that needed Lustre, found its deployment and operations too heavy, and wrote its own instead, with the goal set from day one: not the biggest scale, but the least human effort to deploy and run. In 2014 the core team spun out as the commercial company ThinkParQ, the product was renamed BeeGFS, and it grew from an institute-internal tool into the shared storage you find on many European universities and regional supercomputers.
It solves the “many machines, one high-throughput dataset” problem: a local filesystem lives on a single host, and NFS-style shares cap out quickly under concurrent large-file IO from many clients. Like Lustre, BeeGFS is a parallel filesystem — metadata separated from data, files striped across storage servers, clients reading data blocks directly once they know where they are — so aggregate bandwidth scales with the number of servers. But at every point where it could be complicated, it chose simple: service roles may share one machine, daemons self-register into the cluster when they start (no hand-synced configuration), growing means adding a box, and plain Ethernet works — InfiniBand is not mandatory. The trade: it does not chase extreme scale or the deepest tuning control; what it buys is that a research group or department cluster without a dedicated storage team can be up within a day or two.
Typical fit: HPC clusters from a few up to a couple hundred nodes, the data supply layer for AI training (GPUDirect Storage since 7.3 — storage-to-GPU direct paths, so large training/inference jobs read shared datasets directly), post-production editing media in film and video (many editors reading and writing large concurrent streams without fighting for bandwidth), and department-scale shared volumes of many large files. One-line selection rule: national-lab scale, thousands of nodes, a storage team on staff → Lustre; “usable today, no storage team” → BeeGFS.
Platform: Linux only. BeeGFS daemons run on host networking (BeeGFS speaks its own TCP/UDP protocol, not HTTP, so there is no container port-mapping to hide behind). macOS has no host networking under podman, so
pg addon install beegfsis refused there with a clear error.
The client is out of scope, on purpose. Mounting a BeeGFS filesystem needs the
beegfs-clientkernel module (DKMS). Kernel modules cannot be containerised, so pgcli neither installs nor mounts it — it manages the server services only (three containers on a converged host, the configured subset on a cross-host member).pg addon beegfs mount-hintprints the exact host-side config a machine with thebeegfs-client+beegfs-utilspackages needs. See Mounting a client.
Parameters
| Flag | Default | Description |
|---|---|---|
--name |
beegfs |
Filesystem name (map key); containers are <prefix>-<role> for each role this host runs |
--image |
8.4.0 |
Version suffix — the manager builds ghcr.io/thinkparq/beegfs-<role>:<version> from it (all roles share one tag) |
--data-dir |
<base>/addon/beegfs/<name> |
One dir for the whole filesystem; its services use <dir>/{mgmtd,meta,storage} |
--listen |
0.0.0.0 |
The address clients self-reference as sysMgmtdHost; with host networking the daemons actually bind every interface |
--password |
generated (mgmtd hosts) | The connection-auth secret (ConnAuth) shared by all daemons and every client; generated on first install only if this host runs mgmtd — a join host must pass the cluster’s existing value |
--roles |
all three | v2: which services THIS host runs — comma list of mgmtd,meta,storage; omitting it is the converged single-host filesystem |
--mgmtd-host |
— | v2: address of the mgmtd host this host’s meta/storage register against (required when --roles excludes mgmtd) |
--node-id |
1 |
v2: this host’s BeeGFS node ID (1..655); storage target IDs derive from it (2 → 201, 202, …). Must be unique per service type across hosts |
--storage-target |
single dir | v2: one host dir per storage target — repeat for multiple disks (one target each) |
--license-file |
— | Path to a ThinkParQ license .pem to carry into the mgmtd container (stored in pg.yaml, injected as BEEGFS_LICENSE_FILE_DATA). mgmtd host only — pgcli never generates a license; pg addon beegfs license shows/hot-swaps it later |
--beegfs-mgmtd-port |
8008 |
Move mgmtd’s BeeMsg port (the only movable meta/storage registration target); 0 = stock |
--beegfs-mgmtd-grpc-port |
8010 |
Move mgmtd’s gRPC port (what the beegfs CLI dials); 0 = stock |
--force |
— | Recreate this host’s containers even if they are already running |
There is deliberately no --port for meta or storage: see Ports.
Install
The install is fully non-interactive. It pulls the images for the roles this
host runs up front (so a mid-cluster network failure never leaves a
half-registered filesystem), then brings them up in dependency order:
mgmtd first, then meta, then storage — a join host with --roles storage just
runs its own step. meta/storage block until they log
Registration and management info download complete, the line that proves both
“mgmtd is reachable” and “this service joined”. Each container’s first boot
runs a beegfs-setup-<service> step (driven by beegfs_setup_N env vars the
image’s entrypoint consumes) that initialises its target directory; the marker
is idempotent, so a recreated container against a surviving data dir reuses the
existing target.
Re-running install is safe and cluster-granular (over this host’s roles):
- all of them up → skip (use
--forceto recreate, orpg addon beegfs check); - all of them down and present → just start the stored cluster (no pull, no recreate);
- anything else (partial, or a fresh install) →
EnsureClusterrecreates the set in order.
The install summary lists this host’s roles and their container names, the
images, the mgmtd address and its two ports (marked remote on a join host), the
node ID and data dirs, the conn-auth state, and a pointer to
pg addon beegfs mount-hint. If --listen is the wildcard 0.0.0.0 it prints
an exposure NOTE (a parallel filesystem with TLS off is reachable from any
machine that holds the conn-auth secret).
Cross-host clusters (v2)
A BeeGFS filesystem scales by adding hosts, not by hand-syncing config:
each service daemon starts with sysMgmtdHost=<mgmtd address> plus the shared
conn-auth secret and self-registers into the cluster — mgmtd is the
registry. pgcli mirrors that model exactly: there is no central deploy
command and pgcli never SSHes anywhere. Every host runs its own
pg addon install beegfs --roles <subset> with its own local pg.yaml, and
BeeGFS does the rest — the same declarative, per-host pattern Patroni cross-host
members use (where etcd, not pgcli, aggregates topology; here the role is
mgmtd’s own SQLite registry).
Three hosts, one filesystem:
Clients mount from vm01’s address as usual — cross-host changes nothing about the kernel-module client.
Rules and boundaries, honestly stated:
--mgmtd-hostis required whenever--rolesexcludes mgmtd (there is no localhost to fall back to), and the join must carry the cluster’s existing conn-auth — pgcli never generates a secret for a member of an existing cluster. A wrong secret fails the install in seconds with a pointer topg addon password beegfs, not after a registration timeout.--node-idmust be unique per service type across hosts (node IDs are per-type namespaces:meta_1andstorage_1coexist). It seeds the storage target IDs: node 2’s targets are 201, 202, … (target IDs derive asnode_id*100 + ordinal, capped at node_id 655).- Each
pg.yamlentry stores only what its own host runs;check,start/stop,listand autostart are all per-host views. The cluster’s full membership lives in mgmtd (beegfs ... node list). - mgmtd remains a single point: if it dies, existing client IO keeps
flowing (clients talk to meta/storage directly) but registration and admin
stop. A redundant management pair is an enterprise-license feature, not
pgcli wiring. The license rides on the mgmtd host alone — join hosts
neither store nor need a PEM (
join-hintsays so), since mgmtd verifies the certificate for the whole filesystem. - There is still one filesystem per host — cross-host is the scaling axis, not second-filesystems-per-host (the fixed 8005/8003 ports make those collide).
The data directories and persistence
A converged host owns three host directories, each bind-mounted into its own
container at /data/<role>:
--data-dir <dir> lays all three out as <dir>/{mgmtd,meta,storage} in one
flag — operators never address the roles separately. On a cross-host member
only its own role’s dir is used. Multiple storage targets (the
--storage-target repeat) each get their own host dir mounted at
/data/storage/<targetID> inside the single storage container; a single target
keeps the plain /data/storage path so a v1 store still revives unchanged
(a target directory records its own path internally, so revival must mount the
same path). The three images run as
root and own their data dirs, so unlike Gitea (uid 1000) or Grafana
(uid 472) there is no pre-chown dance — the host dirs are writable as-is.
All data survives container removal: pg addon remove beegfs without
--clean-data keeps the dirs, and reinstalling the same name revives
the filesystem (the install prints a NOTE when it detects an existing
mgmtd.sqlite). Revival keeps whatever registry and connection auth the store
already had — a new --password only reaches freshly-initialised daemons, not
a surviving SQLite registry.
Failure handling: striping for performance, ZFS for reliability
BeeGFS’s cross-node striping is a performance mechanism, not a redundancy one: in the default RAID0 layout, losing ANY one storage target loses the chunks it held — and on a striped file that means the file is corrupt. Node-level data redundancy is buddy mirroring, an enterprise-license feature. So the community edition splits the two concerns, and the storage side needs a local-failure scheme underneath it:
- Across nodes: striping for throughput. Each file’s chunks spread over
many targets on many hosts; aggregate bandwidth scales with the number of
servers. This is what
--num-targets/--chunk-sizetune. It buys speed, never safety. - Within each storage server: a resilient volume under every target. The
recommended baseline is ZFS, because a BeeGFS storage target is “just a
directory” and ZFS turns that directory into a self-healing store for free —
checksums against silent corruption, copy-on-write, snapshots, compression.
The daemon cannot tell the difference, so this layer needs zero pgcli or
BeeGFS involvement: build the pool, make a dataset per target, and point
--storage-targetat it.(mdadm RAID1/5/6 works too, but ZFS’s checksumming catches the silent bit rot a BeeGFS target would otherwise serve straight to clients.) One target per dataset keeps a disk fault entirely out of the BeeGFS target path. - Whole-node loss is still not survivable (community edition): a dead
server takes its storage targets down with it — local ZFS/RAID survives a
disk, not a burnt-out machine. Mitigations without a license: keep
meta/mgmtdservices duplicated across hosts (free — service redundancy, so the filesystem stays mounted and admin keeps working), snapshot the metadata, and treat storage data as rebuild-from-upstream where the workload allows (re-generable training datasets — the honest community trade-off). A temporary full-feature certificate (free, time-limited, one per system) or a paid enterprise license is the only way to get real cross-node mirroring. - Metadata deserves the same care: losing the meta target makes the whole tree unreadable regardless of data redundancy — keep its dir on a local mirror too (it is small). pgcli takes no BeeGFS backups, so protecting the mgmtd registry and meta target is host-level work: a ZFS snapshot of the volume holding the role dirs, or a quiesced file copy while the daemons are stopped.
Ports
The four BeeGFS listener ports are the stock protocol ports, not a pool:
Verified on a live host: BeeGFS 8.4’s meta and storage daemons accept no
listen-port override (connPortsList/connTcpPort are rejected as invalid,
and connMetaPort is accepted but breaks registration). Only mgmtd’s two ports
can move, and moving mgmtd’s BeeMsg port re-targets meta/storage registration
automatically (the manager emits connMgmtdPort=<port> into both). The direct
consequence: exactly one BeeGFS filesystem can exist per host — a second
one could never bind 8005/8003 regardless of what pgcli allocated, so
EnsureCluster guards it out with an explicit error, and the CLI rejects a
hand-pinned non-default meta_port/storage_port in pg.yaml with the real
reason. There is therefore no beegfs_start_port pool — unlike every other
addon, BeeGFS ports are fixed, not allocated.
Connection auth and TLS
Every daemon in the filesystem and every client share one
connection-authentication secret (ConnAuth, delivered as the
CONN_AUTH_FILE_DATA env var that the image’s init writes to
/etc/beegfs/conn.auth). It is generated on first install (20 random chars)
only on a host that runs mgmtd, printed once, and stored in pg.yaml —
retrieve it with pg addon password beegfs. A cross-host join host must be
given that existing value via --password (see
Cross-host clusters); pgcli never generates a secret
for a member of an existing cluster. Like every pgcli secret it is never
defaulted into the config by ApplyDefaults.
TLS is disabled on mgmtd (--tls-disable true, the explicit v1 decision).
This means only clients that pass --tls-disable too can join, and it is a
real exposure consideration: a wildcard --listen 0.0.0.0 with TLS off leaves
ports 8008/8010/8005/8003 open to any machine on the network that holds the
conn-auth secret. For anything not on a private network, either bind loopback
(--listen 127.0.0.1) or firewall those four ports.
Licensing: as of BeeGFS 8 a license is expected on every deployment.
pgcli never generates a license (same principle as Gitea’s crypto secrets)
— but it carries one you obtained: the PEM content is stored in pg.yaml
(so it survives and travels with your config backups) and injected into the
mgmtd container as BEEGFS_LICENSE_FILE_DATA, which the official image’s init
writes to /etc/beegfs/license.pem and mgmtd’s start.sh passes as
--license-cert-file. Only the mgmtd host may carry it: its image is the only
one shipping the license library, and its daemon verifies the certificate for
the whole filesystem (the install refuses --license-file on a join host).
What a license costs, by tier (all three self-serve via ThinkParQ’s portal):
| Tier | Price | What it does |
|---|---|---|
| community | free | Lifts the unlicensed limits (8.4’s five-client mount cap, quota warnings); has a capacity ceiling and does NOT unlock enterprise features |
| temporary | free trial | Unlocks everything for a limited period; since 8.4 one system can hold only one |
| enterprise | paid | Storage pools, quotas, mirroring (buddy groups), the HA management pair |
Mirroring (buddy groups) is an ENTERPRISE feature — it needs a paid license. Without one, mgmtd refuses every mirroring request (
beegfs mirror create→ “no license loaded” — verified on a live cluster). The free community license does NOT unlock it; only a paid enterprise license (or a temporary full-feature trial certificate) does. Data redundancy in the community edition is limited to running multiple meta/mgmtd daemons — that is service redundancy, not mirrored storage targets.
Obtaining a PEM is out-of-band and takes a minute for a community license:
beegfs license --get prints a system-bound URL → open it in a browser →
download the .pem. Then hand it to pgcli — at install time
(--license-file license.pem) or later with pg addon beegfs license --set
(see License status and hot-swap).
The 8.4 realities of running unlicensed: mgmtd logs a quota warning,
every beegfs command nags, client mounts are capped at five (a sixth
mount is rejected), and enterprise features are refused outright (e.g.
beegfs mirror create fails with “no license loaded”). Any valid license —
even the free community one — lifts the warnings and the mount cap.
Mounting a client
The client is a host kernel module — pgcli does not install it. To mount a filesystem from a Debian/Ubuntu host, add ThinkParQ’s apt repository and install the client packages yourself. For 8.4 on Debian 12 (bookworm) — the exact commands, verified on a test host:
On RHEL-family hosts the same repo lives under
https://www.beegfs.io/release/beegfs_8.4/dists/ as an .repo file
(rpm --import …/GPG-KEY-beegfs + wget …/beegfs-rhel9.repo -O /etc/yum.repos.d/beegfs.repo). Always take the URLs for your BeeGFS version
and distribution from the ThinkParQ download
page — pinning the server to 8.4 means clients
should mount from the matching 8.4 repo.
beegfs-client depends on build-essential and recommends kernel headers —
DKMS compiles the beegfs module against linux-headers-$(uname -r) at install
time, so it needs the running kernel’s headers present and will rebuild on every
kernel update.
Then write the config pg addon beegfs mount-hint prints — it renders the three
files verbatim:
conn.auth gotcha: the client’s connection-auth file must end in a newline.
echo <secret> > /etc/beegfs/conn.authworks;printf <secret> > …(no trailing newline) is rejected withtried to authenticate stream with wrong secret— the trailing byte is part of what both sides compare.
Then mount and verify:
Because the cluster runs with TLS disabled, every beegfs CLI call needs the
disabled-TLS form pointed at the gRPC port; node list shows all four node
types (management, meta, storage, and this client once mounted):
Common client-side operations (all verified against 8.4 — the beegfs CLI is
the 8.x rewrite; 7.x beegfs-*-ctl incantations do not apply):
mount-hint is pure local rendering — it never touches podman, since the client
is out of pgcli’s management. It renders the mgmtd address from the config
entry: on a cross-host member that is the stored --mgmtd-host; when the local
--listen is the wildcard, it prints <mgmtd-host-ip> as a placeholder
(0.0.0.0 is not dial-able) rather than a value clients cannot connect to.
Health check
Reports the aggregate state of this host’s roles (running / stopped /
partial (mgmtd,meta)) and, per service, whether its container is up and
whether it logged its readiness marker — mgmtd Waiting for shutdown signal
(serving), meta/storage Registration and management info download complete
(joined). None of the images ships beegfs-ctl/check-servers, so the
daemons’ own logs are the authoritative in-container signal; the cluster-wide
membership lives in mgmtd — for the whole picture use the beegfs gRPC CLI
above (node list), and for a stopped cluster check still works (it reports
state, it does not require running).
Retrieving the connection-auth secret
pg addon list shows only a presence marker (Auth: none / set (connection-auth)) and prints the secret itself only with --show-password, so
it never leaks to a casual listing.
License status and hot-swap
Run on the mgmtd host (the license lives there — see
Connection auth and TLS). Without flags it reports
both sides: whether pg.yaml carries a PEM, and what mgmtd itself logged when
loading it — Successfully loaded license certificate for a valid one, or the
verification errors when mgmtd rejects a bad/expired certificate. pgcli only
shape-checks the PEM (BEGIN/END block); the real verification is always
mgmtd’s.
--set <pem> reads the file (shape-checked for a BEGIN/END PEM block — the
cryptographic verification is mgmtd’s, not pgcli’s), stores the content in
pg.yaml, and recreates the mgmtd container so the new env lands (podman
env only exists from creation). meta/storage and every client re-register on
their own — the swap is a few seconds and IO-safe. --clear removes the PEM
and reloads unlicensed. --set and --clear are mutually exclusive; passing
--license-file at install is the same load without the recreate (the container
is being created anyway).
List
The BeeGFS block (“Infra add-ons (beegfs)”) prints one entry per filesystem
with its aggregate state over this host’s roles, the role list, the mgmtd
address and its two ports (marked remote on a join host), the fixed
meta/storage ports, conn-auth presence, image, the data dirs of the roles this
host runs (one line per storage target), and the container names. When podman
is unavailable it still lists the configured entries (state shown as n/a).
Start and stop
Containers run with podman’s unless-stopped restart policy, so they come back
with the podman service after a host reboot. start/stop manage this host’s
roles explicitly, respecting dependency order:
A cross-host member runs only its own subset (a --roles storage host starts
and stops just storage).
Auto-start on Boot
BeeGFS is the only cluster-granular autostart: pg start --autostart brings
up the filesystem as this host runs it (mgmtd → meta → storage in order, or the
configured subset), not one container.
Because the images run as root and own their dirs, an autostart needs no host
privileges beyond podman itself (no chown step like Gitea’s). The client on
each mounting host re-attaches on its own via beegfs-mounts.conf — pgcli
never manages that. See Autostart for the full
mechanism, status, and disabling.
Remove
Remove tears down this host’s roles in reverse order (storage → meta → mgmtd). --clean-data
refuses to delete a directory that is still a mount point. A client mountpoint
pointing at this host must be unmounted either way — pgcli cannot reach the
client (it is not pgcli’s), so unmount it on the client host first.
Logs
A filesystem spans up to three containers, so pg logs addon beegfs takes
--service to pick one (default mgmtd):
--service is only valid for beegfs (a value like meta against another
addon type is rejected).
Troubleshooting
install beegfs --name fs2refuses with “a BeeGFS filesystem already exists on this host” — this is the one-filesystem-per-host rule, not a bug: meta (8005) and storage (8003) cannot move their listen ports in 8.4, so a second filesystem on the same host could never bind them. Grow by adding roles on ANOTHER host (pg addon beegfs join-hintrenders the commands), orpg addon removethe first one.- meta/storage stuck “not registered” in
check— mgmtd must be serving first; if mgmtd’s container died, both block on registration untilbeegfsSetupTimeout(60 s) fails the install. Checkpg logs addon beegfs --service mgmtdfor why mgmtd exited (port already in use on the host, or an un-writable data dir). installrefuses on themeta_port/storage_portconfig check — a non-default value was hand-edited intopg.yaml. The daemons cannot bind those, so remove the pinned values and let them fall back to 8005/8003.- Client cannot mount /
beegfsCLI says TLS error — the cluster runs with mgmtd TLS disabled; every client and everybeegfsCLI call needs--tls-disabletoo, and/etc/beegfs/conn.authmust be the exactConnAuthfrompg addon password beegfs. - A join host fails fast with “wrong secret” — the
--passwordpassed to its install is not the cluster’s connection-auth value (BeeGFS logstried to authenticate stream with wrong secret). The install aborts in seconds rather than timing out; re-run with the value frompg addon password beegfson the mgmtd host. pg addon beegfs licenseprints a verification error from mgmtd — the stored PEM is not valid for this system (certificates are system-bound — request one withbeegfs license --geton the mgmtd host) or has expired. The container still runs and community functionality is unaffected; mgmtd simply behaves as unlicensed.--clear(or--setwith the right PEM) fixes the config side.- Clone/mount to
localhostfails from another machine — a wildcard--listen 0.0.0.0renderssysMgmtdHostas a placeholder inmount-hint; install with--listen <reachable-ip>(with--force) so clients get a real mgmtd address.
Known limitations
- Linux only. Host networking is mandatory and podman provides it on Linux only; installation is refused on macOS. No bridge fallback exists because BeeGFS is not HTTP.
- One filesystem per host, fixed ports. pgcli pins the stock ports
(8008/8010/8005/8003); meta and storage have no movable listen port in 8.4,
so a single host holds a single filesystem. Scaling out is cross-host
(
--roles), not a second filesystem. - The client is never installed or mounted by pgcli. It is a kernel module
(DKMS) that cannot be containerised; pgcli manages only the server
containers and documents the mount via
mount-hint. - TLS off; licenses are carried, not issued. mgmtd runs
--tls-disable true; thebeegfsgRPC CLI and clients must match. pgcli never generates a license and never handles renewal — it carries a ThinkParQ PEM you obtained out of band (beegfs license --getprints the system-bound URL) to the mgmtd container, andpg addon beegfs licenseshows/hot-swaps it. Mirroring, pools, and quota stay enterprise-license features you still have to buy; an unlicensed 8.4 caps client mounts at five. - Cross-host deployment is per-host and manual by design (v2). Each host
runs its own
pg addon install beegfs --roles ...; pgcli never SSHes and has no “deploy the whole cluster from one box” command. Growing means running one command on the new host (the conn-auth secret is carried by hand — seejoin-hint), and every host’spg.yamldescribes only its own slice. - mgmtd is a single point. The management daemon is the cluster’s registry; if it stops, new registrations and admin operations stop (existing mounted clients keep doing IO against meta/storage). A redundant management pair is a BeeGFS enterprise-license feature and is outside pgcli’s v1/v2 wiring.