Sextant Fleet Handbook

Sextant is a declarative control plane for fleets of NixOS devices. Configuration is data in a git overlay; Nix turns that data into system closures; devices pull and converge (comin). Sextant is the human and API surface that edits the data safely, proves it builds, stages the rollout, and reports what each device actually runs. It is not MDM: declarative pull, no live command channel, every change an audited git commit.

Sextant does not ship the devices’ operating system - your overlay does, on top of a NixOS core. The core it was built against is DAWO, the Dutch government’s open workplace image, and the two are developed together. Nothing here is specific to it: an overlay that publishes dawo.* options is what Sextant configures, whatever core provides them.

This handbook is for operators running a fleet and for engineers working on Sextant itself. It is built with mdBook and served self-hosted; it uses no external CDN.

The lifecycle at a glance

1. Set up an imaging station - a NUC or mini-PC that boots devices over the network and reports what it sees to the console.
2. Image a device - the console dispatches an image job; the station runs the install and reports progress until the device is on disk and converging.
3. Manage it - configuration flows organisation -> group -> device; every change passes the Nix gate and can be reviewed as a change-request.
4. Update it - a rollout ships a new revision in waves. A wave promotes on evidence: enough of its devices reachable for a percentage to mean anything, enough of those healthy on the new revision, and a soak on top. Not on a timer, and optionally not without someone signing off.
5. Retire or wipe it - an audited intent the device acts on locally; a crypto-wipe destroys the disk’s keys, and is armed per device.

To see all of that on your own machine first, clone the repository and run just demo: a console, a database, sixty simulated devices and an imaging line, gone again when you press ctrl-c. To run it for real, start with Install and configure Sextant.

For the hardware end, start with setting up an imaging station.

Three ways to run Sextant

One binary, three deployments. They differ in what carries the database, the secrets and the TLS, not in what the console does.

Every command here was run on 2026-08-22 against release 0.91.0, and the traps are the ones that actually bit.

There is also a NixOS module (nixosModules.default in the flake) which runs the same binary under systemd with DynamicUser. It is the container path without the container; the same database and secret rules apply.

1. just demo

git clone https://codeberg.org/DAWO/DAWO-Sextant.git && cd DAWO-Sextant
just demo

Console on http://127.0.0.1:8080 with sixty simulated devices, a wave plan and a working imaging line. Ctrl-c stops everything and deletes the directory, including the database.

Needs initdb, pg_ctl and createdb on PATH; nix develop provides them.

It runs --dev-auth --gate none --allow-unvalidated, which is why it is a demo: a synthetic owner session on loopback and no Nix validation. The gate cannot be exercised locally at all - the example overlay takes Sextant as a path input, which stops resolving once the overlay is a git repository (issue #74).

2. A container, on NixOS or anywhere

podman run -d --name sextant --network host \
  -e SEXTANT_PG_DSN="postgres://sextant@127.0.0.1:5432/sextant?sslmode=disable" \
  -e SEXTANT_CHECKIN_TOKEN="…" \
  -e SEXTANT_SECRET_KEY="$(head -c 32 /dev/urandom | base64)" \
  -v /srv/sextant/overlay:/data/overlay:z \
  forgejo.bb-open.com/bb-open/sextant:0.91.0 \
  --addr 127.0.0.1:8080 --repo /data/overlay --write --gate remote \
  --gate-url https://gate.example.org

The image runs as uid 65532 and will not start on a volume it cannot write. The state directory defaults to <repo>/.sextant-state, so a bind mount owned by your own user fails with:

sextant: state dir: mkdir /data/overlay/.sextant-state: permission denied

Chown the volume to that uid before the first run (podman unshare chown -R 65532:65532 /srv/sextant/overlay for a rootless podman), or point --state-dir at a volume that is writable.

--dev-auth only works on loopback, which inside a container is the container’s own. --network host is why the example above works; a real deployment uses an IdP and does not need it.

Verified on 0.91.0: five capabilities mounted, /station answered 200, and a setting saved in the console landed in fleet.json inside the mounted overlay.

Put TLS in front of it. Behind TLS on a non-loopback address the console refuses to ship session cookies without --secure-cookies, deliberately.

3. The Helm chart

kubectl create namespace sextant
kubectl -n sextant create secret generic sextant \
  --from-literal=SEXTANT_CHECKIN_TOKEN='…' \
  --from-literal=SEXTANT_SECRET_KEY="$(head -c 32 /dev/urandom | base64)" \
  --from-literal=SEXTANT_SESSION_KEY="$(head -c 32 /dev/urandom | base64)" \
  --from-literal=SEXTANT_OIDC_CLIENT_SECRET='…'
helm install sextant ./deploy/helm -n sextant -f my-values.yaml

This is the only path that brings its own database: the chart creates a CloudNativePG cluster and wires the console to it, so SEXTANT_PG_DSN is not in that secret. It needs the CloudNativePG operator installed first.

The full walk-through, including the values that bite, is Install and configure Sextant.

What differs, and what does not

The console is the same binary in all three. What changes:

• The database. The demo makes one and throws it away; the container and the NixOS module expect one; the chart creates one.
• Validation. The demo has no gate. The container and the chart should use --gate remote with a nix-capable gate-runner - the console image ships no nix on purpose.
• Backups. Only the chart has an opinion, and its backup is off by default while the database holds the only copy of the LUKS recovery keys. Whichever path you take, that is your decision to make and not one to inherit.

Install and configure Sextant

Git is not an integration in Sextant - it is the storage. A fleet’s whole configuration is data in a git overlay repository (fleet.json, policies, overlays). Sextant reads and writes that repository directly: every change is an audited commit pushed to the remote, and the remote is the source of truth. So “connecting Sextant to git” means pointing the console at your overlay repo; there is nothing else to wire.

What you need

• An overlay repository - a git repo that consumes a NixOS core flake and holds your fleet.json. One repo per organisation (tenant). It is the same repo the devices follow via comin.
• Postgres - the observed plane (check-ins, tokens, image jobs, prefs). A single instance next to the console is fine; it partitions per tenant.
• An OIDC identity provider - console login (one IdP per instance), mapped to roles by group. Optional LDAP for the group-picker source.
• A validation gate - eval runs nix in-process; remote delegates to a small nix-capable gate-runner (the console image itself ships no nix). Use remote in production; it is fail-closed.

Before you start

To see it working before you deploy anything, just demo starts a console, a throwaway database, sixty simulated devices and an imaging line on your own machine, and deletes all of it on ctrl-c.

Connect the console to your overlay repo

Point the console at the repo and a push remote:

--repo /data/overlay        # the working tree the console edits
--git-remote origin         # the push remote (HA source of truth)

Under Helm (deploy/helm), the same as values:

gitRemote:
  url: https://your-forge.example.com/org/sextant-overlay.git
  branch: main
  netrcSecret: sextant-overlay-netrc   # credentials for a private remote
gateMode: remote                       # or eval / none
oidc:
  issuer: https://id.example.com
  clientId: "<client-id>"

The console clones the repo, keeps its snapshot in sync (the remote is authoritative - commits made by engineers or CI show up without a restart), and the devices follow the same repo: comin on each device tracks rings/<group> (or main), so a rollout that advances a ring branch lands on the devices in that ring.

Settings that are environment-only

Secrets never go in the fleet document or in Helm values as plain text. These are read from the environment, and the chart mounts them from one secret (secretName):

Every flag has an environment equivalent under the same prefix (SEXTANT_SECURE_COOKIES, SEXTANT_TRUST_PROXY, SEXTANT_SHUTDOWN_GRACE, and so on); sextant --help lists the flags.

First deploy, end to end

1. Create the overlay repo from your NixOS core (a fleet.json with your org and the core as a flake input). Push it to your forge.

2. Deploy the console from the chart in this repository:

   kubectl create namespace sextant
   kubectl -n sextant create secret generic sextant \
     --from-literal=SEXTANT_CHECKIN_TOKEN='...' \
     --from-literal=SEXTANT_SECRET_KEY="$(head -c 32 /dev/urandom | base64)" \
     --from-literal=SEXTANT_SESSION_KEY="$(head -c 32 /dev/urandom | base64)" \
     --from-literal=SEXTANT_OIDC_CLIENT_SECRET='...'
   helm install sextant ./deploy/helm -n sextant -f my-values.yaml
   

   The chart creates a CloudNativePG cluster and wires the console to it, so the DSN is not in that secret. Its backup is off by default, because a default pointing at an object store this chart cannot assume would fail every install - and this database holds the only copy of the LUKS recovery keys, since a device erases its own once the console acknowledges it. Turn cnpg.backup on before the fleet grows.

   Or the container, or the NixOS module, with the gitRemote, gateMode, oidc and Postgres settings above. Behind TLS, set --secure-cookies (the console refuses session cookies without it on a non-loopback address).

3. Deploy a gate-runner if gateMode: remote; it keeps a warm clone of the overlay and evaluates each candidate before the console commits.

4. Log in via your IdP. Enroll a device, assign it to a group, edit settings - every change passes the gate and commits to the overlay. Stage a rollout to land updates in waves.

Build-before-promote

At scale, ring promotion should not mean 10,000 devices each independently compiling the same closures on weak edge hardware. With gateMode: remote and a gate-runner cache configured (a signing key secret, and optionally a dedicated cache host), a rollout’s wave builds its release into that signed binary cache before its ring branch flips - devices then substitute (download) the pre-built closure instead of building it. Enable the console side with releaseCache: true. See Scaling to 10,000+ devices for the reasoning, and Ship an update for what a wave’s Building status means day to day.

Notification e-mail (SMTP)

In-app notifications work with no extra setup. To also deliver them by mail, an owner configures SMTP per organisation under E-mail (SMTP) in the console (host, port, from, security). The password is set one of two ways:

• A secret reference (recommended) - enter the name of a secret; the value lives in agenix or a cluster Secret mounted at SECRET_DIR (default /run/secrets/<name>). Sextant reads only the name, never storing the value.
• A typed password - available only when SEXTANT_SECRET_KEY is set (a base64 32-byte key). The password is then sealed (AES-256-GCM) and stored in Postgres. Without the key this option is disabled and the console says so.

Both can also be set at deploy time. SEXTANT_SECRET_KEY is an environment-only secret; add it to the same secret the chart mounts (secretName).

Multi-tenant (model B): one overlay repo per organisation, isolated stores, one console instance per repo. See docs/adr/ for the decisions behind this.

Troubleshooting

The console refuses to start / refuses session cookies. Behind TLS on a non-loopback --addr, Sextant refuses to ship session cookies without --secure-cookies (or SEXTANT_SECURE_COOKIES=true) - this is deliberate fail-closed behaviour, not a bug. Set the flag.

Every write is refused with gateMode: remote. The gate is fail-closed: no reachable gate-runner means no writes, by design. Check the gate-runner’s /healthz before flipping gateMode to remote, and after any gate-runner redeploy.

A rollout wave never leaves “Building”. With build-before-promote enabled (releaseCache: true + gateRunner.cache), check the gate-runner’s cache is healthy and its signing key secret is present - a wave cannot promote until its release lands in the signed cache.

Devices check in but the console shows no status, and /station is 503. The observed plane has no database. The console mounts what it can and keeps going rather than refusing to start, so this looks like a product missing features rather than a missing setting. Check SEXTANT_PG_DSN reached the pod, and that the CloudNativePG cluster is Ready if the chart created one.

SMTP is configured but no mail arrives. Confirm the password resolved: a secret reference must exist under Secrets with a value the runtime can actually read (agenix or the mounted SECRET_DIR); a typed password needs SEXTANT_SECRET_KEY set on the deployment, or the console disables that option outright.

Set up an imaging station (NUC)

An imaging station (the “inspoelstraat”) is a small always-on box that boots target devices over PXE and reports what it finds to the console. From there you dispatch imaging jobs at the target devices - the station itself is set up once (and occasionally re-provisioned), then left running.

This chapter is the full procedure, kale NUC to working station. Steps marked Manual step are not driven by the console today - they are candidates for future turnkey automation, but for now someone does them by hand at a keyboard or a workstation shell.

Checklist

Work through these in order. The station is done when the last box is ticked.

• Manual step - write a NixOS minimal USB installer
• Manual step - boot the NUC from the USB, bring up SSH
• Manual step - run nixos-anywhere from your workstation to install the -install variant (systemd-boot + LUKS passphrase, Secure Boot off)
• Register the station in the console and mint its report credential (owner, console step)
• Manual step - put the credential on the station and point the agent at the console
• Verify the station shows as registered and reachable in the console
• Manual step - PXE-boot a test target device on the station’s network and confirm it appears as discovered
• Wire the station to your fleet overlay so it can dispatch real imaging jobs (see Image a device from the console)
• (Later, once the fleet needs it) Manual step - re-provision the station on the -sb variant (Secure Boot on) and enrol TPM2, then move it to the steady-state dawo-inspoelstraat configuration

1. Write a NixOS minimal USB installer — manual step

lsblk                              # find the whole stick, e.g. /dev/sdb (NOT sdb1)
sudo dd if=<nixos-minimal>.iso of=/dev/sdX bs=4M status=progress oflag=sync

Double-check the device node against lsblk before running dd - the whole disk, not a partition, and never the workstation’s own disk.

2. Boot the NUC and bring up SSH — manual step

Plug in the USB and wired ethernet. Power on; the NUC’s boot menu is F10 if it does not auto-boot the stick (some models use F2 or Esc - check the splash screen). At the installer prompt:

sudo systemctl start sshd
sudo mkdir -p /root/.ssh
sudo tee /root/.ssh/authorized_keys < keys/id_ed25519.pub   # operator pubkey
ip -brief a                                                  # note the NUC's IP

3. Install with nixos-anywhere — manual step

Pick a LUKS passphrase and store it in your password manager first, then drive the install from your workstation (not the NUC itself):

cd inspoelstraat-appliance
printf %s '<LUKS-passphrase>' > /tmp/luks.key
NIX_SSHOPTS="-o IdentitiesOnly=yes -o StrictHostKeyChecking=no" \
  nix run github:nix-community/nixos-anywhere -- \
    --flake .#dawo-inspoelstraat-install \
    --target-host root@<installer-ip> \
    -i ../keys/id_ed25519 \
    --generate-hardware-config nixos-generate-config ./hardware-configuration.nix \
    --phases disko,install \
    --disk-encryption-keys /tmp/luks.key /tmp/luks.key
rm -f /tmp/luks.key

Notes:

• --generate-hardware-config re-probes the NUC and writes its real kernel modules before building - do not reuse a hardware config from a different box.
• Disko targets /dev/nvme0n1 (the NUC standard). Adjust the disko module if your hardware differs.
• printf %s (not echo) writes the passphrase with no trailing newline. This matters again in step 4 - see the troubleshooting note below.
• This installs the -install variant: Secure Boot off, no TPM2 key enrolled, systemd-boot + a LUKS passphrase at boot. That is deliberate - Secure Boot and TPM2 are enrolled later, once the box is confirmed working.

The NUC reboots into its fresh, minimal NixOS install once this completes.

4. Register the station in the console

In the console, under Organisation -> Imaging stations (owner reach):

1. Register an imaging station: give it a tag (e.g. dawo-inspoelstraat, matching [a-z0-9][a-z0-9-]*), an optional description and site.
2. Mint credential: the console generates a bearer token and shows the report endpoint (https://<console-host>/api/station/<tag>/report) and the token together. This is a one-shot reveal - the token is never shown again, so copy both before leaving the page.

5. Put the credential on the station — manual step

Configure the station’s report-agent (in the inspoelstraat-appliance flake) with the endpoint and bearer token from step 4, so it can POST its discovery reports to the console. Exactly how the agent picks up the token is a station-flake concern (a runtime credential file is the usual pattern, same shape as the per-device agent credential - see Install and configure Sextant); the important operational rule is the same one that trips up the per-device credential too:

Write the token with no trailing newline. printf %s '<token>' > path or a paste that strips the newline both work; echo '<token>' > path (or a plain heredoc) leaves a \n at the end of the file. A bearer token with a stray newline never matches what the console minted, and the station’s reports fail authentication with no more detail than “unauthorized” - see Troubleshooting below.

6. Verify the station is live

Back in Organisation -> Imaging stations, open the station. If the agent is configured correctly it appears registered and (once it has reported at least once) shows a device count from its PXE network. If nothing shows up yet, PXE-boot a target device on the station’s network next - see step 7.

7. Confirm discovery — manual step (boot a target device)

PXE-boot any spare target device on the station’s network. It should appear under the station’s discovered devices within a minute or two. This proves the whole chain: agent -> report endpoint -> console -> discovered plane.

8. Wire the station to your fleet overlay

The station is now registered and reporting, but imaging real devices from it needs the station’s runner wired to your fleet overlay (so it can build and install the right configuration for each target). See Image a device from the console for the imaging flow itself, driven from Enrollment in the console.

Later: Secure Boot and TPM2

The steps above deliberately leave Secure Boot off and TPM2 unenrolled - get a working station first. Once it is confirmed:

1. Manual step - re-provision with the -sb flake variant (Secure Boot enabled, still a LUKS passphrase at boot).
2. Manual step - enrol TPM2 on the device so the passphrase is no longer needed at boot (it remains as break-glass recovery - see Manage secrets).
3. Move the station onto the steady-state dawo-inspoelstraat configuration.

This same Install -> Secure Boot -> TPM2 -> Done progression is what the provisioning wizard walks an operator through per target device - see Image a device from the console.

Troubleshooting

The station never shows a device count / discovered devices. Check, in order: the station’s network actually serves PXE to the target device; the agent process is running on the station and can reach the console’s report endpoint (curl -i <report-url> from the station should at least get a 401, not a connection failure); and the credential file has no trailing newline (see step 5).

Minting a new credential “loses” the station. It does not - re-minting only invalidates the previous token. Update the credential file on the station with the new one (again, no trailing newline) and it resumes reporting.

nixos-anywhere fails partway through disko/install. Re-run it - nixos-anywhere --phases disko,install is safe to repeat against a booted installer. If it fails consistently on the same step, boot the installer again fresh (a half-partitioned disk can confuse a second disko pass) before retrying.

SSH from the workstation hangs or is refused. Confirm ip -brief a on the installer shows an address reachable from your workstation (not just link-local), and that sudo systemctl start sshd was run after the installer environment finished booting, not during POST.

Image a device from the console

Once a station is registered and reporting (see Set up an imaging station), imaging target devices is driven entirely from the console’s Enrollment page - no shell access to the station is needed for a routine batch.

Step 1: choose the station

Open Enrollment, pick the station whose PXE network the target devices are on, and continue. If no station is listed, none is registered yet - see Set up an imaging station.

Step 2: batch-dispatch the discovered devices

Pre-flight (one firmware visit per device, before PXE): enable the Security Chip (TPM2) and clear it once; enable Secure Boot AND reset it to setup mode (clears the factory keys). With that state set, everything after Dispatch runs hands-off - signed install, key enrolment, TPM2 sealing, verification - with no BIOS visit afterwards.

PXE-boot the target devices; they appear as discovered rows (MAC, vendor, model, disk size) under the chosen station. Imaging is batch-only by design - one audited pass images a whole rack rather than one device at a time:

1. Set the shared hardware profile (suggested from make and model where the profile catalog matches), class and group for this rack of otherwise-identical machines.
2. Tick the devices to include, and give each one a CMDB name (the tag it will enrol under).
3. Dispatch imaging. This creates each device’s record, captures its hardware specs, and queues an image job per device.

Step 3: watch the jobs run

The imaging-jobs table shows each device’s status live: pending -> imaging (with a progress bar and current step) -> installed, or failed. Under the hood, the station runner claims each job, resolves the target’s IP from its DHCP lease, runs nixos-anywhere against that device’s generated configuration, and bakes in the device’s one-time agent credential.

The install stages everything the security ceremony will need, so nothing has to be generated or typed on the device later:

• Per-device Secure Boot signing keys, generated on the station (sbctl) and shipped with the install. When the device’s resolved config enables secureboot.enable, the boot chain is signed during the install itself - no separate “audit mode” deploy round-trip.
• A disk-encryption recovery phrase - six plain words (diceware), typed exactly as revealed, instead of a 32-character random string. It is sealed into the per-device secret store; revealing it is owner-only and audited.
• A one-shot TPM2 enrol key, staged root-only inside the encrypted volume; the on-device executor uses it once to seal the disk to the TPM2 and then shreds it.

After the install the device reboots into its new system by itself.

Open the provisioning wizard (linked from the jobs table) for the guided, per-device view of the same batch. It walks a four-phase stepper - Install -> Secure Boot -> TPM2 -> Done - and adapts to what each device’s hardware profile actually needs:

• Which phases apply is decided by the device’s resolved config (secureboot.enable, diskUnlock.tpm2.enable) and its hardware (no EFI -> no Secure Boot phase; no TPM2 chip -> no TPM2 phase). A device that needs neither goes straight from installed to done on its first check-in.
• Exactly one manual step remains: when a device reaches the Secure Boot phase, the wizard shows a brand-specific firmware action (the entry key and the exact BIOS steps - e.g. Lenovo/ThinkPad enters on F1, Intel NUC on F2, HP on F10) and an in-console reboot control. Enable Secure Boot and reset to setup mode; everything after that is automatic - the device enrols its (pre-staged) keys, reboots enforcing, seals the disk to the TPM2 and reboots once more.
• Every phase turns green only on what the device itself reports (firmware state, executor acknowledgements) - the final done card is a verification: Secure Boot observed enforcing, TPM2 sealing confirmed by the executor that performed it.
• Each row carries a plain-language “Now:” line telling a non-expert exactly what is happening or what to do; the page refreshes itself.
• The wizard also tells you when it is safe to unplug a device (once its phase reads done - before that, keep it cabled so it can keep checking in and converging).
• If a device produced a one-time LUKS recovery key during provisioning, the wizard shows it once (a break-glass secret store keeps it recoverable later for an organisation owner - see Manage secrets).

Step 4: converge

Each imaged device boots, checks in, and converges its configuration. It then shows on the device page with its facts and posture (Secure Boot / TPM2 state), and on Compliance if anything about it needs attention.

Troubleshooting

A discovered device never gets an IP / the job stalls at “imaging”. The runner resolves the target’s IP from its DHCP lease; if the station’s PXE network hands out leases slowly or the device NICs are ambiguous (multiple discovered rows with a similar MAC prefix), remove the stale discovered row and re-PXE-boot the device to get a fresh lease and report.

A job fails. The jobs table and the wizard both surface the failure message inline. Most failures are hardware-profile mismatches (wrong disk device assumed by disko) or a target that lost network mid-install - cancel the job, fix the profile or cabling, and dispatch it again; a failed job does not block the rest of the batch.

The Secure Boot step never completes. Check the manual firmware action shown in the wizard was actually completed in the BIOS (Enabled + Reset to Setup Mode -> Save & Exit) before rebooting; a reboot without that toggle just returns the device to the same phase. The key enrolment itself is automatic once the firmware is in setup mode.

The device converges but its posture still shows Secure Boot/TPM2 as not enforcing. Give it one more check-in cycle - posture is self-reported by the device on check-in, so it lags the wizard’s own phase tracking by up to one interval.

Ship an update

Updates (in the sidebar under Shipping) is the one board for the whole journey from a proposed configuration change to it running on the fleet: propose -> review -> ready -> rolling out in waves. Changes and Rollout are drill-downs reached from cards on this board, not separate destinations you navigate to directly.

Step 1: review proposed changes

Every configuration edit - a setting, a policy, an integration - is staged as a change: a named, audited unit of work on its own branch. A change moves through:

• Draft - staged edits, not yet validated.
• Building - the Nix gate is evaluating the change (see Safe writes and the Nix gate).
• Ready - the gate passed; it can be reviewed and merged.
• Failed - the gate rejected it; the distilled error is shown inline (see Troubleshooting below).

Open a change directly from the Updates board (Change ID + title, then Open change), or let one open automatically: if your organisation requires change requests (Access -> Approval flows), saving settings from the Configuration editor stages the same edits as a fresh change instead of committing straight to git, and lands you back here.

For a Ready change: View diff shows exactly what will change on disk per host; Approve merges it (blocked on yourself if four-eyes review is required); Reject abandons it. A Draft can be Submitted (sends it to the gate) or abandoned.

Step 2: roll out

Once there is a current revision to ship, this section is either an active rollout in progress or the button to start one against the latest merged revision.

Starting a rollout kicks off the wave plan from Step 3 (below) against the current revision. Each wave shows a live status label as it progresses:

An owner can approve an awaiting wave to let the rollout proceed, or cancel the whole run.

Step 3: the rollout procedure

Two panels, both owner reach:

Wave plan. Each wave (a “ring”) pins one device group plus its promotion gates: a name, the group, a soak window (minutes healthy on target before the next wave may start), a minimum healthy % (defaults to 100 - every device healthy), an optional max at once (a count-capped canary that widens cohort by cohort instead of releasing the whole group at once), and whether the wave requires approval (a human checkpoint - the enterprise “test sign-off” step). Size a wave small first (a canary), then wider; tune the progression by ordering waves and sizing their groups.

Governance. Three checkboxes, organisation-wide (mirrored on Access -> Approval flows):

• Require change request - configuration edits must go through a reviewed change (this board’s Step 1), never a direct commit.
• Require four-eyes - a change may not be merged by its own author.
• Require test wave - a rollout must have a gated (manual-approval) wave before it starts; an owner can explicitly skip this per rollout, and the skip is logged.

See How a rollout ships and Approval flows for the concepts behind both panels.

Troubleshooting

A change is stuck in Failed. The card shows the distilled error - the actionable line pulled out of the gate’s evaluation trace (e.g. “device X: unknown hardware profile ‘Y’”). Fix the underlying edit and resubmit; if the message is not enough to act on, the merge/submit response (and the gate-runner’s own logs) carry the full multi-line trace behind a “technical detail” fold.

A wave sits on “Building” for a long time. Build-before-promote realises the wave’s whole release into the signed binary cache before its branch flips - this is centralised, one-time compute per distinct configuration shape, not per device, but it is real wall-clock on the build workers. Check the gate-runner/build-worker health before assuming something is stuck; a missing build worker delays a rollout, it does not corrupt one.

A wave never leaves “Deploying” / “Soaking”. Devices pull on their own schedule (comin), so convergence is not instantaneous - check the Devices page or Compliance for the specific machines that have not landed on the target revision yet; “behind” incidents there point at exactly which ones and since when.

“Require test wave” blocks starting a rollout. Either add a wave with require approval checked to the plan, or (if you are an owner and this run genuinely does not need one) use the explicit skip offered on the start form - it is logged either way.

Configure a hardware model

Some settings follow the machine rather than the organisation. A Lenovo needs the driver for its fingerprint reader; a Dell standing next to it in the same group does not. Group membership cannot say that - both are in infra - and setting it on each device by hand stops working the day the fleet grows.

The Hardware page is where that is said once.

What the page shows

One row per model the fleet runs. That is the union of two lists, because either one alone would mislead:

• the models the overlay’s hardware-profiles.json describes, whether or not any device carries one yet;
• the models devices actually report, whether or not the overlay has heard of them.

A model in the second list and not the first is flagged not in the catalog. That is worth seeing: nothing can image one, because imaging needs the model’s disk layout and steps. The fix is in the overlay, not in the console.

Configuring one

Configure this model opens a window with the model’s settings, the keys it locks, and who it applies to - the whole fleet, or only the devices of that model inside one group.

Saving writes three things in one commit: a policy holding the settings, a filter that selects exactly that model, and an assignment binding them. That is what an operator could always have assembled by hand, and the reason this page exists is that assembling it by hand took three edits in the right order, so nobody did.

It is written as a policy on purpose. A policy carries a name and a description, so fprint.enable = true stops being a value on a machine and becomes “Fingerprint reader (Lenovo T495s)” in the audit trail, reusable and explainable six months later.

Configuring again is editing: the settings are refreshed and the assignment moves if you changed who it applies to. It never leaves a second one behind.

Removing it

Remove configuration deletes the policy and the assignment. The filter survives if another assignment still points at it - it is a named thing somebody may have reused.

Why hardware is not a scope

Settings resolve organisation → group → device. Hardware is deliberately not a fourth level in that ladder: it is a different axis, not a finer one, and a second mechanism for per-model settings would mean every operator has to know which one a fleet used before they can predict what a device does. See decision record 27.

The practical consequence: a hardware policy contributes at the scope it is assigned to. If the organisation enforces a key, the model’s setting does not override it - enforcement runs the other way, from general to specific, and that is the governance direction on purpose.

Track compliance

Compliance answers one question: which devices are not to spec right now, and why. It is the full drill-down behind the compliance donut on Overview - the donut caps its attention queue at 8 items; this page lists every open incident.

Reading the page

The three summary chips (All, Critical, Warning) double as a filter - click one to narrow the device table below it. Every active (non-retired) device appears, worst status first:

• Critical - an error the device reported, or a wipe that failed or was refused.
• Warning - offline, never checked in, or behind its target revision.
• To spec - no open incident.

Each row lists the device’s issues with a short title, a detail (the specifics - e.g. which revision it is running versus its target, or when it was last seen), and a suggested action.

What raises an incident

A single device can carry several incidents at once (e.g. offline and behind). Suggested actions point at the fix: verify imaging and connectivity for never-seen, check power/network for offline, check the rollout and device logs for behind, open the device to inspect the failure for errored, and re-arm or clear the interlock for a refused wipe (see Update, retire and wipe).

Policy exposure

Below the device table, a per-policy table shows where each policy is assigned (its scope targets) and, of the devices under those targets, how many currently carry an open incident - a revision-level proxy for “this policy may not actually be applied everywhere it is assigned yet”. A policy with no assignments shows as unassigned; it has no effect until targeted.

Troubleshooting

A device shows as behind right after a rollout started. Expected - “behind” just means the device has not yet pulled and converged to its target revision. Give it a check-in cycle or two before treating it as stuck; if it stays behind well past the wave’s soak window, check Ship an update for whether the wave itself is stalled.

A device never clears “never seen”. It has an enrolled record but has not reported at all. Confirm it was actually imaged (see Image a device from the console) and that its agent credential and network reach the console - the same credential/newline pitfall documented in Set up an imaging station applies to a device’s own agent credential too.

A wipe keeps showing “refused”. The root executor requires the device to be locked first (or the intent to be forced) before it will act on a wipe. Re-arm the wipe intent, or lock the device first, then retry - see Update, retire and wipe.

Update, retire and wipe

Update

Updates ship as a rollout: a new revision promotes through ordered waves, each gated on health and a soak window, with an optional manual test gate. Day to day this is driven from the Updates board - see Ship an update for the full walkthrough and How a rollout ships for the concept behind it.

Provisional: a device that exists but has not spoken

Enrolling a device creates its record immediately, in the provisional state. The first successful check-in promotes it to active.

That state exists because an install can fail, and a record left behind by a failed attempt used to break more than itself: a ring made up entirely of devices that never arrived could not converge by definition, so the whole rollout waited for machines that were never coming.

So a provisional device is counted differently. It is a real record - you can see it, name it, and re-image the same chassis onto it rather than minting a second one - but it does not hold a ring back, because it has never claimed to be running anything.

Abandoned enrolments are listed rather than deleted. Somebody starts an installation that never reports: unfamiliar hardware, a slow link, a station operator called away, a laptop enrolled on Friday that does not boot until Monday. The console surfaces those as a list for an operator to act on, because the two mistakes are not symmetric - reaping too early deletes a record somebody is still using, reaping too late leaves a stale row in a list.

Retire

Retiring a device keeps its record for audit but stops image builds, check-ins and rollout counting. Reactivation is an explicit, audited step.

Lock and wipe

Lock and wipe are intent-as-data: the console records the intent as an audited change (on a device’s own page, under the red-bordered remote-actions panel); the device pulls it on check-in and acts locally. There is no live command channel.

• Lock locks all sessions and persists across reboot; clear the intent to release.
• Wipe cryptographically erases the device by destroying its LUKS key slots. It is irreversible and gated: the root executor refuses a wipe unless the device is locked first, unless the intent is explicitly sent with force (which the console’s wipe action does, backed instead by a type-the-device-tag confirmation as the human safety net). Arm a device for wipe only when it is cleared to be wiped.

Troubleshooting

A wipe shows “refused” on the device page or in Compliance. The device declined the intent - typically because a local interlock blocked it even with force set. Clear the intent, confirm the device is locked, then re-send the wipe.

A wipe shows “failed”. The device attempted the crypto-wipe but never confirmed completion back to the console. Treat the disk as not yet confirmed destroyed - verify by other means before reusing or disposing of the hardware.

Retiring a device does not remove it from the fleet count. That is by design - retiring keeps the audit record and simply stops new image builds, check-ins and rollout counting. Use Remove instead if the device should be unenrolled entirely; unlike a retire, removal cannot be undone by reactivation.

Endpoint controls

Four settings groups decide what a person can do on their own machine without calling anyone: USB device control, printing, user rights, and the local administrator account. They are ordinary fleet settings, so they resolve organisation → group → device and can be locked like any other.

USB device control

usbDevices.enable turns on USBGuard: a device plugged into a running machine is blocked unless a rule allows it.

usbDevices.allowlist takes extra USBGuard rules, one per line, for the things that must keep working - a specific model of card reader, a signature pad.

The enable carries a high risk class, so the console asks for an explicit extra confirmation before it changes. That is not ceremony: switching it on across a fleet stops hardware people are holding in their hands, and switching it off removes a control somebody signed for.

Printing

printing.enable turns on CUPS. printing.discover finds printers announced on the local network over mDNS/IPP, which is what most offices want and what most home setups need.

printing.drivers chooses the driver set:

• open - the standard IPP/PostScript path. Covers most office printers made this decade, and keeps the closure small.
• broad - the wider vendor driver set, for hardware the open path does not reach.

User rights

userRights.enable lets ordinary users change desktop settings that would otherwise need an administrator. Each right is set individually under userRights.options.*, and each takes one of four modes:

Every mode requires a local, active session - never SSH, never a background or remote one. That clause is what makes granting these safe, so it is part of every rule rather than something a mode can opt out of.

The rights on offer are the ones that otherwise generate a support call from somebody who cannot work: approving a dock when it is plugged in, applying a firmware update fwupd offers, editing a network connection that applies to the whole machine.

What is deliberately not on the list

Four things are absent, and it is not an oversight:

• Creating, deleting or re-grouping accounts. On a fleet machine the account set comes from the directory. A local user who can add accounts can add one that outlives their own and answers to nobody.
• Autologin and the greeter’s user list. Autologin turns full-disk encryption into a locked door with the key taped to it.
• Changing the hostname. The fleet identifies devices by name.
• Firmware downgrades. Rolling firmware back to an older version is a way to reintroduce a fixed vulnerability, so the upgrade path is offered and the downgrade path is not.

They are absent rather than explicitly denied. An explicit no would short-circuit polkit for administrators too, which would take away the path that is supposed to remain.

Local administrator

localAdmin.enable creates an administrator who can sign in when the directory or the network is unreachable. localAdmin.username is the login name - pick one per organisation rather than inheriting a default - and localAdmin.passwordSecret names a secret reference holding the password hash, so no credential is shared between fleets and none of it passes through the console.

This is also high risk class, in both directions. Off locks the account on every device it applies to; on, it creates a way in that does not depend on your directory being up.

Related: Manage secrets for registering the password reference, and Approve a request for privilege for the path a user takes when they hit something none of these rights cover.

Approving a request for privilege

Somebody on a fleet laptop tries to do one privileged thing - install a printer driver, change a network setting - and the system asks for an administrator. This is where that request lands.

The problem it solves

polkit’s answer to “you may not do this” is a dialog asking for an administrator’s password. On a managed fleet machine that means the local admin account. Away from the office, that is not a slower path - it is no path at all.

The predictable outcome is that somebody shares the admin password to get the user unstuck, and then it is shared again, until it has stopped being a secret.

So the request becomes something an administrator answers centrally, and it is logged.

What an approval is, and what it is not

Sextant decides. It never asserts.

The grant does not come from the console and cannot: polkit will not let an agent vouch for an identity. The answer travels through polkit’s own setuid helper, which runs PAM, and PAM turns the answer into an authentication - or does not. The console’s role is to say yes or no; the device’s own machinery enforces it.

This matters for how you read the request.

Approve on who and where, not on what

Each request carries four things:

The action is context, not proof. PAM is never told the polkit action id, so the device’s own session supplies that string - and a session is not a trustworthy narrator about itself. Approve on the strength of who is asking and where, both of which are established by the device’s authenticated check-in, and read the action as what the user says they are doing.

Five minutes, and why it is short

A request waits five minutes for an answer.

That is deliberately short. Somebody is standing in front of a frozen dialog for the whole window, so a generous timeout is not generosity. Five minutes is long enough for an administrator who is watching, and short enough that a user who gets no answer finds out while they still remember what they were trying to do.

Four outcomes, and one of them is not a decision

• Approved - the console said yes; PAM completes the authentication.
• Denied - the console said no.
• Pending - waiting, within the window.
• Expired - nobody answered in time.

Expired is a distinct outcome rather than a flavour of denied, on purpose. “We said no” and “nobody was there” are different conversations, and a list of expired requests is a staffing problem, not a policy one. If they are piling up, the answer is not to tighten the rules.

Expiry is derived from the clock rather than written down, so a request cannot sit Pending for ever because whatever was supposed to expire it died.

Where to find them

The Requests page in the console. Every decision records who made it and when, and lands in the audit log like any other action.

Manage secrets

Sextant handles two different kinds of secret, and the Secrets page (plus a device’s own page) is where both are managed. They are not interchangeable: one is a reference an operator points settings at; the other is material the platform generated on a device’s behalf and holds so it can be recovered later.

Secret references (for settings and integrations)

A setting field that needs a secret value - a NetBird setup key, an LDAP bind password, a Wazuh enrollment secret, an SMTP password - never accepts the value itself in the console. Instead you register a name, and settings pick that name from a list:

1. Open Secrets.
2. Register a secret: give it a name ([a-z0-9][a-z0-9-]*, e.g. netbird-setupkey) and an optional description.
3. Point any secret-typed setting field at it, in Configuration editor or Integrations - the field renders as a picker of registered names, with a shortcut to register a new one inline if you started from the setting itself.

The console never sees or stores the plaintext: only the name travels through the config repo and Sextant’s own state. The device resolves the name to the decrypted material at runtime via agenix. Removing a registered name breaks the build for anything still pointing at it - the console warns before you confirm.

Per-device secrets (break-glass recovery)

Some material is generated for a specific device during provisioning and has nowhere else to live: a LUKS disk-encryption recovery passphrase, or a break-glass local-administrator password. TPM2 enrolment makes the LUKS passphrase unnecessary at day-to-day boot, but it remains the recovery path if TPM2 unsealing ever fails - so it has to survive somewhere, encrypted at rest, reachable only to someone who genuinely needs it.

Sextant seals this material (AES-256-GCM by default; a drop-in external key manager such as OpenBao/Vault is the production posture) the moment it is produced - at provisioning, from the imaging wizard - and never stores it in the clear.

Revealing it:

• Reveal is organisation-owner reach only - not editor, not viewer.
• From a device’s page (or the provisioning wizard, while a job is still fresh), Reveal shows the plaintext exactly once, rendered directly on the response - never redirected, so it never lands in a URL, browser history, or an access-log line.
• Every reveal is recorded: who, and when. There is no silent read.
• Once revealed, treat it as no longer fresh - the console flags a previously-revealed secret as such, since anyone who saw it once could have copied it.

If the secret store is not configured (no encryption key set), Sextant does not store per-device secrets at all rather than write them in the clear - the one-time value is then shown only at the moment it is generated (during imaging) and never again.

Who can decrypt them, and where that identity lives

Write this down for your own organisation. That it was never written down is how BB Open ended up, on 2026-07-31, with four fleet secrets and no living person able to open them - see the note at the end of this section.

An overlay’s secrets/*.age files are encrypted to a RECIPIENT SET: a list of public keys, any one of which can decrypt. In a Sextant fleet that set has two kinds of member.

Device host keys. Every imaged device gets an SSH host keypair, and its public half is added as a recipient so the device can decrypt the secrets it needs at activation. The imaging station reports the public key, the console records it, and scripts/rekey-secrets.sh re-encrypts for all of them. This part is automatic and it is not the part that goes wrong.

An admin identity. A key a PERSON holds, so somebody can rekey after a device is replaced, add a secret, or read one in an emergency. rekey-secrets.sh takes it with -i and always includes its public key, precisely so an operator cannot lock themselves out.

The trap is that the second kind is easy not to have. Nothing fails without it: devices decrypt fine, activation succeeds, the fleet converges. It only surfaces the day you need to rekey and discover that the only identities that can open your secrets are the machines themselves - and if you re-image those machines, imaging mints a fresh host key and destroys the old one.

So:

• Decide which key is the admin identity, and say so somewhere durable. A personal SSH key (~/.ssh/<you>.pub) is a fine choice and means an everyday key works.
• Keep it out of the fleet it protects. A copy in a password manager and a break-glass copy somewhere the cluster’s failure cannot reach.
• Name at least two people. One identity is a single point of failure wearing a different hat.
• Check it periodically by using it. An identity nobody has exercised is a belief, not a capability.

What happened here, and why the guidance above is phrased that way

BB Open had no admin identity. An older secrets.nix recorded that device host private keys live in a password manager and are baked onto devices at install, so the de-facto admin identity was A DEVICE HOST KEY. A stopgap from an earlier test run quietly became policy: nobody decided it, it just stayed, and it was recorded nowhere.

It surfaced on the eve of a re-image. The four secrets were recovered in time through a device that still held the key, but only because somebody asked the question first. Imaging that device an hour later would have made them unrecoverable.

Two details worth carrying: re-encrypting must be BYTE-EXACT (three of those four carry no trailing newline; use printf '%s', never echo), and a credential that has passed through a terminal during recovery should be rotated at its source on a normal schedule afterwards.

Troubleshooting

A setting’s secret picker is empty. No secret has been registered yet under that name pattern - register one on the Secrets page first, or use the inline shortcut next to the field.

Removing a secret reference breaks a build. Expected - any setting still pointing at that name fails the Nix gate on the next change. Re-point the setting at a different registered name (or clear it) before removing the reference, not after.

“Reveal” is not available on a device. Either the secret store is not configured for this deployment (no SEXTANT_SECRET_KEY / no external sealer wired up), no such secret was ever generated for this device, or you are not signed in with organisation-owner reach - reveal is deliberately not available to editors or viewers.

The revealed LUKS passphrase does not unlock the device. Confirm you copied it in full (it is shown once, select-all) and that you are unlocking the current value - if the device was re-imaged since the secret was last generated, an old reveal (or a note copied from an earlier session) no longer matches.

Notifications

The bell icon in the top app bar opens Notifications: an in-app inbox for things that need your attention - a change ready for review, a wave awaiting approval, an incident, and similar events. An unread count badges the bell.

• Click a notification to jump straight to the thing it is about (the change, the device, the rollout) and mark it read at the same time.
• Mark all read clears the unread badge without visiting each item.
• An empty inbox and “notifications unavailable” (the backing store is not configured for this deployment) are both shown plainly rather than as an error.

In-app notifications work with no extra setup. To also receive them by e-mail, an organisation owner configures SMTP once under Organisation -> E-mail (SMTP) - see Install and configure Sextant for the setup details (host/port/from, and the two ways to hold the password). Mail delivery is additive: turning it on does not change what shows up in the in-app inbox, only whether the same events also arrive by e-mail.

Push notifications are not implemented yet.

Troubleshooting

Notifications never arrive at all (in-app). The page shows “unavailable” rather than an empty inbox when the notifications backing store is not configured for this deployment - that is a deploy-time gap, not a per-user setting.

In-app notifications work but e-mail does not. Check the SMTP configuration under Organisation -> E-mail (SMTP): host, port and the password source (a registered secret reference, or a typed password sealed with SEXTANT_SECRET_KEY). If SEXTANT_SECRET_KEY is not set on this deployment, the typed-password option is disabled and the console says so - use a secret reference instead.

A notification links to something that no longer exists. The event still happened (the notification records history), but its target (a since-abandoned change, a completed rollout) may no longer show the same detail page - this is expected once enough time has passed.

Integrations

Integrations are device-side capabilities the overlay publishes so you can turn them on and configure them per scope from the console, without editing Nix. The console shows a card per integration; a card is available once the overlay publishes its options in the catalog, and not published otherwise.

Three ship with the BB Open overlay:

• NetBird - join a self-hosted WireGuard mesh, so a roaming device stays reachable and can pull and push from anywhere. You set the management URL and a setup key.
• Directory login (LDAP) - device login against your directory over SSSD (LDAP, AD or IPA). You set the provider, domain, server and a bind secret.
• Wazuh - an endpoint security agent that reports to a Wazuh manager. You set the manager address, an agent group and an enrollment secret.

Secrets are references, never values

A field that holds a secret - a setup key, a bind password, an enrollment password - is stored as a reference: the name of a secret you registered, not the secret itself. The console renders it as a picker of registered names, so a raw secret can never be typed into the console or committed to git. The device resolves the name to the decrypted material at runtime (agenix); only the name travels through the config repo.

Register the secrets first on the Secrets page (see Manage secrets), then point the integration field at one by name.

Setting it up

1. Open Integrations. A card that reads available is ready to configure.
2. Enable it and fill the fields. Secret fields offer the registered references, with a shortcut to register a new one inline.
3. Save. This writes at the organisation scope - the Integrations page is an org-wide quick-config surface for the catalog keys the overlay publishes. To narrow an integration to one group or device instead (or to review it alongside every other setting), open the same keys from Settings (the Configuration editor) and pick a group or device with its scope selector - integration settings flow down the scope chain like any other setting.
4. Every save passes the Nix gate and commits to git like every edit (or stages as a change, if your organisation requires change requests - see Ship an update).

If a card reads not published, the overlay has not exported that integration’s options yet: add its module to the overlay and regenerate the catalog (nix eval .#catalog --json > catalog.json).

Want one that does not ship with the overlay? An integration is a NixOS module that publishes options - no console change needed. See Build your own integration.

Troubleshooting

A card reads “not published” even though I added the overlay module. Regenerate the catalog (nix eval .#catalog --json > catalog.json) and restart or reload the console’s config snapshot - the catalog is generated, not live-read from the overlay’s Nix source.

Saving an integration field fails the gate. Same as any setting - the distilled error names the actionable line (e.g. an out-of-range value or an unknown option). See Safe writes and the Nix gate.

A secret field’s picker is empty. No secret reference has been registered yet - use the inline shortcut next to the field, or register one first on the Secrets page.

Build your own integration

An integration in Sextant is not a plugin. There is no SDK to learn, no interface to implement and no code to add to the console. An integration is a NixOS module in your overlay that publishes options, and the console picks them up because they are in the catalog.

That is the whole mechanism. If you can write a NixOS module, you can add an integration, and the console will render it, validate it, scope it, gate it and audit it for you.

This page walks through one from nothing to working.

What you are actually building

Two things:

1. Options - options.dawo.<yourthing> with types, defaults and descriptions. These become the fields an operator fills in.
2. Config - what those options do on the device: a systemd unit, a package, a file, whatever the thing needs.

Everything else is done for you:

Step 1: write the module

Put it in your overlay, next to the modules you already have. A minimal integration - reporting to a metrics collector - looks like this:

# modules/telemetry.nix
{ config, lib, pkgs, ... }:
let
  cfg = config.dawo.telemetry;
in
{
  options.dawo.telemetry = {
    enable = lib.mkEnableOption "report metrics to a collector";

    endpoint = lib.mkOption {
      type = lib.types.str;
      default = "";
      example = "https://metrics.example.org/ingest";
      description = "Collector the agent posts to.";
    };

    intervalSeconds = lib.mkOption {
      type = lib.types.ints.between 30 3600;
      default = 300;
      description = "How often to report, in seconds.";
    };

    # The annotation is what makes this a secret-ref picker in the console
    # rather than a free-text box. See "Secrets" below.
    token = lib.mkOption {
      type = lib.types.str;
      default = "";
      description = "Secret-ref name of the collector's bearer token.";
    } // { secret = true; };
  };

  config = lib.mkIf cfg.enable {
    assertions = [{
      assertion = cfg.endpoint != "";
      message = "dawo.telemetry: endpoint must be set when enabled.";
    }];

    systemd.services.dawo-telemetry = {
      serviceConfig.Type = "oneshot";
      script = ''
        ${pkgs.curl}/bin/curl -sS -X POST "${cfg.endpoint}" \
          -H "Authorization: Bearer $(cat /run/agenix/${cfg.token})" \
          --data-binary @/proc/loadavg
      '';
    };

    systemd.timers.dawo-telemetry = {
      wantedBy = [ "timers.target" ];
      timerConfig = {
        OnBootSec = "2min";
        OnUnitActiveSec = "${toString cfg.intervalSeconds}s";
      };
    };
  };
}

Three things in there are worth copying every time.

Types do real work. lib.types.ints.between 30 3600 is not decoration: the console renders it as a bounded field, the catalog type-check rejects 5 before Nix ever runs, and the gate rejects it again if somebody edits fleet.json by hand. A type is the cheapest validation you will ever write.

Descriptions are the UI. The description is what an operator reads in the console. Write it for them, not for you: “Collector the agent posts to” beats “the endpoint”.

Assert what enabling implies. enable = true with an empty endpoint is a configuration that cannot work. An assertion turns that from a device that silently does nothing into a change the gate refuses, with the reason attached.

Step 2: add it to your class’s module list

The module has to be in the image the class builds. In the BB Open overlay that is coreModulesForClass; in yours it is wherever you assemble the NixOS module list per device class.

This is also where class-scoping happens. A module you add only to the laptop class publishes options tagged as laptop-only, and the console will tell an operator that setting them on a station reaches nothing - rather than letting them configure something that will never apply.

Step 3: regenerate the catalog

catalog.json is derived output. The console reads it, not your Nix source, so until you regenerate it your integration does not exist as far as the console is concerned.

nix eval .#catalog --json > catalog.json

Commit it with the module. In this repository CI enforces that they match (examples/overlay/regen-catalog.sh --check), and it is worth having the same guard in your overlay: a stale catalog is a console showing an option set that the fleet no longer has.

Step 4: use it

Open Settings in the console. Your options are there, under their key names, with the types and descriptions you wrote. Set them at the org, a group or one device. Save; the gate builds it; it becomes a commit; the ring rolls it out.

That is the integration finished. No console change, no restart, no deployment.

Secrets

Never put a credential in an option value. It would land in fleet.json, in git, in every clone, forever.

Annotate the option // { secret = true; } instead. The console then renders a picker of registered secret names, and what gets stored is the name. On the device, agenix decrypts the material to /run/agenix/<name> and your module reads it from there - which is why the example above does cat /run/agenix/${cfg.token} rather than interpolating a value.

Register the name first on the Secrets page, then point the field at it. See Manage secrets.

Marking a dangerous option

Some options change the security posture of a device. Annotate those:

    disableFirewall = lib.mkOption {
      type = lib.types.bool;
      default = false;
      description = "Stop filtering inbound traffic.";
    } // { riskClass = "high"; };

The console shows a warning badge and asks for an explicit extra confirmation before saving. Use it for what genuinely deserves it; a badge on everything is a badge on nothing.

Getting a card on the Integrations page

Options alone give you a fully working integration in Settings. The Integrations page is a curated shortcut on top of that - a card per integration, configured at org scope in one place - and its list lives in the console’s source (internal/http/web/integrations.go, knownIntegrations).

So:

• Adding options to your overlay needs no console change and is the normal case. Your integration works, in Settings, scoped like everything else.
• A card on the Integrations page needs a four-line entry in that list, which means a pull request to Sextant itself. Worth doing for an integration that many fleets will run; not worth it for one that is specific to yours.

If you build something other organisations would use, send the card entry with it. That is one of the more welcome contributions there is.

What does not belong here

Some integrations are not device-side at all: the console’s own SSO, its directory lookups, its outbound mail. Those are adapters behind ports (internal/adapters/), configured at deploy time rather than per fleet, because they belong to the console’s operator and not to the fleet’s configuration.

The test is simple: if every device would need to know about it, it is an overlay module. If only the console needs to know, it is an adapter.

Checklist

• Options under dawo.<name>, with types that describe the real range.
• A description on every option, written for the operator.
• An assertion for each way that enabling it could be incoherent.
• Secrets as // { secret = true; } refs, read from /run/agenix/<name>.
• The module in the right class’s module list.
• catalog.json regenerated and committed.
• Enabled on one device first, and actually checked on the hardware.

Policies, and how they differ from settings

A setting is a value. A policy is a value with a reason attached, and that difference is the whole point: an auditor asking “why is USB storage blocked on these laptops” wants a name and a justification, not a key with false next to it.

Policies are a layer over settings, not a wrapper around them. They can do two things a setting cannot: carry the compliance controls they implement, and state requirements about a device’s observed state rather than its configuration.

What a policy carries

Enforced versus checked - the distinction that matters

This is the one thing worth reading twice.

A setting can be enforced. The fleet converges on it, drift is corrected, and a lock stops a lower scope weakening it. If the value is wrong today, the system makes it right.

A condition can only be checked. There is nothing to write: a disk does not get emptier because a policy says it should be. A failing condition is a finding to report, never a state to converge on.

Showing “enforced” beside a free-disk-space requirement would promise something the system cannot deliver, so the console does not. Conditions appear as findings; settings appear as configuration.

There is a second rule that follows from the same honesty, and it is deliberate: a device that never reported a metric is not accused of failing it. An older agent, or a probe that did not run, produces silence - and silence is unknown, not failure. A board that reports “disk below 15%” for machines it cannot measure teaches operators to ignore the finding, which costs more than the finding was worth.

Assigning a policy

A policy on its own does nothing. An assignment binds it to a scope:

• Target - org, group:<name> or device:<tag>.
• Filter - optional; narrows the assignment to devices matching a rule set (all rules, or any). Without one it covers every device in the target.
• Priority - decides which policy wins when two set the same key.

So the same policy can be assigned broadly and narrowed by filter, rather than copied per group with small edits. Editing the policy updates everywhere it applies.

Where the values end up

Policy settings resolve into a device’s effective configuration alongside ordinary scope settings, along the usual organisation → group → device chain. The device page shows each value with where it came from, so “set by policy Baseline hardening” reads the same way as “set by group bb-laptops”.

An enforced key from a policy behaves exactly like an enforced key on a scope: a lower scope may not weaken it, and the console says so rather than silently discarding the edit.

Profiles and drift

A policy created from an overlay-published profile records where it came from, as name@hash. The console then keeps the two in view and distinguishes three different ways they can disagree:

• Reapply - the overlay’s profile moved on and this policy did not.
• Edited - the policy has been changed by hand since it was applied. The stamp alone cannot see this; it is found by comparing the actual settings.
• Conflict - a hand-made policy has taken the id a profile wants.

A profile that was never instantiated simply offers to be applied, and one that matches reads as current.

All of it is provenance. Resolution ignores the profile entirely; nothing changes on a device because a profile drifted. It exists so an operator can see that the recommendation they started from has moved, and decide - which is a different thing from being moved for them.

When to use a policy instead of a plain setting

Use a plain setting when the value is simply what you want. Use a policy when somebody will eventually ask why, or when you need to say the same thing in several places without repeating yourself:

• A rule you must justify to an auditor, with the control reference attached.
• A rule that applies to a set of devices defined by a property rather than by group membership - that is what filters are for.
• A requirement about observed state, which a setting cannot express at all.

For “turn this app on for this group”, a setting is the right tool and a policy is ceremony.

How a rollout ships

An update does not reach the whole fleet at once. It promotes through ordered waves (also called rings). Each wave is a group of devices; the next wave only starts once the current one is converged healthy through its soak window. A wave can require a manual approval gate - a human checkpoint that the update was tested.

Promotion is on measured evidence, not on a timer. That is the difference from the deferral-days model most fleet tools use, where wave two starts a fixed number of days after wave one whatever wave one did.

What a wave has to prove

Three things, and it needs all of them:

A wave that cannot reach its floor because devices reached the target and turned out demonstrably unwell does not wait: that is a bad release, and the run halts.

At scale, a wave’s release is also built before it is promoted: the delivery pipeline realises the wave’s closures on build workers and pushes them to a signed binary cache before its branch flips, so devices substitute (download) a pre-built release instead of each compiling it independently. While this is happening the wave shows as Building on the Updates board; once the release lands in the cache the branch flips and the wave moves to Deploying. See Scaling to 10,000+ devices for the reasoning and the numbers behind it.

The Updates board (Step 3, “the rollout procedure”) shows the plan as a ladder: each wave with its device count (size it small first - a canary - then wider), soak, health floor, evidence floor and gate. Size a wave by its group and order; refine each wave with the gates - including an optional max at once cap, so a wave widens cohort by cohort rather than releasing its whole group in one shot.

An organisation can require a gated test wave before any rollout starts; an owner may skip it for a specific rollout, and that is logged.

How “behind” is judged

A device is behind when the revision it reports differs from its group’s target pin. For that comparison to work, the deployed revision the agent reports must be the git revision the config was built from - the same kind of value the pin holds - not a store label.

This requires one line in the overlay flake: set system.configurationRevision = self.rev (or self.shortRev) on each host. The Sextant agent module then publishes it to /etc/sextant/configuration-revision, and the agent reports it on every check-in. Without that line the field is empty and the agent falls back to the store label, which can never equal a git-hash pin - so every pinned device reads as falsely behind. A flake with uncommitted changes has no self.rev; commit before building, or the revision is unavailable.

Troubleshooting

A wave is stuck on Building. Build-before-promote is centralised, one-time compute per distinct configuration shape - not per device - but it still takes real time on the build workers. Check the gate-runner/build-worker’s health before assuming the rollout is stuck; a missing or overloaded build worker delays a promotion, it does not corrupt it.

A wave never reads Complete even though every device shows online and on the target revision. It may still be inside its soak window, or waiting on a manual approval - check the wave’s status label on the Updates board rather than only the device list.

Approval flows

How changes and updates are reviewed before they take effect is configurable per organisation, under Access -> Approval flows (owner). The same three toggles also live on the Updates board’s governance panel (Step 3) - they are the same setting, shown in both places:

• Four-eyes - a change may not be merged by its own author.
• Require change-request - configuration edits must go through a reviewed change, never a direct commit. When this is on, saving from the Configuration editor does not fail - it transparently stages the same edits as a fresh change and sends you to the Updates board to see it through review.
• Require test wave - a rollout must have a gated test wave first; an owner can skip it per rollout, and the skip is logged.

Every configuration change is a git commit that passes the Nix gate first, so an edit that would not build never reaches the fleet.

Troubleshooting

Saving settings redirects to the Updates board instead of just saving. Expected when Require change-request is on - the edits were not lost, they were staged as a new change. Continue the review from there (diff, submit, approve).

A change cannot be merged even though it is Ready. If Four-eyes is on, the merge button is unavailable to the change’s own author - have a different reviewer approve it.

Safe writes and the Nix gate

Every configuration change - a setting, a policy, an overlay, an integration - is a write to the git overlay. No write reaches git unless it first evaluates: the change is applied to a candidate fleet.json, and the Nix generator is run over the affected hosts. If the evaluation fails - an unknown option, a wrong type, a value out of range - the write is rejected and nothing is committed. If it succeeds, the change commits with the author’s identity from their session.

This is the core safety property: the console cannot commit a fleet that does not build.

Fail-closed

In production the gate runs remote: the console ships no Nix, and delegates the evaluation to a small Nix-capable gate-runner. Remote means fail-closed - a write that does not evaluate, or that the runner cannot be reached to evaluate, is refused. A broken or unreachable gate blocks writes rather than waving them through.

Three modes exist: eval (evaluate in-process, for a Nix-capable console), remote (delegate to the gate-runner, the production posture) and none (no gate, for tests or a console with no flake).

Where it sits

• A direct setting edit is gated before it commits.
• A change-request is gated when it is opened and again before it merges, so a reviewer never approves something that will not build.
• A rollout advances branch refs the gate already validated; it ships evaluated revisions, it does not re-open the edit path.

The gate is a type-and-build check, not a policy check. Governance - who may edit, four-eyes review, a required test wave - sits on top of it, in the change-request and rollout flows (see Approval flows and Ship an update).

Troubleshooting

A write is rejected with a short error. That message is a distilled line pulled out of the gate’s evaluation trace - usually the actual cause (an unknown option, a wrong type, a value out of range), not the whole trace. A change-request’s failure card, and any rejected save, carries the full multi-line detail behind a “technical detail” fold if the short line is not enough to act on.

Every write is refused, even ones that should evaluate fine. In remote mode the gate is fail-closed: if the gate-runner cannot be reached at all, writes are refused rather than committed unvalidated. Check the gate-runner is up and reachable before assuming the change itself is at fault.

Architecture overview

Sextant is a hexagonal Go application: a pure domain (model, resolver, policy compiler, filter evaluator) under application services, behind ports, with adapters for git, Nix, Postgres, LDAP and OIDC, and two thin transports - a server-rendered console and a JSON API over the same services.

Three planes carry the work:

• Config plane - the git overlay (fleet.json + catalog) is the source of truth. Writes are serialized, pass the Nix eval gate, and commit with SSO-attributed authorship.
• Observed plane - device check-ins, posture and hardware facts live in Postgres, tenant-namespaced.
• Imaging plane - discovery and image jobs provision new hardware from an imaging station (the inspoelstraat).

The whole picture

flowchart TB
    subgraph people[People and identity]
        op[Operator]
        idp[Zitadel OIDC<br/>+ LDAP groups]
    end

    subgraph control[Control plane - one cell per organisation]
        console[Console + API<br/>hexagonal Go]
        pg[(Postgres<br/>observed plane)]
        overlay[(Overlay git repo<br/>fleet.json + catalog)]
    end

    subgraph workers[Workers - scale out independently]
        gate[Gate-runner<br/>eval workers]
        cache[(Signed binary cache)]
    end

    subgraph fleet[Fleet]
        dev1[Device]
        dev2[Device]
        devN[Device ...]
    end

    subgraph imaging[Imaging plane]
        station[Imaging station NUC<br/>PXE + facter + agent]
        bare[Bare hardware]
    end

    op -->|SSO login| idp
    op -->|edit config| console
    console -->|resolve roles| idp
    console -->|read/write, gated| overlay
    console -->|check-ins, status| pg
    console -->|validate every write| gate
    console -->|build release before promote| gate
    gate -->|eval + build against| overlay
    gate -->|publish signed closures| cache

    overlay -->|comin follows rings/branch| dev1
    overlay -->|comin follows rings/branch| dev2
    overlay -->|comin follows rings/branch| devN
    cache -.->|substitute release| dev1
    cache -.->|substitute release| dev2
    cache -.->|substitute release| devN
    dev1 -->|agent check-in facts| console
    dev2 -->|agent check-in facts| console

    bare -->|PXE boot| station
    station -->|report discovered| console
    console -->|image + enrol| station
    station -->|becomes| devN

Solid arrows are the steady control flow; dotted arrows are the binary-cache substitution path that only appears once build-before-promote is enabled.

How a change reaches a device

sequenceDiagram
    actor Op as Operator
    participant C as Console
    participant G as Gate-runner
    participant R as Overlay repo
    participant D as Device (comin)

    Op->>C: edit setting / policy / re-parent
    C->>G: validate candidate (scoped to blast radius)
    Note over G: nix eval of affected hosts,<br/>in memory-bounded batches
    G-->>C: accept / reject (distilled reason)
    C->>R: commit (SSO-attributed)
    Note over C,R: slow validation? detach,<br/>notify the operator with the outcome
    Op->>C: start rollout (waves)
    C->>G: build ring release
    G->>G: realise closures, sign, publish to cache
    C->>R: move rings/<group> branch to target
    D->>R: comin pulls the ring branch
    D-->>D: substitute release from cache (no local build)
    D->>C: agent check-in: on target

Workers and their knobs

The eval/build work is deliberately its own tier so it scales without touching the control plane. See Scaling to 10,000+ devices for the measured numbers.

A gate-runner is stateless apart from its warm overlay clone and its cache, so adding capacity is adding a worker; the fail-closed gate lives in the control plane’s availability domain (writes are refused, never committed unvalidated, when no worker is reachable).

The imaging station (inspoelstraat)

A station turns bare hardware into an enrolled fleet member. It runs its own NixOS appliance (PXE, nixos-facter, the imaging runner) and the Sextant agent, and is registered in the fleet so the console can mint its report credential and offer it as an imaging target.

flowchart LR
    bare[Bare device] -->|1. PXE boot| disc[Discovered<br/>in console]
    disc -->|2. operator picks<br/>hardware profile| job[Image job]
    job -->|3. partition, install,<br/>Secure Boot + TPM2| prov[Provisioned]
    prov -->|4. per-device credential<br/>shown once| enrol[Enrolled device]
    enrol -->|5. agent check-in| fleet[Fleet member]
    fleet -->|6. comin converges| target[On target revision]

The station itself is a NixOS host that reports facts and self-updates via comin, and is tracked in the fleet’s infra group - Sextant manages the machine that images the fleet the same way it manages the fleet.

Reference station

The BB Open reference station, a sizing baseline for a municipal deployment:

A station is modest hardware: imaging is bursty and operator-attended, and a station doubling as a build worker only runs heavy nix work when no imaging run is active.

Multi-tenancy

Each organisation runs as its own cell - a private console, database and overlay repo, no shared process (see decision record 0009). The diagrams above describe one cell; scaling to many customers is running more cells, managed as declarative data the same way Sextant manages devices.

See the decision records for the reasoning behind each choice.

Scaling to 10,000+ devices

Sextant targets fleets of 10,000+ devices per organisation. This chapter is the reference architecture for that scale: which parts grow with the fleet, which do not, and the numbers behind each decision. The pilot deployment is this exact architecture at N=1 per tier - scaling out means adding workers to a tier, not redesigning.

Four planes, scaled independently

Every component is FOSS and self-hosted in the customer’s datacenter or a sovereign cloud. No proprietary or extra-territorial dependency.

The check-in path (observed plane) is not the bottleneck: 10,000 devices at one check-in per 60s is ~170 writes/s, comfortably inside a single Postgres with batched upserts and a partitioned status table.

Why devices fetch instead of build

comin converges each device by evaluating the overlay and rebuilding locally. At pilot scale that is fine; at 10,000 devices it means the same derivations are compiled 10,000 times on weak edge hardware, and a rollout’s wall-clock is the slowest device’s build.

At scale the pipeline builds once, centrally, per ring - build-before- promote: the delivery pipeline realises ring N’s closures on the build workers, pushes them to the signed binary cache, and only then flips the ring branch. Devices substitute (download) instead of compiling. Ring ordering naturally staggers cache load.

This is the largest gap between the pilot and the enterprise posture, and the first slice to build.

Why the gate is batched, and what the numbers say

The gate proves a change evaluates before it reaches git (see Safe writes). Forcing every host’s toplevel in a single nix process scales memory with the fleet - a whole-fleet evaluation OOM-killed the runner well before 100 hosts. The gate therefore bounds memory per unit of work: peak memory is the unit, not the fleet, and every affected host is still evaluated.

The numbers below are from July 2026 and predate the measurements. An earlier version of this page quoted a chunk size of 12 as safe. It was not: one host toplevel peaks at about 1.5 GiB, so the chunker was re-derived from measurement to 1, and then largely superseded by nix-eval-jobs, which bounds memory per worker so an oversized evaluation fails its own host instead of killing the runner. Verdicts are now also memoised per configuration shape, so a normal edit re-evaluates only what it changed.

For the measured floor - what belongs to NixOS, what belongs to us, and the figures behind each - read the scale architecture note in the repository. That document is maintained against measurements; this chapter is the shape of the system around them.

Units of work are independent, which makes them the unit of horizontal scaling. The July projection for org-wide validation of 10,000 hosts, at the chunk size believed safe at the time and ~45s per chunk:

Measured on a synthetic 10,000-device fleet (4 hardware profiles x 3 device classes x 10 groups, two filtered org policies, 1% device-level overrides; classes_bench_test.go):

The conclusions fall out of the table:

• A scoped change stays interactive. Its blast radius is a handful of hosts; the gate evaluates only those (AffectedHosts). Metadata-only changes (groups, access, governance) skip the evaluation entirely - they cannot alter any device’s build.
• A genuinely org-wide change is validated asynchronously. It flows through the delivery pipeline, where chunk-parallel workers evaluate the full fleet before the first ring promotes. Nobody waits 10 hours at a save button.
• Interactive org-wide feedback uses equivalence classes. 10,000 devices produced by the same generator collapse to dozens of distinct config shapes (hardware profile x settings signature). An option or type error fails every host in its class, so evaluating one representative per class catches it in minutes. The full per-host evaluation still runs in the pipeline - sampling narrows feedback latency, never the guarantee. The class partitioner is security-critical code and is treated accordingly (tested exhaustively, reviewed as part of the gate).

Availability

The gate is fail-closed: if no gate worker is reachable, config writes are refused rather than committed unvalidated. Gate workers therefore live in the control plane’s availability domain (at least two at scale), not on hardware that may be powered off. Build workers may come and go - a missing build worker delays a rollout, never corrupts one.

Dogfooding the infra

Eval and build workers are themselves NixOS machines - so they are enrolled as Sextant devices in an infra group and managed declaratively by the product they serve. Scaling the build plane is enrolling another worker.

Status of this posture

1. Build-before-promote - SHIPPED: the pipeline builds a ring’s closures into the signed cache before the ring branch flips; devices substitute (see Ship an update).
2. Memory-bounded parallel gate - SHIPPED. nix-eval-jobs bounds memory per worker, pinned to the fleet’s own nixpkgs; the chunker remains as the fallback for a gate image built before it existed. The pilot deployment runs 1 worker within its memory limit.
3. Verdict memoisation - SHIPPED: a gate verdict is remembered per (source, globals, configuration shape), so an edit costs the shapes it changed rather than every shape in the fleet.
4. Equivalence-class sampling - SHIPPED for every unbounded validation (direct writes, change submit, merge revalidation).
5. Infra group - PLANNED: build/eval workers enrolled and managed by Sextant itself (scheduled with the hardware test round).

Decision records

Architecture decisions are recorded as ADRs in the repository under docs/adr/. They cover the ground-up rebuild in Go, the policy model that compiles to the scope chain, Postgres-only durable state, the SSR + API contract, the generated settings catalog, assurance and approvals, API and credential security, tenant isolation, language choice per workload, the update funnel’s ring branches, the remote gate-runner, cohort/canary pinning within a wave, custom overlays managed from the console, and why hardware is a filter rather than a fourth scope.

Read them in docs/adr/ in the source repository.

Roadmap after 1.0

What 1.0 contains is in the fit/gap specification; this is what comes after, with the trigger that forces each item rather than a wish list.

It lives in the repository at docs/roadmap.md, next to the decision records and the scale measurements it refers to, so a change to the plan and the change that caused it land in the same commit.

How this project works

Sextant Fleet is a DAWO community project. This chapter is the part of the handbook that is not about running a fleet: who decides, how a change reaches a release, what we ask of a contributor, and where to send a vulnerability.

It exists because those questions get asked by people evaluating the project, not just by people writing code, and answering them once in public is cheaper than answering them per e-mail.

The short version

What stewardship means, and what it does not

The steward keeps the roadmap, reviews what comes in, cuts the releases and answers for the security process. That is work, and somebody has to do it.

It is not a licence to change the deal. The code is EUPL 1.2 and stays that way; an organisation that wants to run, support or resell Sextant Fleet needs nobody’s permission, and a fork is a right rather than a threat. The one thing that is not covered by the licence is the name the project ships under. Sextant Fleet, and sextantfleet.com with it, is claimed as a trademark by BB Open Solutions B.V.; the application is filed and the registration is not granted yet. A fork may say it is based on Sextant Fleet; it may not present itself as Sextant Fleet.

Decisions are written before they are code

Anything that shapes the product goes into an architecture decision record before it goes into the codebase. The ADRs live in docs/adr/ and are numbered in the order they were taken; they say what was decided, what it rules out, and what would have to change for the decision to be revisited.

The rule behind it: we would rather argue about a design in writing than discover the disagreement in review. It also means somebody joining two years from now can read why the product is shaped the way it is without asking anyone.

Contributions

Contributions come in under the EUPL 1.2, like everything else here; you keep your copyright. The full standard is in CONTRIBUTING.md; the parts people trip over are:

• An issue first. Features, fixes and enhancements start as an issue, and the pull request references it.
• Conventional Commits, checked by a hook while you write and by CI when you push. The subject is the smaller half; the body should say why.
• Stage the files the commit is about, by name. Not git add -A. A message can pass every check and still describe the wrong contents.
• just ci green before a merge. It mirrors the CI workflow exactly: formatting, vet, lint, race tests, the coverage floor, the Nix build, the catalog drift guard and the agent checks.

Pull requests on Codeberg are welcome and are not turned away for being in the wrong place; a maintainer applies them to canonical with your authorship intact.

AI assistance is disclosed

Where AI assists, we say so and name the model, with a commit trailer such as Code AI-assisted (Claude Fable 5); testing, review and integration by a human.

The code remains the developer’s responsibility either way, and it has to be understandable by a human who has to troubleshoot it at three in the morning. That is a stricter bar than “it works”.

Security

Send it to security@bb-open.com. You do not need a proof of concept or certainty — “this looks wrong and here is why” is a useful report.

• Acknowledged within three working days. If you hear nothing, chase it; assume the mail went astray rather than that you are being ignored.
• An assessment within ten working days: whether we can reproduce it, how serious we judge it, and roughly when a fix lands.
• Credit in the release notes unless you would rather not be named.

We ask for a reasonable window before publication, and we will not use that window to argue about severity. If we disagree with your assessment we will say so plainly, and you remain free to publish. The full policy, including what is in scope and what we consider serious, is in SECURITY.md.

Releases

Releases are cut from canonical, tagged, and shipped with notes that name what changed and who reported what. Supported releases get security backports. Organisations with a support agreement hear about a security release directly; everyone else reads the advisory in the repository.