DAWO Handbook

DAWO is a declarative control-plane for fleets of NixOS devices. Configuration is data in a git overlay; Nix turns that data into signed system closures; devices pull and converge (comin). Sextant is the human and API surface that edits the data safely, proves it builds, stages the rollout, and reports what each device actually runs. It is not MDM: declarative pull, no live command channel, every change an audited git commit.

This handbook is for operators running a DAWO fleet and for engineers working on it. It is built with mdBook and served self-hosted; it uses no external CDN.

The lifecycle at a glance

1. Set up an imaging station - a NUC or mini-PC that boots devices over the network and reports what it sees to the console.
2. Image a device - the console dispatches an image job; the station runs the install and reports progress until the device is on disk and converging.
3. Manage it - configuration flows organisation -> group -> device; every change passes the Nix gate and can be reviewed as a change-request.
4. Update it - a rollout ships a new revision in waves, each gated on health and soak, with an optional manual test gate.
5. Retire or wipe it - an audited intent the device acts on locally; a crypto-wipe destroys the disk’s keys, and is armed per device.

Start with setting up an imaging station.

Install and configure Sextant

Git is not an integration in Sextant - it is the storage. A fleet’s whole configuration is data in a git overlay repository (fleet.json, policies, overlays). Sextant reads and writes that repository directly: every change is an audited commit pushed to the remote, and the remote is the source of truth. So “connecting Sextant to git” means pointing the console at your overlay repo; there is nothing else to wire.

What you need

• An overlay repository - a git repo that consumes the DAWO core flake and holds your fleet.json. One repo per organisation (tenant). It is the same repo the devices follow via comin.
• Postgres - the observed plane (check-ins, tokens, image jobs, prefs). A single instance next to the console is fine; it partitions per tenant.
• An OIDC identity provider - console login (one IdP per instance), mapped to roles by group. Optional LDAP for the group-picker source.
• A validation gate - eval runs nix in-process; remote delegates to a small nix-capable gate-runner (the console image itself ships no nix). Use remote in production; it is fail-closed.

Connect the console to your overlay repo

Point the console at the repo and a push remote:

--repo /data/overlay        # the working tree the console edits
--git-remote origin         # the push remote (HA source of truth)

Under Helm (deploy/helm), the same as values:

gitRemote:
  url: https://your-forge.example.com/org/sextant-overlay.git
  branch: main
  netrcSecret: sextant-overlay-netrc   # credentials for a private remote
gateMode: remote                       # or eval / none
oidc:
  issuer: https://id.example.com
  clientId: "<client-id>"

The console clones the repo, keeps its snapshot in sync (the remote is authoritative - commits made by engineers or CI show up without a restart), and the devices follow the same repo: comin on each device tracks rings/<group> (or main), so a rollout that advances a ring branch lands on the devices in that ring.

First deploy, end to end

1. Create the overlay repo from the DAWO core (a fleet.json with your org and the core as a flake input). Push it to your forge.
2. Deploy the console (Helm chart, container, or the NixOS module) with the gitRemote, gateMode, oidc and Postgres settings above. Behind TLS, set --secure-cookies (the console refuses session cookies without it on a non-loopback address).
3. Deploy a gate-runner if gateMode: remote; it keeps a warm clone of the overlay and evaluates each candidate before the console commits.
4. Log in via your IdP. Enroll a device, assign it to a group, edit settings - every change passes the gate and commits to the overlay. Stage a rollout to land updates in waves.

Build-before-promote

At scale, ring promotion should not mean 10,000 devices each independently compiling the same closures on weak edge hardware. With gateMode: remote and a gate-runner cache configured (a signing key secret, and optionally a dedicated cache host), a rollout’s wave builds its release into that signed binary cache before its ring branch flips - devices then substitute (download) the pre-built closure instead of building it. Enable the console side with releaseCache: true. See Scaling to 10,000+ devices for the reasoning, and Ship an update for what a wave’s Building status means day to day.

Notification e-mail (SMTP)

In-app notifications work with no extra setup. To also deliver them by mail, an owner configures SMTP per organisation under E-mail (SMTP) in the console (host, port, from, security). The password is set one of two ways:

• A secret reference (recommended) - enter the name of a secret; the value lives in agenix or a cluster Secret mounted at SECRET_DIR (default /run/secrets/<name>). Sextant reads only the name, never storing the value.
• A typed password - available only when SEXTANT_SECRET_KEY is set (a base64 32-byte key). The password is then sealed (AES-256-GCM) and stored in Postgres. Without the key this option is disabled and the console says so.

Both can also be set at deploy time. SEXTANT_SECRET_KEY is an environment-only secret; add it to the same secret the chart mounts (secretName).

Multi-tenant (model B): one overlay repo per organisation, isolated stores, one console instance per repo. See docs/adr/ for the decisions behind this.

Troubleshooting

The console refuses to start / refuses session cookies. Behind TLS on a non-loopback --addr, Sextant refuses to ship session cookies without --secure-cookies (or SEXTANT_SECURE_COOKIES=true) - this is deliberate fail-closed behaviour, not a bug. Set the flag.

Every write is refused with gateMode: remote. The gate is fail-closed: no reachable gate-runner means no writes, by design. Check the gate-runner’s /healthz before flipping gateMode to remote, and after any gate-runner redeploy.

A rollout wave never leaves “Building”. With build-before-promote enabled (releaseCache: true + gateRunner.cache), check the gate-runner’s cache is healthy and its signing key secret is present - a wave cannot promote until its release lands in the signed cache.

SMTP is configured but no mail arrives. Confirm the password resolved: a secret reference must exist under Secrets with a value the runtime can actually read (agenix or the mounted SECRET_DIR); a typed password needs SEXTANT_SECRET_KEY set on the deployment, or the console disables that option outright.

Set up an imaging station (NUC)

An imaging station (the “inspoelstraat”) is a small always-on box that boots target devices over PXE and reports what it finds to the console. From there you dispatch imaging jobs at the target devices - the station itself is set up once (and occasionally re-provisioned), then left running.

This chapter is the full procedure, kale NUC to working station. Steps marked Manual step are not driven by the console today - they are candidates for future turnkey automation, but for now someone does them by hand at a keyboard or a workstation shell.

Checklist

Work through these in order. The station is done when the last box is ticked.

• Manual step - write a NixOS minimal USB installer
• Manual step - boot the NUC from the USB, bring up SSH
• Manual step - run nixos-anywhere from your workstation to install the -install variant (systemd-boot + LUKS passphrase, Secure Boot off)
• Register the station in the console and mint its report credential (owner, console step)
• Manual step - put the credential on the station and point the agent at the console
• Verify the station shows as registered and reachable in the console
• Manual step - PXE-boot a test target device on the station’s network and confirm it appears as discovered
• Wire the station to your fleet overlay so it can dispatch real imaging jobs (see Image a device from the console)
• (Later, once the fleet needs it) Manual step - re-provision the station on the -sb variant (Secure Boot on) and enrol TPM2, then move it to the steady-state dawo-inspoelstraat configuration

1. Write a NixOS minimal USB installer — manual step

lsblk                              # find the whole stick, e.g. /dev/sdb (NOT sdb1)
sudo dd if=<nixos-minimal>.iso of=/dev/sdX bs=4M status=progress oflag=sync

Double-check the device node against lsblk before running dd - the whole disk, not a partition, and never the workstation’s own disk.

2. Boot the NUC and bring up SSH — manual step

Plug in the USB and wired ethernet. Power on; the NUC’s boot menu is F10 if it does not auto-boot the stick (some models use F2 or Esc - check the splash screen). At the installer prompt:

sudo systemctl start sshd
sudo mkdir -p /root/.ssh
sudo tee /root/.ssh/authorized_keys < keys/id_ed25519.pub   # operator pubkey
ip -brief a                                                  # note the NUC's IP

3. Install with nixos-anywhere — manual step

Pick a LUKS passphrase and store it in your password manager first, then drive the install from your workstation (not the NUC itself):

cd inspoelstraat-appliance
printf %s '<LUKS-passphrase>' > /tmp/luks.key
NIX_SSHOPTS="-o IdentitiesOnly=yes -o StrictHostKeyChecking=no" \
  nix run github:nix-community/nixos-anywhere -- \
    --flake .#dawo-inspoelstraat-install \
    --target-host root@<installer-ip> \
    -i ../keys/id_ed25519 \
    --generate-hardware-config nixos-generate-config ./hardware-configuration.nix \
    --phases disko,install \
    --disk-encryption-keys /tmp/luks.key /tmp/luks.key
rm -f /tmp/luks.key

Notes:

• --generate-hardware-config re-probes the NUC and writes its real kernel modules before building - do not reuse a hardware config from a different box.
• Disko targets /dev/nvme0n1 (the NUC standard). Adjust the disko module if your hardware differs.
• printf %s (not echo) writes the passphrase with no trailing newline. This matters again in step 4 - see the troubleshooting note below.
• This installs the -install variant: Secure Boot off, no TPM2 key enrolled, systemd-boot + a LUKS passphrase at boot. That is deliberate - Secure Boot and TPM2 are enrolled later, once the box is confirmed working.

The NUC reboots into its fresh, minimal NixOS install once this completes.

4. Register the station in the console

In the console, under Organisation -> Imaging stations (owner reach):

1. Register an imaging station: give it a tag (e.g. dawo-inspoelstraat, matching [a-z0-9][a-z0-9-]*), an optional description and site.
2. Mint credential: the console generates a bearer token and shows the report endpoint (https://<console-host>/api/station/<tag>/report) and the token together. This is a one-shot reveal - the token is never shown again, so copy both before leaving the page.

5. Put the credential on the station — manual step

Configure the station’s report-agent (in the inspoelstraat-appliance flake) with the endpoint and bearer token from step 4, so it can POST its discovery reports to the console. Exactly how the agent picks up the token is a station-flake concern (a runtime credential file is the usual pattern, same shape as the per-device agent credential - see Install and configure Sextant); the important operational rule is the same one that trips up the per-device credential too:

Write the token with no trailing newline. printf %s '<token>' > path or a paste that strips the newline both work; echo '<token>' > path (or a plain heredoc) leaves a \n at the end of the file. A bearer token with a stray newline never matches what the console minted, and the station’s reports fail authentication with no more detail than “unauthorized” - see Troubleshooting below.

6. Verify the station is live

Back in Organisation -> Imaging stations, open the station. If the agent is configured correctly it appears registered and (once it has reported at least once) shows a device count from its PXE network. If nothing shows up yet, PXE-boot a target device on the station’s network next - see step 7.

7. Confirm discovery — manual step (boot a target device)

PXE-boot any spare target device on the station’s network. It should appear under the station’s discovered devices within a minute or two. This proves the whole chain: agent -> report endpoint -> console -> discovered plane.

8. Wire the station to your fleet overlay

The station is now registered and reporting, but imaging real devices from it needs the station’s runner wired to your fleet overlay (so it can build and install the right configuration for each target). See Image a device from the console for the imaging flow itself, driven from Enrollment in the console.

Later: Secure Boot and TPM2

The steps above deliberately leave Secure Boot off and TPM2 unenrolled - get a working station first. Once it is confirmed:

1. Manual step - re-provision with the -sb flake variant (Secure Boot enabled, still a LUKS passphrase at boot).
2. Manual step - enrol TPM2 on the device so the passphrase is no longer needed at boot (it remains as break-glass recovery - see Manage secrets).
3. Move the station onto the steady-state dawo-inspoelstraat configuration.

This same Install -> Secure Boot -> TPM2 -> Done progression is what the provisioning wizard walks an operator through per target device - see Image a device from the console.

Troubleshooting

The station never shows a device count / discovered devices. Check, in order: the station’s network actually serves PXE to the target device; the agent process is running on the station and can reach the console’s report endpoint (curl -i <report-url> from the station should at least get a 401, not a connection failure); and the credential file has no trailing newline (see step 5).

Minting a new credential “loses” the station. It does not - re-minting only invalidates the previous token. Update the credential file on the station with the new one (again, no trailing newline) and it resumes reporting.

nixos-anywhere fails partway through disko/install. Re-run it - nixos-anywhere --phases disko,install is safe to repeat against a booted installer. If it fails consistently on the same step, boot the installer again fresh (a half-partitioned disk can confuse a second disko pass) before retrying.

SSH from the workstation hangs or is refused. Confirm ip -brief a on the installer shows an address reachable from your workstation (not just link-local), and that sudo systemctl start sshd was run after the installer environment finished booting, not during POST.

Image a device from the console

Once a station is registered and reporting (see Set up an imaging station), imaging target devices is driven entirely from the console’s Enrollment page - no shell access to the station is needed for a routine batch.

Step 1: choose the station

Open Enrollment, pick the station whose PXE network the target devices are on, and continue. If no station is listed, none is registered yet - see Set up an imaging station.

Step 2: batch-dispatch the discovered devices

PXE-boot the target devices; they appear as discovered rows (MAC, vendor, model, disk size) under the chosen station. Imaging is batch-only by design - one audited pass images a whole rack rather than one device at a time:

1. Set the shared hardware profile (suggested from make and model where the profile catalog matches), class and group for this rack of otherwise-identical machines.
2. Tick the devices to include, and give each one a CMDB name (the tag it will enrol under).
3. Dispatch imaging. This creates each device’s record, captures its hardware specs, and queues an image job per device.

Step 3: watch the jobs run

The imaging-jobs table shows each device’s status live: pending -> imaging (with a progress bar and current step) -> installed, or failed. Under the hood, the station runner claims each job, resolves the target’s IP from its DHCP lease, runs nixos-anywhere against that device’s generated configuration, and bakes in the device’s one-time agent credential.

Open the provisioning wizard (linked from the jobs table) for the guided, per-device view of the same batch. It walks a four-phase stepper - Install -> Secure Boot -> TPM2 -> Done - and adapts to what each device’s hardware profile actually needs:

• A device whose group needs no Secure Boot simply skips that phase and goes straight from installed to done.
• When a device reaches the Secure Boot phase, the wizard shows a brand-specific manual action (the firmware entry key and the exact BIOS steps - e.g. Lenovo/ThinkPad enters on F1, Intel NUC on F2, HP on F10) and an in-console reboot control, since only a human at the keyboard can toggle Secure Boot in firmware.
• The wizard also tells you when it is safe to unplug a device (once its phase reads done - before that, keep it cabled so it can keep checking in and converging).
• If a device produced a one-time LUKS recovery key during provisioning, the wizard shows it once (a break-glass secret store keeps it recoverable later for an organisation owner - see Manage secrets).

Step 4: converge

Each imaged device boots, checks in, and converges its configuration. It then shows on the device page with its facts and posture (Secure Boot / TPM2 state), and on Compliance if anything about it needs attention.

Troubleshooting

A discovered device never gets an IP / the job stalls at “imaging”. The runner resolves the target’s IP from its DHCP lease; if the station’s PXE network hands out leases slowly or the device NICs are ambiguous (multiple discovered rows with a similar MAC prefix), remove the stale discovered row and re-PXE-boot the device to get a fresh lease and report.

A job fails. The jobs table and the wizard both surface the failure message inline. Most failures are hardware-profile mismatches (wrong disk device assumed by disko) or a target that lost network mid-install - cancel the job, fix the profile or cabling, and dispatch it again; a failed job does not block the rest of the batch.

The Secure Boot step never completes. Check the manual firmware action shown in the wizard was actually completed in the BIOS (Setup Mode -> Enabled -> Save & Exit) before rebooting; a reboot without that toggle just returns the device to the same phase.

The device converges but its posture still shows Secure Boot/TPM2 as not enforcing. Give it one more check-in cycle - posture is self-reported by the device on check-in, so it lags the wizard’s own phase tracking by up to one interval.

Ship an update

Updates (in the sidebar under Shipping) is the one board for the whole journey from a proposed configuration change to it running on the fleet: propose -> review -> ready -> rolling out in waves. Changes and Rollout are drill-downs reached from cards on this board, not separate destinations you navigate to directly.

Step 1: review proposed changes

Every configuration edit - a setting, a policy, an integration - is staged as a change: a named, audited unit of work on its own branch. A change moves through:

• Draft - staged edits, not yet validated.
• Building - the Nix gate is evaluating the change (see Safe writes and the Nix gate).
• Ready - the gate passed; it can be reviewed and merged.
• Failed - the gate rejected it; the distilled error is shown inline (see Troubleshooting below).

Open a change directly from the Updates board (Change ID + title, then Open change), or let one open automatically: if your organisation requires change requests (Access -> Approval flows), saving settings from the Configuration editor stages the same edits as a fresh change instead of committing straight to git, and lands you back here.

For a Ready change: View diff shows exactly what will change on disk per host; Approve merges it (blocked on yourself if four-eyes review is required); Reject abandons it. A Draft can be Submitted (sends it to the gate) or abandoned.

Step 2: roll out

Once there is a current revision to ship, this section is either an active rollout in progress or the button to start one against the latest merged revision.

Starting a rollout kicks off the wave plan from Step 3 (below) against the current revision. Each wave shows a live status label as it progresses:

An owner can approve an awaiting wave to let the rollout proceed, or cancel the whole run.

Step 3: the rollout procedure

Two panels, both owner reach:

Wave plan. Each wave (a “ring”) pins one device group plus its promotion gates: a name, the group, a soak window (minutes healthy on target before the next wave may start), a minimum healthy % (defaults to 100 - every device healthy), an optional max at once (a count-capped canary that widens cohort by cohort instead of releasing the whole group at once), and whether the wave requires approval (a human checkpoint - the enterprise “test sign-off” step). Size a wave small first (a canary), then wider; tune the progression by ordering waves and sizing their groups.

Governance. Three checkboxes, organisation-wide (mirrored on Access -> Approval flows):

• Require change request - configuration edits must go through a reviewed change (this board’s Step 1), never a direct commit.
• Require four-eyes - a change may not be merged by its own author.
• Require test wave - a rollout must have a gated (manual-approval) wave before it starts; an owner can explicitly skip this per rollout, and the skip is logged.

See How a rollout ships and Approval flows for the concepts behind both panels.

Troubleshooting

A change is stuck in Failed. The card shows the distilled error - the actionable line pulled out of the gate’s evaluation trace (e.g. “device X: unknown hardware profile ‘Y’”). Fix the underlying edit and resubmit; if the message is not enough to act on, the merge/submit response (and the gate-runner’s own logs) carry the full multi-line trace behind a “technical detail” fold.

A wave sits on “Building” for a long time. Build-before-promote realises the wave’s whole release into the signed binary cache before its branch flips - this is centralised, one-time compute per distinct configuration shape, not per device, but it is real wall-clock on the build workers. Check the gate-runner/build-worker health before assuming something is stuck; a missing build worker delays a rollout, it does not corrupt one.

A wave never leaves “Deploying” / “Soaking”. Devices pull on their own schedule (comin), so convergence is not instantaneous - check the Devices page or Compliance for the specific machines that have not landed on the target revision yet; “behind” incidents there point at exactly which ones and since when.

“Require test wave” blocks starting a rollout. Either add a wave with require approval checked to the plan, or (if you are an owner and this run genuinely does not need one) use the explicit skip offered on the start form - it is logged either way.

Track compliance

Compliance answers one question: which devices are not to spec right now, and why. It is the full drill-down behind the compliance donut on Overview - the donut caps its attention queue at 8 items; this page lists every open incident.

Reading the page

The three summary chips (All, Critical, Warning) double as a filter - click one to narrow the device table below it. Every active (non-retired) device appears, worst status first:

• Critical - an error the device reported, or a wipe that failed or was refused.
• Warning - offline, never checked in, or behind its target revision.
• To spec - no open incident.

Each row lists the device’s issues with a short title, a detail (the specifics - e.g. which revision it is running versus its target, or when it was last seen), and a suggested action.

What raises an incident

A single device can carry several incidents at once (e.g. offline and behind). Suggested actions point at the fix: verify imaging and connectivity for never-seen, check power/network for offline, check the rollout and device logs for behind, open the device to inspect the failure for errored, and re-arm or clear the interlock for a refused wipe (see Update, retire and wipe).

Policy exposure

Below the device table, a per-policy table shows where each policy is assigned (its scope targets) and, of the devices under those targets, how many currently carry an open incident - a revision-level proxy for “this policy may not actually be applied everywhere it is assigned yet”. A policy with no assignments shows as unassigned; it has no effect until targeted.

Troubleshooting

A device shows as behind right after a rollout started. Expected - “behind” just means the device has not yet pulled and converged to its target revision. Give it a check-in cycle or two before treating it as stuck; if it stays behind well past the wave’s soak window, check Ship an update for whether the wave itself is stalled.

A device never clears “never seen”. It has an enrolled record but has not reported at all. Confirm it was actually imaged (see Image a device from the console) and that its agent credential and network reach the console - the same credential/newline pitfall documented in Set up an imaging station applies to a device’s own agent credential too.

A wipe keeps showing “refused”. The root executor requires the device to be locked first (or the intent to be forced) before it will act on a wipe. Re-arm the wipe intent, or lock the device first, then retry - see Update, retire and wipe.

Update, retire and wipe

Update

Updates ship as a rollout: a new revision promotes through ordered waves, each gated on health and a soak window, with an optional manual test gate. Day to day this is driven from the Updates board - see Ship an update for the full walkthrough and How a rollout ships for the concept behind it.

Retire

Retiring a device keeps its record for audit but stops image builds, check-ins and rollout counting. Reactivation is an explicit, audited step.

Lock and wipe

Lock and wipe are intent-as-data: the console records the intent as an audited change (on a device’s own page, under the red-bordered remote-actions panel); the device pulls it on check-in and acts locally. There is no live command channel.

• Lock locks all sessions and persists across reboot; clear the intent to release.
• Wipe cryptographically erases the device by destroying its LUKS key slots. It is irreversible and gated: the root executor refuses a wipe unless the device is locked first, unless the intent is explicitly sent with force (which the console’s wipe action does, backed instead by a type-the-device-tag confirmation as the human safety net). Arm a device for wipe only when it is cleared to be wiped.

Troubleshooting

A wipe shows “refused” on the device page or in Compliance. The device declined the intent - typically because a local interlock blocked it even with force set. Clear the intent, confirm the device is locked, then re-send the wipe.

A wipe shows “failed”. The device attempted the crypto-wipe but never confirmed completion back to the console. Treat the disk as not yet confirmed destroyed - verify by other means before reusing or disposing of the hardware.

Retiring a device does not remove it from the fleet count. That is by design - retiring keeps the audit record and simply stops new image builds, check-ins and rollout counting. Use Remove instead if the device should be unenrolled entirely; unlike a retire, removal cannot be undone by reactivation.

Manage secrets

Sextant handles two different kinds of secret, and the Secrets page (plus a device’s own page) is where both are managed. They are not interchangeable: one is a reference an operator points settings at; the other is material the platform generated on a device’s behalf and holds so it can be recovered later.

Secret references (for settings and integrations)

A setting field that needs a secret value - a NetBird setup key, an LDAP bind password, a Wazuh enrollment secret, an SMTP password - never accepts the value itself in the console. Instead you register a name, and settings pick that name from a list:

1. Open Secrets.
2. Register a secret: give it a name ([a-z0-9][a-z0-9-]*, e.g. netbird-setupkey) and an optional description.
3. Point any secret-typed setting field at it, in Configuration editor or Integrations - the field renders as a picker of registered names, with a shortcut to register a new one inline if you started from the setting itself.

The console never sees or stores the plaintext: only the name travels through the config repo and Sextant’s own state. The device resolves the name to the decrypted material at runtime via agenix. Removing a registered name breaks the build for anything still pointing at it - the console warns before you confirm.

Per-device secrets (break-glass recovery)

Some material is generated for a specific device during provisioning and has nowhere else to live: a LUKS disk-encryption recovery passphrase, or a break-glass local-administrator password. TPM2 enrolment makes the LUKS passphrase unnecessary at day-to-day boot, but it remains the recovery path if TPM2 unsealing ever fails - so it has to survive somewhere, encrypted at rest, reachable only to someone who genuinely needs it.

Sextant seals this material (AES-256-GCM by default; a drop-in external key manager such as OpenBao/Vault is the production posture) the moment it is produced - at provisioning, from the imaging wizard - and never stores it in the clear.

Revealing it:

• Reveal is organisation-owner reach only - not editor, not viewer.
• From a device’s page (or the provisioning wizard, while a job is still fresh), Reveal shows the plaintext exactly once, rendered directly on the response - never redirected, so it never lands in a URL, browser history, or an access-log line.
• Every reveal is recorded: who, and when. There is no silent read.
• Once revealed, treat it as no longer fresh - the console flags a previously-revealed secret as such, since anyone who saw it once could have copied it.

If the secret store is not configured (no encryption key set), Sextant does not store per-device secrets at all rather than write them in the clear - the one-time value is then shown only at the moment it is generated (during imaging) and never again.

Troubleshooting

A setting’s secret picker is empty. No secret has been registered yet under that name pattern - register one on the Secrets page first, or use the inline shortcut next to the field.

Removing a secret reference breaks a build. Expected - any setting still pointing at that name fails the Nix gate on the next change. Re-point the setting at a different registered name (or clear it) before removing the reference, not after.

“Reveal” is not available on a device. Either the secret store is not configured for this deployment (no SEXTANT_SECRET_KEY / no external sealer wired up), no such secret was ever generated for this device, or you are not signed in with organisation-owner reach - reveal is deliberately not available to editors or viewers.

The revealed LUKS passphrase does not unlock the device. Confirm you copied it in full (it is shown once, select-all) and that you are unlocking the current value - if the device was re-imaged since the secret was last generated, an old reveal (or a note copied from an earlier session) no longer matches.

Notifications

The bell icon in the top app bar opens Notifications: an in-app inbox for things that need your attention - a change ready for review, a wave awaiting approval, an incident, and similar events. An unread count badges the bell.

• Click a notification to jump straight to the thing it is about (the change, the device, the rollout) and mark it read at the same time.
• Mark all read clears the unread badge without visiting each item.
• An empty inbox and “notifications unavailable” (the backing store is not configured for this deployment) are both shown plainly rather than as an error.

In-app notifications work with no extra setup. To also receive them by e-mail, an organisation owner configures SMTP once under Organisation -> E-mail (SMTP) - see Install and configure Sextant for the setup details (host/port/from, and the two ways to hold the password). Mail delivery is additive: turning it on does not change what shows up in the in-app inbox, only whether the same events also arrive by e-mail.

Push notifications are not implemented yet.

Troubleshooting

Notifications never arrive at all (in-app). The page shows “unavailable” rather than an empty inbox when the notifications backing store is not configured for this deployment - that is a deploy-time gap, not a per-user setting.

In-app notifications work but e-mail does not. Check the SMTP configuration under Organisation -> E-mail (SMTP): host, port and the password source (a registered secret reference, or a typed password sealed with SEXTANT_SECRET_KEY). If SEXTANT_SECRET_KEY is not set on this deployment, the typed-password option is disabled and the console says so - use a secret reference instead.

A notification links to something that no longer exists. The event still happened (the notification records history), but its target (a since-abandoned change, a completed rollout) may no longer show the same detail page - this is expected once enough time has passed.

Integrations

Integrations are device-side capabilities the overlay publishes so you can turn them on and configure them per scope from the console, without editing Nix. The console shows a card per integration; a card is available once the overlay publishes its options in the catalog, and not published otherwise.

Three ship with the BB Open overlay:

• NetBird - join a self-hosted WireGuard mesh, so a roaming device stays reachable and can pull and push from anywhere. You set the management URL and a setup key.
• Directory login (LDAP) - device login against your directory over SSSD (LDAP, AD or IPA). You set the provider, domain, server and a bind secret.
• Wazuh - an endpoint security agent that reports to a Wazuh manager. You set the manager address, an agent group and an enrollment secret.

Secrets are references, never values

A field that holds a secret - a setup key, a bind password, an enrollment password - is stored as a reference: the name of a secret you registered, not the secret itself. The console renders it as a picker of registered names, so a raw secret can never be typed into the console or committed to git. The device resolves the name to the decrypted material at runtime (agenix); only the name travels through the config repo.

Register the secrets first on the Secrets page (see Manage secrets), then point the integration field at one by name.

Setting it up

1. Open Integrations. A card that reads available is ready to configure.
2. Enable it and fill the fields. Secret fields offer the registered references, with a shortcut to register a new one inline.
3. Save. This writes at the organisation scope - the Integrations page is an org-wide quick-config surface for the catalog keys the overlay publishes. To narrow an integration to one group or device instead (or to review it alongside every other setting), open the same keys from Settings (the Configuration editor) and pick a group or device with its scope selector - integration settings flow down the scope chain like any other setting.
4. Every save passes the Nix gate and commits to git like every edit (or stages as a change, if your organisation requires change requests - see Ship an update).

If a card reads not published, the overlay has not exported that integration’s options yet: add its module to the overlay and regenerate the catalog (nix eval .#catalog --json > catalog.json).

Troubleshooting

A card reads “not published” even though I added the overlay module. Regenerate the catalog (nix eval .#catalog --json > catalog.json) and restart or reload the console’s config snapshot - the catalog is generated, not live-read from the overlay’s Nix source.

Saving an integration field fails the gate. Same as any setting - the distilled error names the actionable line (e.g. an out-of-range value or an unknown option). See Safe writes and the Nix gate.

A secret field’s picker is empty. No secret reference has been registered yet - use the inline shortcut next to the field, or register one first on the Secrets page.

How a rollout ships

An update does not reach the whole fleet at once. It promotes through ordered waves (also called rings). Each wave is a group of devices; the next wave only starts once the current one is converged healthy through its soak window. A wave can require a manual approval gate - a human checkpoint that the update was tested.

At scale, a wave’s release is also built before it is promoted: the delivery pipeline realises the wave’s closures on build workers and pushes them to a signed binary cache before its branch flips, so devices substitute (download) a pre-built release instead of each compiling it independently. While this is happening the wave shows as Building on the Updates board; once the release lands in the cache the branch flips and the wave moves to Deploying. See Scaling to 10,000+ devices for the reasoning and the numbers behind it.

The Updates board (Step 3, “the rollout procedure”) shows the plan as a ladder: each wave with its device count (size it small first - a canary - then wider), soak, health floor and gate. Size a wave by its group and order; refine each wave with the gates - including an optional max at once cap, so a wave widens cohort by cohort rather than releasing its whole group in one shot.

An organisation can require a gated test wave before any rollout starts; an owner may skip it for a specific rollout, and that is logged.

Troubleshooting

A wave is stuck on Building. Build-before-promote is centralised, one-time compute per distinct configuration shape - not per device - but it still takes real time on the build workers. Check the gate-runner/build-worker’s health before assuming the rollout is stuck; a missing or overloaded build worker delays a promotion, it does not corrupt it.

A wave never reads Complete even though every device shows online and on the target revision. It may still be inside its soak window, or waiting on a manual approval - check the wave’s status label on the Updates board rather than only the device list.

Approval flows

How changes and updates are reviewed before they take effect is configurable per organisation, under Access -> Approval flows (owner). The same three toggles also live on the Updates board’s governance panel (Step 3) - they are the same setting, shown in both places:

• Four-eyes - a change may not be merged by its own author.
• Require change-request - configuration edits must go through a reviewed change, never a direct commit. When this is on, saving from the Configuration editor does not fail - it transparently stages the same edits as a fresh change and sends you to the Updates board to see it through review.
• Require test wave - a rollout must have a gated test wave first; an owner can skip it per rollout, and the skip is logged.

Every configuration change is a git commit that passes the Nix gate first, so an edit that would not build never reaches the fleet.

Troubleshooting

Saving settings redirects to the Updates board instead of just saving. Expected when Require change-request is on - the edits were not lost, they were staged as a new change. Continue the review from there (diff, submit, approve).

A change cannot be merged even though it is Ready. If Four-eyes is on, the merge button is unavailable to the change’s own author - have a different reviewer approve it.

Safe writes and the Nix gate

Every configuration change - a setting, a policy, an overlay, an integration - is a write to the git overlay. No write reaches git unless it first evaluates: the change is applied to a candidate fleet.json, and the Nix generator is run over the affected hosts. If the evaluation fails - an unknown option, a wrong type, a value out of range - the write is rejected and nothing is committed. If it succeeds, the change commits with the author’s identity from their session.

This is the core safety property: the console cannot commit a fleet that does not build.

Fail-closed

In production the gate runs remote: the console ships no Nix, and delegates the evaluation to a small Nix-capable gate-runner. Remote means fail-closed - a write that does not evaluate, or that the runner cannot be reached to evaluate, is refused. A broken or unreachable gate blocks writes rather than waving them through.

Three modes exist: eval (evaluate in-process, for a Nix-capable console), remote (delegate to the gate-runner, the production posture) and none (no gate, for tests or a console with no flake).

Where it sits

• A direct setting edit is gated before it commits.
• A change-request is gated when it is opened and again before it merges, so a reviewer never approves something that will not build.
• A rollout advances branch refs the gate already validated; it ships evaluated revisions, it does not re-open the edit path.

The gate is a type-and-build check, not a policy check. Governance - who may edit, four-eyes review, a required test wave - sits on top of it, in the change-request and rollout flows (see Approval flows and Ship an update).

Troubleshooting

A write is rejected with a short error. That message is a distilled line pulled out of the gate’s evaluation trace - usually the actual cause (an unknown option, a wrong type, a value out of range), not the whole trace. A change-request’s failure card, and any rejected save, carries the full multi-line detail behind a “technical detail” fold if the short line is not enough to act on.

Every write is refused, even ones that should evaluate fine. In remote mode the gate is fail-closed: if the gate-runner cannot be reached at all, writes are refused rather than committed unvalidated. Check the gate-runner is up and reachable before assuming the change itself is at fault.

Architecture overview

Sextant is a hexagonal Go application: a pure domain (model, resolver, policy compiler, filter evaluator) under application services, behind ports, with adapters for git, Nix, Postgres, LDAP and OIDC, and two thin transports - a server-rendered console and a JSON API over the same services.

Three planes carry the work:

• Config plane - the git overlay (fleet.json + catalog) is the source of truth. Writes are serialized, pass the Nix eval gate, and commit with SSO-attributed authorship.
• Observed plane - device check-ins, posture and hardware facts live in Postgres, tenant-namespaced.
• Imaging plane - discovery and image jobs provision new hardware from an imaging station (the inspoelstraat).

The whole picture

flowchart TB
    subgraph people[People and identity]
        op[Operator]
        idp[Zitadel OIDC<br/>+ LDAP groups]
    end

    subgraph control[Control plane - one cell per organisation]
        console[Console + API<br/>hexagonal Go]
        pg[(Postgres<br/>observed plane)]
        overlay[(Overlay git repo<br/>fleet.json + catalog)]
    end

    subgraph workers[Workers - scale out independently]
        gate[Gate-runner<br/>eval workers]
        cache[(Signed binary cache)]
    end

    subgraph fleet[Fleet]
        dev1[Device]
        dev2[Device]
        devN[Device ...]
    end

    subgraph imaging[Imaging plane]
        station[Imaging station NUC<br/>PXE + facter + agent]
        bare[Bare hardware]
    end

    op -->|SSO login| idp
    op -->|edit config| console
    console -->|resolve roles| idp
    console -->|read/write, gated| overlay
    console -->|check-ins, status| pg
    console -->|validate every write| gate
    console -->|build release before promote| gate
    gate -->|eval + build against| overlay
    gate -->|publish signed closures| cache

    overlay -->|comin follows rings/branch| dev1
    overlay -->|comin follows rings/branch| dev2
    overlay -->|comin follows rings/branch| devN
    cache -.->|substitute release| dev1
    cache -.->|substitute release| dev2
    cache -.->|substitute release| devN
    dev1 -->|agent check-in facts| console
    dev2 -->|agent check-in facts| console

    bare -->|PXE boot| station
    station -->|report discovered| console
    console -->|image + enrol| station
    station -->|becomes| devN

Solid arrows are the steady control flow; dotted arrows are the binary-cache substitution path that only appears once build-before-promote is enabled.

How a change reaches a device

sequenceDiagram
    actor Op as Operator
    participant C as Console
    participant G as Gate-runner
    participant R as Overlay repo
    participant D as Device (comin)

    Op->>C: edit setting / policy / re-parent
    C->>G: validate candidate (scoped to blast radius)
    Note over G: nix eval of affected hosts,<br/>in memory-bounded batches
    G-->>C: accept / reject (distilled reason)
    C->>R: commit (SSO-attributed)
    Note over C,R: slow validation? detach,<br/>notify the operator with the outcome
    Op->>C: start rollout (waves)
    C->>G: build ring release
    G->>G: realise closures, sign, publish to cache
    C->>R: move rings/<group> branch to target
    D->>R: comin pulls the ring branch
    D-->>D: substitute release from cache (no local build)
    D->>C: agent check-in: on target

Workers and their knobs

The eval/build work is deliberately its own tier so it scales without touching the control plane. See Scaling to 10,000+ devices for the measured numbers.

A gate-runner is stateless apart from its warm overlay clone and its cache, so adding capacity is adding a worker; the fail-closed gate lives in the control plane’s availability domain (writes are refused, never committed unvalidated, when no worker is reachable).

The imaging station (inspoelstraat)

A station turns bare hardware into an enrolled fleet member. It runs its own NixOS appliance (PXE, nixos-facter, the imaging runner) and the Sextant agent, and is registered in the fleet so the console can mint its report credential and offer it as an imaging target.

flowchart LR
    bare[Bare device] -->|1. PXE boot| disc[Discovered<br/>in console]
    disc -->|2. operator picks<br/>hardware profile| job[Image job]
    job -->|3. partition, install,<br/>Secure Boot + TPM2| prov[Provisioned]
    prov -->|4. per-device credential<br/>shown once| enrol[Enrolled device]
    enrol -->|5. agent check-in| fleet[Fleet member]
    fleet -->|6. comin converges| target[On target revision]

The station itself is a NixOS host that reports facts and self-updates via comin, and is tracked in the fleet’s infra group - Sextant manages the machine that images the fleet the same way it manages the fleet.

Reference station

The BB Open reference station, a sizing baseline for a municipal deployment:

A station is modest hardware: imaging is bursty and operator-attended, and a station doubling as a build worker only runs heavy nix work when no imaging run is active.

Multi-tenancy

Each organisation runs as its own cell - a private console, database and overlay repo, no shared process (see decision record 0009). The diagrams above describe one cell; scaling to many customers is running more cells, managed as declarative data the same way Sextant manages devices.

See the decision records for the reasoning behind each choice.

Scaling to 10,000+ devices

Sextant targets fleets of 10,000+ devices per organisation. This chapter is the reference architecture for that scale: which parts grow with the fleet, which do not, and the numbers behind each decision. The pilot deployment is this exact architecture at N=1 per tier - scaling out means adding workers to a tier, not redesigning.

Four planes, scaled independently

Every component is FOSS and self-hosted in the customer’s datacenter or a sovereign cloud. No proprietary or extra-territorial dependency.

The check-in path (observed plane) is not the bottleneck: 10,000 devices at one check-in per 60s is ~170 writes/s, comfortably inside a single Postgres with batched upserts and a partitioned status table.

Why devices fetch instead of build

comin converges each device by evaluating the overlay and rebuilding locally. At pilot scale that is fine; at 10,000 devices it means the same derivations are compiled 10,000 times on weak edge hardware, and a rollout’s wall-clock is the slowest device’s build.

At scale the pipeline builds once, centrally, per ring - build-before- promote: the delivery pipeline realises ring N’s closures on the build workers, pushes them to the signed binary cache, and only then flips the ring branch. Devices substitute (download) instead of compiling. Ring ordering naturally staggers cache load.

This is the largest gap between the pilot and the enterprise posture, and the first slice to build.

Why the gate is batched, and what the numbers say

The gate proves a change evaluates before it reaches git (see Safe writes). Forcing every host’s toplevel in a single nix process scales memory with the fleet - a whole-fleet evaluation OOM-killed the runner well before 100 hosts. The gate therefore evaluates in memory-bounded batches (chunkSize, default 50, deployed at 12): peak memory is the batch, not the fleet, and every affected host is still evaluated.

Batches are independent, which makes them the unit of horizontal scaling. Org-wide validation of 10,000 hosts at chunk size 12 and ~45s per chunk:

Measured on a synthetic 10,000-device fleet (4 hardware profiles x 3 device classes x 10 groups, two filtered org policies, 1% device-level overrides; classes_bench_test.go):

The conclusions fall out of the table:

• A scoped change stays interactive. Its blast radius is a handful of hosts; the gate evaluates only those (AffectedHosts). Metadata-only changes (groups, access, governance) skip the evaluation entirely - they cannot alter any device’s build.
• A genuinely org-wide change is validated asynchronously. It flows through the delivery pipeline, where chunk-parallel workers evaluate the full fleet before the first ring promotes. Nobody waits 10 hours at a save button.
• Interactive org-wide feedback uses equivalence classes. 10,000 devices produced by the same generator collapse to dozens of distinct config shapes (hardware profile x settings signature). An option or type error fails every host in its class, so evaluating one representative per class catches it in minutes. The full per-host evaluation still runs in the pipeline - sampling narrows feedback latency, never the guarantee. The class partitioner is security-critical code and is treated accordingly (tested exhaustively, reviewed as part of the gate).

Availability

The gate is fail-closed: if no gate worker is reachable, config writes are refused rather than committed unvalidated. Gate workers therefore live in the control plane’s availability domain (at least two at scale), not on hardware that may be powered off. Build workers may come and go - a missing build worker delays a rollout, never corrupts one.

Dogfooding the infra

Eval and build workers are themselves NixOS machines - so they are enrolled as Sextant devices in an infra group and managed declaratively by the product they serve. Scaling the build plane is enrolling another worker.

Status of this posture

1. Build-before-promote - SHIPPED: the pipeline builds a ring’s closures into the signed cache before the ring branch flips; devices substitute (see Ship an update).
2. Chunk-parallel gate - SHIPPED (--eval-workers); the pilot deployment runs 1 worker within its memory limit.
3. Equivalence-class sampling - SHIPPED for every unbounded validation (direct writes, change submit, merge revalidation).
4. Infra group - PLANNED: build/eval workers enrolled and managed by Sextant itself (scheduled with the hardware test round).

Decision records

Architecture decisions are recorded as ADRs in the repository under docs/adr/. They cover the ground-up rebuild in Go, the policy model that compiles to the scope chain, Postgres-only durable state, the SSR + API contract, the generated settings catalog, assurance and approvals, API and credential security, tenant isolation, language choice per workload, the update funnel’s ring branches, the remote gate-runner, cohort/canary pinning within a wave, and custom overlays managed from the console.

Read them in docs/adr/ in the source repository.