Raft contract
This is the implementation reference for maintainers and advanced operators. For a first run, use host setup and your first box.
Raft manages persistent unprivileged native-architecture Incus system containers on configured ARM64 or AMD64 Linux hosts. Containers share the host kernel. Images must match host CPU architecture; foreign-architecture emulation and VM workflows are not implemented.
Executable location plugins
Users can register trusted controller executables as locations with raft plugin add LOCATION -- COMMAND .... plugin info inspects their versioned capability
handshake; plugin remove unregisters them without changing provider resources.
Plugins own their dependencies and provider guarantees. The dependency-free
Python CLI keeps Incus as its built-in backend.
Plugin entries use plugin argv and optional options, not ssh. Selected
commands negotiate protocol 1 before invoking a declared action with inherited
stdio and a JSON RAFT_PLUGIN_REQUEST environment value. Unsupported commands
fail without invoking the action; uncertain actions are never retried. Plugin box
IDs are opaque after the location separator. Explicit and configured new
defaults apply; omitted built-in Incus sizing does not. config reports plugin
host policies as null. Incus deployment tools select only SSH locations.
Incus-specific guarantees in the sections below do not apply to plugins. Plugins run with the controller user's privileges and inherited credentials; they are not sandboxed. See user plugins for the complete protocol, output contract and a sandbox-sdk adapter.
Image releases
- Versioned releases provide clean ARM64 and AMD64 templates with SHA-256 checksums, package inventories, and build provenance.
- Publication requires native lifecycle checks, recovery verification, and fresh AMD64 Ubuntu VM provisioning and reboot checks.
- Disposable AMD64 KVM verification uses a sparse 100 GiB disk and does not certify physical storage capacity or ARM64 host reboot behavior.
CLI distribution
- npm publishes
@microck/raftwith the same Python sources as the wheel and source archive. Its executable israft; Linux and Python 3.11.8+ are required. Installation runs no scripts and downloads no workspace images. raft --versionreports the release version without reading host configuration. The Python package, npm metadata and release tag must agree before publication.- Releases include Python wheels, source archives, npm tarballs and both native images with checksums. The release workflow verifies installation outside the checkout and publishes npm with provenance before creating the GitHub release.
- Self-update:
raft self-update [--check | --version X.Y.Z]takes the latest version from the GitHub releases API and compares versions numerically. It upgrades only npm global installs (through the registry, at the same prefix) and uv tools (a release wheel verified against the releaseSHA256SUMS), and refuses checkouts and other layouts. It never downgrades without--version, never changes hosts, and printsraft doctoras the next step.--checkexits100when a newer release exists. - Release notice: At most once per 24 hours a command checks the latest release in the background and, if it is newer, prints one stderr line after the command. It never installs, prompts, changes stdout or exit status, or waits more than 2 seconds; failures stay silent.
RAFT_NO_UPDATE_CHECK=1disables it, andself-updateand--versionskip it.
Infrastructure and deployment
- Strict infrastructure validation: Deployment refuses conflicting storage or network names and incomplete infrastructure rather than overwriting or repairing state.
- Storage pool sizing: The shared Btrfs pool
raft-datamust retain its configured 60 GiB size. The own-disk poolraft-disksis opt-in: deployment creates it only when--disk-pool SIZEis given, on a fresh host or on any later redeploy, as a loop-backed LVM thin pool (lvm.use_thinpool=true) of SIZE whose file is allocated in full on the host disk and whose thin pool uses--discards nopassdown. A fresh host needs 80 GiB free, plus SIZE when--disk-poolis given; adding the pool later needs SIZE plus 20 GiB free. A host withoutraft-disksis healthy. An existingraft-diskskeeps its size, a different explicit--disk-poolis refused, and a pool with another driver or thick volumes is rejected. Deployment never resizes either pool. - Firewall precedence: Redeployment reapplies firewall rules. Incus startup requires successful firewall application before the daemon starts, including socket activation.
- Guided onboarding:
raft host add LOCATION --ssh TARGETprobes the target read-only and refuses unsupported hosts, including macOS, before any change. It prints the exact plan and applies it only after an interactive yes or--yes;--dry-runstops after the plan. It runs the same deployer asdeploy/deploy.py, imports an image only after its SHA-256 matches (releaseSHA256SUMSor the local archive digest) and its architecture matches the host, then registers the location with the image fingerprint. It finishes withdoctorand a disposable create/exec/destroy smoke test; if either fails, it removes the new location from the configuration and leaves host changes for a rerun. It uses existing key-based SSH and never collects passwords or changes SSH, other firewall rules or IP configuration. It requires the deployment files from the npm package or a source checkout. - CLI coverage: The single-operator workflow spans lifecycle, execution, files, snapshots, jobs, tunnels, backup, and advisory capacity inspection; see the command reference.
- Capability discovery: Installed
--helpand nested command help describe the current commands, arguments and operational constraints without contacting hosts.config --location NAMEanddoctor --location NAMEinspect only that configured host; omitting the flag preserves inspection of all hosts. A location filter onconfig set/unsetis rejected because defaults belong to the controller. - Read latency: Listing collects policy and inventory through one SSH connection per host. Capacity collects resources and inventory through one connection. Doctor performs its sequential checks through one connection, retaining the worker compatibility lock. These are read-only observations, not locked snapshots; lifecycle validation remains inside its existing transactions.
Configuration and lifecycle defaults
| Setting | Behavior |
|---|---|
| Host identification | Configuration requires named SSH targets using letters, digits, underscores, or hyphens. |
| Qualified handles | Every handle includes location and instance name (<location>:<name>). |
| Image pinning | Workspace creation requires the immutable 64-hex image fingerprint, not an updating alias. |
| Lifecycle defaults | Built-in defaults are the first configured location, 1 CPU, 2 GiB RAM, the shared pool, a persistent box and 600-second lifetime. ~/.config/raft/config.json can override them for new only; explicit flags win, and --persistent overrides a disposable default. Boxes from a snapshot (new --from) and forks keep the source's CPU, RAM and disk instead of any default, configured or built-in; each flag overrides only its own value. --cpu takes any positive whole number, --memory any positive whole GiB (16GiB), --disk a positive whole GiB or shared, and TTL 60 to 2,592,000 seconds (30 days). |
| Box count and size | Raft limits neither how many boxes a host holds nor their size. CPU or RAM limits larger than the whole host (CPU count, or MemTotal rounded up to whole GiB) are refused under the host lock before any change, because Incus would clip them silently. Other failures come from Incus, such as too little pool space. |
| TTL calculation | Running TTL begins after launch completes, using the host clock. Cold image preparation does not consume TTL. |
| Host locking | Creation, lifecycle transitions, and expiry are serialized by host file lock (/run/lock/raft-incus.lock). Noninteractive transactions and metadata lookups close controller stdin so commands cannot wait for piped input while holding this lock or consume payload input. Synchronous raft exec and interactive SSH forward controller stdin; file transfers use their own streams. Native Incus admin calls bypass CLI policy. |
Locations, notes and capacity inspection
raft locations lists configured locations as JSONL with location, default and note fields without contacting hosts. raft note LOCATION TEXT and raft note LOCATION --clear edit the optional note field in the controller configuration atomically, preserving location order and mode 0600. Notes are advisory guidance; Raft shows them in locations, capacity and on stderr from new, and never enforces them.
Boxes accept an optional note with a fixed template: purpose (required), owner and next, all nonempty strings. raft new --purpose TEXT [--owner TEXT] [--next TEXT] and raft note BOX ... store it as JSON in the box's user.raft.note Incus key; raft note BOX --clear removes it. Each edit replaces the whole note. raft list reports it as note, or null, reports each box's disk as "shared" or its own disk size, and reports deleteAt, the Unix time retention deletes a stopped box, or "". new always writes the key, so new --from never inherits a snapshot's note. Forks and recovered backups keep it unless fork or recover gets note flags, which replace the whole note. Box notes never change lifecycle behavior.
Controller defaults and host policies
- Controller defaults:
~/.config/raft/config.jsonholds{"defaults": {...}}with optionallocation,ttl,cpu,memory,diskanddisposablefornew. Precedence is explicit flag, then configured default, then built-in default.cpu,memoryanddiskapply only to boxes from the image:new --fromkeeps the snapshot's location, CPU, RAM and disk unless a flag overrides them, and--disk sharedis refused there.ttlanddisposableapply tonew --fromas well, because a snapshot carries no lifecycle;new --disposableand--persistentare mutually exclusive overrides.forktakes no defaults and is persistent without--disposable. Adiskdefault on a host withoutraft-disks, or larger than its unreserved space, fails like--diskand namesdefaults.diskas the source. Other commands take no defaults. A value is valid exactly when the same text is valid for the flag. Unknown keys, bad values and malformed JSON failnewwith the file path and key, before any host access.raft config set/unsetvalidate first, write atomically under a controller lock, keep mode0600, print the change and can repair the key they edit. The file is separate fromincus.json, so location names never collide with setting names. - Policy storage:
raft policy LOCATION [--require FIELDS|none] [--delete-stopped-after HOURS|never]sets either host policy or both; each option changes only its own. Required box note fields, which must includepurpose, live in the Incus project keyuser.raft.require, and--require noneempties it. Stopped-box retention lives inuser.raft.delete-stopped-afteras a positive whole number of hours, with the time it was set inuser.raft.delete-stopped-after.since;neverempties both. Both values are validated before any host change. Each change runs under the host lock, prints the old and new value, records UTC time, the SSH account (SUDO_USER) and the self-reported controller user and hostname in the policy's.changedkey, and logs to the host journal with tagraft. Redeployment leaves these keys unchanged. - Note policy enforcement:
new(including--from),fork,recoverandraft note BOXcheck the note the box will carry. The controller reads the policy first and fails with the missing flags and a complete corrected command before changing anything. The host repeats the check inside the transaction that writes the note, under the host lock, so a concurrent policy change cannot be missed. That transaction also holds the oversize and own-disk reservation checks; all of them run before Incus creates anything.forkchecks the given note or, without one, the source note read under the lock.recoverchecks a given note before upload; an archived note is checked after import, and a failing import is deleted in the same transaction.note --clearand replacement notes that drop a required field are rejected. - Retention enforcement: The host expiry worker deletes a stopped Raft box, with its snapshots, once the hours have passed since the later of its
user.raft.stopped-atand the policy's.sincetime, so setting, shortening or renewing a policy never deletes a box sooner than its hours after the change. It runs on the 15-second timer and onraft gc, under the host lock, and writes each deletion to the host journal with tagraft. Running boxes are never touched. A stopped Raft box withoutuser.raft.stopped-at, from a guest poweroff, a native stop, a failed start orraft recover, gets the worker's current time as its stop time, never an earlier guess, whether or not a policy is set. A malformed retention value makes the worker delete nothing by retention, still enforce deadlines, and exit nonzero; the CLI reports it with the fix. Setting retention requires the current worker protocol; clearing it does not. - Policy scope: Policies are guardrails for agents, not access controls. Anyone with host SSH and sudo can change them or create boxes with native Incus commands, and controllers without note policy support do not check it. A native
incus startleaves a stale stop time that retention trusts.raft configreports every location's policies and last changes.
raft capacity [--location NAME] [--json] inspects hosts without modifying state. Text and JSON (notice) lead with: "Advisory estimate only. Raft enforces no box count or size budget; create the boxes your task needs." It reports effective CPUs, total and available RAM, saved and active box counts, active resource limits, headroom, free space in the host disk and the shared pool, and for raft-disks its data space, reserved and used bytes and metadata use (boxDiskPool, null when own box disks are not enabled).
- Reserve calculations: Estimates leave out 1 CPU and the larger of 2 GiB or 10% host RAM.
- Example shapes:
estimateslists shapes doubling from 1 CPU / 2 GiB at 2 GiB per CPU while they fit an empty host, withfitEmptyHostandfitNowcounts. No count is capped. - Headroom: Subtracts active configured limits and keeps available RAM above the reserve. Every non-
Stoppedstate (includingFrozen) counts as active. - Advisory only: Nothing in the report blocks creation, resume or fork. Disk space below 5 GiB and
raft-disksthin pool metadata above 80% trigger warnings; a missingraft-disksdoes not.
Container lifecycle and execution
- Stop and resume: For a persistent box, stop terminates guest processes and preserves disk contents; stopped boxes retain disk but do not reserve guest RAM. Resume starts the container anew with an explicit TTL without restoring process memory.
- Lifetime extension:
raft extend --ttlresets a running box's deadline from current host time without restarting; stopped boxes requireraft resume. - Host worker contract:
doctor, disposable creation/fork, prune and setting a retention policy require the current worker protocol (2 since 0.6.0, which added retention).doctorchecks it without running expiry or changing boxes, and reports the failing location. A mismatch fails before executing deletion policy or creating a disposable box; deploy the current checkout explicitly. Installing a new CLI does not update host workers. - Disposable lifecycle:
new --disposable, or a configureddisposabledefault without--persistent, opts a box into deletion on stop or TTL expiry, including its snapshots. Clones and recovered boxes default to persistent whatever the default; fork accepts an explicit--disposable. - Manual cleanup:
prune --older-than DAYSpreviews persistent stopped boxes with a recorded stop timestamp.--yesdeletes matching boxes under the host lifecycle lock. Running boxes and boxes without a recorded stop time are excluded. Automatic deletion of persistent stopped boxes happens only under a host retention policy. - Resizing: Resume and fork accept optional CPU/RAM overrides, preserving omitted values. Resume validates stopped state under the lock before changing sizing. Neither changes a box's disk.
- Expiration cleanup: The host timer checks expirations every 15 seconds, stopping expired persistent containers, deleting expired disposable containers and applying stopped-box retention. Expiry may be delayed by graceful shutdown, long locked transactions, and host outage until startup.
- Destroy: Removes the container instance and its snapshots, releasing storage. Destroy confirms absence; SSH failure is not proof of deletion. Disposable boxes created with
new --disposableare deleted on explicit stop or expiry. Persistent boxes still require explicit destroy, prune or a retention policy for deletion. - Execution transport: Incus exec and file APIs travel over authenticated host SSH without a public Incus listener or injected credentials. Interactive SSH runs a PTY over host SSH and Incus exec.
- State inspection and tunnels:
raft listandraft inforead native metadata without guest process counters during shutdown; tunnels use the managed NIC. - Resource reporting:
raft usagereports running cgroup memory, CPU microseconds, CPU affinity, and filesystem space withfilesystemScope: the shared pool, orown SIZE box disk. It provides no billing. - Recursive transfers:
upload/download --recursiverequire a running workspace and Python 3.11.8+ in the guest. They stream a tar archive and publish a new destination directory atomically without overwriting or merging. Relative links contained within the tree, regular files, directories and internal hardlinks are supported; escaping paths/links and special files are rejected. Source entries are pinned with no-follow descriptors before metadata inspection, file reads or directory descent, so pathname replacement cannot redirect access outside the selected tree. Failed transfers leave existing destinations unchanged. - Directory transfer completion: The receiver verifies the sender's exact archive byte count before publication. A truncated stream, including one ending between complete entries, is rejected without publishing a partial tree.
- File transfer failures: Controller write errors return an error before replacing a download destination or publishing a backup archive.
Source sync
sync BOX SOURCE DESTINATION incrementally copies source into a running guest.
It requires rsync on the controller and guest. By default, Git selects tracked
and unignored files and excludes .git; submodules are rejected. --all opts
into the entire source tree, including ignored and hidden files. Git filtering
is not a secret scanner. Missing tracked files are omitted.
Matching files are replaced, but guest-only files are never deleted. Checksums
find changed content even when size and timestamps match. Relative symlinks
that stay within the source tree are copied; rsync skips unsafe links and
special files and reports them. Destination must be an absolute non-root path;
its parent must exist. --dry-run reports changes without writing guest files.
Files are replaced individually through rsync temporary files, not as an atomic
tree. An interrupted sync can leave some files updated; rerunning converges.
No host filesystem path is exposed: rsync runs through Incus inside the guest.
Security and network isolation
- User namespaces: Guest root maps to an unprivileged host UID. Container nesting and intercepted syscalls support Docker without mounting the host Docker socket.
- Network filtering: Dedicated bridge
rfbr0allows DNS, DHCP, and outbound internet. Interface-scoped firewall rules block guest access to host services, cloud metadata, and private address ranges. - Bridge peer isolation: A native nftables bridge rule blocks all inter-workspace frames on
rfbr0. IPv6 and link-local peer traffic are blocked. - Private tunnels: Forwarding listeners bind strictly to controller loopback (
127.0.0.1). Guest services must listen on0.0.0.0or their managed IP, not guest loopback. - Image hygiene: Development images omit coding agents, personal keys, and host credentials. SSH host keys are generated fresh on first boot. Release inspection rejects root and non-root SSH content, guest host keys, and credential-path links or nonregular entries.
Storage, snapshots, and cloning
- Shared pool: Boxes default to a dedicated 60 GiB Btrfs pool per host with instant snapshots and clones. Btrfs quotas cannot bound a box there, because nested subvolumes (Docker's, for example) escape them. The native compressed image cache resides outside both pools, so host disk space still matters.
- Own box disks:
new --disk SIZEputs the box's root on an LVM thin volume of SIZE inraft-disks, formatted ext4. All guest writes, including Docker data, land in that filesystem, so writes fail with ENOSPC at SIZE. New own-disk boxes are thin snapshots of the cached image volume, and forks, templates and snapshots of own-disk boxes are thin snapshots of their source. The pool is never overcommitted: under the host lock, beforenew,new --from,fork,snapshotorrecoveradds thin volumes toraft-disks, Raft sums the sizes of every thin volume in the pool (fromlvs) plus the new ones and refuses, before creating anything, if the total exceeds the thin pool's data space. A new volume counts as the larger of its size and its source's; the first own-disk box on a host also counts the image volume (volume.size, or 10 GiB); a recovered backup counts its size once per snapshot plus once.restorereserves nothing. On a host withoutraft-disks,new --diskand recovery of an own-disk backup fail before creating anything and name the deploy command that enables the pool.forkandnew --fromkeep the source disk;new --from ... --disk SIZEcopies onto an own disk of SIZE. A disk too small for its contents fails creation. - Stopped-state requirement: Snapshots, restores and forks require a stopped source for consistent disk contents. Operations run under the host lifecycle lock.
- Snapshot restore:
raft restorerolls back disk blocks via--diskonly, retaining container configuration, limits, and MAC address. - Snapshot management:
snapshot-delete <box> <name>removes one named snapshot under the lock.new --from LOCATION:BOX/SNAPSHOTcreates a same-host box from that snapshot, with the snapshot's CPU, RAM and disk unless--cpu,--memoryor--diskoverrides them, and a fresh network identity. The source may run; the named snapshot is immutable. Templates retain filesystem secrets. - Workspace forks:
raft forkcopies filesystem data and secrets to a new container with an independent Incus name, fresh MAC address, and renewed TTL. Source snapshots are not inherited.
Background jobs and desktop
- Detached jobs: Launching requires a running guest and rejects unavailable guests before allocating a record. Commands run as guest systemd units. A transient host supervisor captures the first 1 MiB of combined stdout/stderr in a root-private host store and records completion.
statusreturns JSON with state, exitCode, timestamps and droppedBytes;logsreturns the retained bytes. A confirmed normal exit records its status; signal termination records 128 plus the signal number. Completed results survive workspace stop, deletion and host reboot. Running jobs do not survive guest stop or host reboot; a stopping guest or missing supervisor is reported as interrupted, even if a shutdown handler returns zero. Cancel stops the guest unit's control group and records cancellation even if its host supervisor has already exited. Cancellation waits at most 120 seconds; a timeout preserves the cancellation request and requires inspection or retry. Unknown IDs and mismatched workspace handles are rejected without starting the guest. - Job supervision bounds: Host submission is asynchronous and waits at most 10 seconds. A submission timeout preserves the record for status inspection because the supervisor may already be queued. Post-run guest inspection waits at most 60 seconds; an unconfirmed result is interrupted. Host reconciliation waits at most 10 seconds and reports an error without changing job states if systemd does not respond. Failed pre-submission setup removes its incomplete record.
- Job retention:
/var/lib/raft-jobsholds at most 512 records, with one 1 MiB log per record. Results expire seven days after completion. Commands prune expired records when the store is accessed; an idle host may keep expired files until that next access. Full stores reject new jobs rather than evicting unexpired results. Completed records and captured logs are flushed to disk before publication. Unreadable records remain on the host for inspection and count toward the record limit, but do not block access to other jobs. Logs can contain secrets and stay root-private. No historical guest-journal fallback is provided; jobs created by an older controller must be inspected with that version before upgrading. - Desktop service: On-demand Xvfb and Openbox on display
:99without a login manager.raft desktopstarts Voxl by default on guest port 6080;--backend novncstarts noVNC on guest port 6081. Both share the same desktop and may run together. The controller tunnel defaults to local port 6080 for either backend and prints the viewer URL (/for Voxl,/vnc.htmlfor noVNC). No automatic fallback or existing-image migration occurs. Voxl 0.3.2 is built from a checksum-pinned source crate with its locked dependencies. It runs asdeveloper, has its file root at/workspace, disables audio/microphone and password login, and retains host/origin checks. SSH authenticates private tunnel access; neither backend port is public ingress. Voxl needs H.264 WebCodecs in a secure browser context such as localhost. Closing a tunnel leaves services running; stoppingraft-desktopstops both streaming backends. Desktop provides full workspace access, not browser confinement. - Voxl distribution: The x264-linked executable is distributed under GPL-2.0; Voxl's own source retains its MIT license. Images include GPL/MIT texts, a notice, and
/opt/raft/voxl-source.tar.gzwith vendored locked Rust sources, matching Debian native-library sources and patches, and build instructions. Native release assets include the extracted source bundle and notices, with checksums. Publication rejects images missing these materials.
Backup and recovery
- Export safeguards:
raft backupexports stopped containers and snapshots to a local archive. Destination path must not exist, requires mode0600, and uses atomic hard-link publication. - Host locking: Export holds the host lifecycle lock, temporarily delaying expiry checks on the source host.
- Recovery:
raft recoverimports archives into a stopped container with a fresh name, new MAC address, and cleared TTL. It reads the archive's recorded pool frombackup/index.yaml: own-disk backups return toraft-diskswith their size after the reservation check, failing on a host without the pool, and other archives land inraft-data. The host note policy applies after the reservation check. Recovery never overwrites existing boxes. - Archive security: Archives contain full container filesystems, including credentials. Native archive configuration is not sanitized; backups do not imply scheduled off-host uploads. Store archives privately and import only trusted exports.
Disposable foreground CI
This section specifies the disposable CI behavior introduced in 0.7.0.
run creates a uniquely marked disposable box and establishes its fixed host
expiry before startup. It streams one foreground command without retrying it,
optionally transfers a Git-filtered or explicitly complete source tree, collects
result directories even after payload failure, then checks ownership and deletes
under the lifecycle lock. The controller handles timeout and cancellation with
bounded collection/cleanup attempts and reports unconfirmed cleanup as failure.
Unacknowledged creation is cancelled under the same lock with a private host
record; a queued creation checks it before allocation. These root-only records
remain in /run/raft-ci-cancelled until reboot so late SSH requests cannot outlive
their cancellation. Ordinary acknowledged runs leave no record.
Host expiry is a backstop dependent on the worker and lock, not a strict deletion
guarantee after transport failure. Existing nonzero payload status is preserved.
This is not a registered runner, a scoped CI identity or VM-grade isolation.
See Disposable CI jobs for the full contract.
Structured output and completion
raft --json COMMAND ... writes only JSON Lines events to stdout. Native
structured reports use {"type":"data","value":...}. Other output uses
stdout or stderr events with encoding: "base64" and a data field so
arbitrary bytes survive unchanged. Stream chunk boundaries are not lines.
Expected failures add an error event with code (usage or operation) and
message. The final exit event contains code, matching the process status,
including payload failures. Local subprocess signal termination becomes
128 + signal in both the exit event and process status. Usage errors return 2;
top-level operation errors return 1. run also emits operation errors for setup,
artifact collection and cleanup failures, while retaining its existing payload,
timeout and cleanup exit-status precedence.
Help and version also end with an exit event. Release notices are disabled in
JSON mode. --json belongs before the command; arguments after a payload's
-- belong to that payload. Existing command-local JSON flags keep their
existing output when global JSON mode is absent.
Interactive SSH and foreground tunnels/desktop reject JSON mode before any side effect because these commands require terminal or session ownership. Other commands retain their existing behavior. JSON events are flushed as output arrives; memory use does not grow with payload output.
raft completions bash|zsh|fish prints a shell integration script without
reading host configuration. It asks the installed parser for command, option
and fixed-choice candidates when completing, so added commands and options
need no second maintained catalog. It does not contact hosts or suggest
workspace IDs or secret values.