raft / docs
Guides

Fix a failed operation

Start with read-only checks on the controller. Replace lab with the failing location; do not repeat a mutating command until you know whether it completed remotely.

Controller diagnostics

raft doctor --location lab
raft list --location lab
raft capacity --location lab

doctor checks Incus, expiry-worker compatibility, host services, filesystem usage, storage pools and saved-box count. It does not validate your image pin or firewall rules. A host without own box disks is healthy; capacity supplies low-space warnings. An unreachable host does not mean its boxes are absent.

Symptom or errorWhat to do
Cannot reach the SSH targetVerify the host key interactively, then run ssh -o BatchMode=yes ubuntu@192.0.2.10 'sudo -n true' with your host target. Fix keys, routing or passwordless sudo.
Missing ~/.config/raft/incus.json, or "Configure at least one host"Onboard a host, or repair the inventory with a location and nonempty ssh value. An empty file is invalid JSON.
"Unknown location" or an invalid defaultCheck raft locations and controller configuration. Repair defaults.location with raft config set location lab or unset it.
"Build an image and configure its full immutable fingerprint"Set the imported development image's full 64-character lowercase hexadecimal fingerprint, not an alias or base-image hash.
"Host expiry worker differs from this CLI"Redeploy code matching your CLI. An active timer alone does not prove worker compatibility. Do not bypass the check.
Missing required --purpose, --owner or --nextInspect raft config --location lab and supply the required note fields. Note flags replace the whole note.
"Stop the source box before resume, snapshot, restore, fork or backup"Inspect its state. Stop a persistent box when interruption is acceptable. Do not stop a disposable source; that deletes it.

Fix sizing or disk failures

Symptom or errorWhat to do
"Requested N CPUs, but this host has M" or oversized RAMUse a size no larger than the host total shown by capacity. CPU counts are whole positive numbers; RAM and disk use whole GiB.
"has no raft-disks pool"Own disks are opt-in. Use --disk shared for image-based creation, unset a disk default, or enable the pool.
"own box disks would need ... more of raft-disks"Check unreserved capacity. Every own-disk box, snapshot and cached image reserves space. Choose a smaller disk, use shared storage, or back up and delete only data you no longer need.
"No space left on device" in an own-disk boxRemove unneeded guest files or Docker data. For a larger persistent copy, stop it, snapshot it and use new --from with --disk SIZE; see disk copies.
Low shared-pool or host disk warningcapacity warns below 5 GiB. Review images, snapshots and boxes before deleting anything. Shared-pool growth can affect every box.
Thin pool metadata warningAt 80% or more, inspect the host with sudo lvs before creating more own-disk boxes. Full metadata can stop all own-disk boxes and require offline repair.

A configured defaults.disk can cause creation failures even without --disk; the error names it. --disk shared is not accepted with --from.

Diagnose an unavailable service

For forwarding, bind the guest service to 0.0.0.0 or its managed address, not guest loopback. Check its logs and choose a different local port if occupied. For Voxl, use the printed localhost URL and a browser with H.264 WebCodecs support. Select --backend novnc explicitly if needed, then include /vnc.html in the viewer URL. Missing backend services require a fresh box from an updated image, not a CLI-only update. See private services.

Boxes cannot contact peers on the same bridge by design. An open tunnel, SSH session or detached job does not extend TTL. For a stopped box use resume, not extend.

Find an expired or deleted box

Check stopAt and deleteAt in raft list, and retention in raft config --location lab. Persistent expiry stops processes and keeps disk; disposable expiry deletes the box. Retention later deletes stopped boxes and snapshots. Deleted data requires an existing off-host backup.

On the host, inspect without changing services:

HOST
sudo systemctl status raft-network.service raft-expire.timer --no-pager
sudo journalctl -u raft-expire.service -n 50 --no-pager
sudo journalctl -t raft
sudo incus --project raft list
sudo incus --project raft storage info raft-data

Host journals may contain guest arguments and secrets. Outages or a long backup can delay the 15-second expiry checks. raft gc --location lab enforces due expiry and retention immediately, which can delete other due boxes; inspect before invoking it.

Host crash during recovery

If SSH drops or recovery is interrupted:

  1. Find the Recovery handle: LOCATION:BOX printed to stderr.
  2. Inspect that exact handle with raft info and the target inventory. The import may already exist even without a successful response.
  3. If a host staging file remains, identify the exact /tmp/raft-recovery.XXXXXXXX by owner and timestamp and confirm no transfer still uses it.
  4. Remove only that verified owned file. Never delete /tmp/raft-recovery.* by wildcard.

If creation or execution loses its connection, inspect its effects before retrying too. See scripting failures and backup recovery.

On this page