Fix a failed operation
Start with read-only checks on the controller. Replace lab with the failing location; do not repeat a mutating command until you know whether it completed remotely.
Controller diagnostics
raft doctor --location lab
raft list --location lab
raft capacity --location labdoctor checks Incus, expiry-worker compatibility, host services, filesystem usage, storage pools and saved-box count. It does not validate your image pin or firewall rules. A host without own box disks is healthy; capacity supplies low-space warnings. An unreachable host does not mean its boxes are absent.
| Symptom or error | What to do |
|---|---|
| Cannot reach the SSH target | Verify the host key interactively, then run ssh -o BatchMode=yes ubuntu@192.0.2.10 'sudo -n true' with your host target. Fix keys, routing or passwordless sudo. |
Missing ~/.config/raft/incus.json, or "Configure at least one host" | Onboard a host, or repair the inventory with a location and nonempty ssh value. An empty file is invalid JSON. |
| "Unknown location" or an invalid default | Check raft locations and controller configuration. Repair defaults.location with raft config set location lab or unset it. |
| "Build an image and configure its full immutable fingerprint" | Set the imported development image's full 64-character lowercase hexadecimal fingerprint, not an alias or base-image hash. |
| "Host expiry worker differs from this CLI" | Redeploy code matching your CLI. An active timer alone does not prove worker compatibility. Do not bypass the check. |
Missing required --purpose, --owner or --next | Inspect raft config --location lab and supply the required note fields. Note flags replace the whole note. |
| "Stop the source box before resume, snapshot, restore, fork or backup" | Inspect its state. Stop a persistent box when interruption is acceptable. Do not stop a disposable source; that deletes it. |
Fix sizing or disk failures
| Symptom or error | What to do |
|---|---|
| "Requested N CPUs, but this host has M" or oversized RAM | Use a size no larger than the host total shown by capacity. CPU counts are whole positive numbers; RAM and disk use whole GiB. |
| "has no raft-disks pool" | Own disks are opt-in. Use --disk shared for image-based creation, unset a disk default, or enable the pool. |
| "own box disks would need ... more of raft-disks" | Check unreserved capacity. Every own-disk box, snapshot and cached image reserves space. Choose a smaller disk, use shared storage, or back up and delete only data you no longer need. |
| "No space left on device" in an own-disk box | Remove unneeded guest files or Docker data. For a larger persistent copy, stop it, snapshot it and use new --from with --disk SIZE; see disk copies. |
| Low shared-pool or host disk warning | capacity warns below 5 GiB. Review images, snapshots and boxes before deleting anything. Shared-pool growth can affect every box. |
| Thin pool metadata warning | At 80% or more, inspect the host with sudo lvs before creating more own-disk boxes. Full metadata can stop all own-disk boxes and require offline repair. |
A configured defaults.disk can cause creation failures even without --disk; the error names it. --disk shared is not accepted with --from.
Diagnose an unavailable service
For forwarding, bind the guest service to 0.0.0.0 or its managed address, not guest loopback. Check its logs and choose a different local port if occupied. For Voxl, use the printed localhost URL and a browser with H.264 WebCodecs support. Select --backend novnc explicitly if needed, then include /vnc.html in the viewer URL. Missing backend services require a fresh box from an updated image, not a CLI-only update. See private services.
Boxes cannot contact peers on the same bridge by design. An open tunnel, SSH session or detached job does not extend TTL. For a stopped box use resume, not extend.
Find an expired or deleted box
Check stopAt and deleteAt in raft list, and retention in raft config --location lab. Persistent expiry stops processes and keeps disk; disposable expiry deletes the box. Retention later deletes stopped boxes and snapshots. Deleted data requires an existing off-host backup.
On the host, inspect without changing services:
sudo systemctl status raft-network.service raft-expire.timer --no-pager
sudo journalctl -u raft-expire.service -n 50 --no-pager
sudo journalctl -t raft
sudo incus --project raft list
sudo incus --project raft storage info raft-dataHost journals may contain guest arguments and secrets. Outages or a long backup can delay the 15-second expiry checks. raft gc --location lab enforces due expiry and retention immediately, which can delete other due boxes; inspect before invoking it.
Host crash during recovery
If SSH drops or recovery is interrupted:
- Find the
Recovery handle: LOCATION:BOXprinted to stderr. - Inspect that exact handle with
raft infoand the target inventory. The import may already exist even without a successful response. - If a host staging file remains, identify the exact
/tmp/raft-recovery.XXXXXXXXby owner and timestamp and confirm no transfer still uses it. - Remove only that verified owned file. Never delete
/tmp/raft-recovery.*by wildcard.
If creation or execution loses its connection, inspect its effects before retrying too. See scripting failures and backup recovery.