TroubleshootingCopy Anchor Link

Diagnostic procedures and solutions for common deployment, configuration, and runtime issues.

Controller diagnosticsCopy Anchor Link

Preflight checks with raft doctorCopy Anchor Link

Run raft doctor on your controller to inspect host health across configured locations:

raft doctor

raft doctor connects to each host over SSH and reports:

• Incus version running in project raft
• Active status of host systemd services (incus, raft-expire.timer, raft-network.service)
• Host root filesystem disk usage (df -h /)
• Status and usage of the raft-data Btrfs pool
• Count of existing saved containers

Listing active and stopped containersCopy Anchor Link

Inspect workspace states across locations:

raft list

# Filter by a single location
raft list --location lab

If a host is unreachable over SSH, the CLI reports a connection failure. An unreachable host does not mean containers are absent or deleted.

Host-side diagnosticsCopy Anchor Link

Connect to the host over SSH to inspect underlying services when commands fail.

Inspecting Incus project statusCopy Anchor Link

Verify container states and storage pools in project raft:

sudo incus --project raft list
sudo incus --project raft storage info raft-data

Checking host systemd unitsCopy Anchor Link

Raft installs two systemd units on the host:

• raft-network.service: Applies firewall and bridge isolation rules before Incus starts.
• raft-expire.timer / raft-expire.service: Runs the expiration worker every 15 seconds to stop expired workspaces.

Inspect unit logs with journalctl:

sudo systemctl status raft-network.service
sudo systemctl status raft-expire.timer
sudo journalctl -u raft-expire.service -n 50 --no-pager

Manually triggering garbage collectionCopy Anchor Link

The host timer stops expired containers every 15 seconds. Trigger an immediate expiration run from the controller:

raft gc

Expired containers transition from Running to Stopped. Expiration never deletes container files or snapshots.

Common error scenariosCopy Anchor Link

Configuration errorsCopy Anchor Link

Missing configuration file errorCopy Anchor Link

If ~/.config/raft/incus.json does not exist, the CLI raises FileNotFoundError. Create the configuration directory and file. Set mode 0600 on the file only.

"Configure at least one host in ~/.config/raft/incus.json"Copy Anchor Link

The configuration file contains valid JSON that is {} or is not an object (an empty file raises a JSON decoding error instead). Ensure the file defines at least one location with a non-empty ssh string.

"Build an image and configure its full immutable fingerprint before creating boxes"Copy Anchor Link

The image field is missing or is not a 64-character hexadecimal SHA-256 fingerprint. Supply the full fingerprint from sudo incus --project raft image list raft-dev --format json.

Admission and capacity errorsCopy Anchor Link

"Location already holds four boxes"Copy Anchor Link

Raft enforces a hard limit of four saved boxes per location. Both running and stopped containers count toward this limit. Destroy unused boxes with raft destroy <box> before creating, forking, or recovering a box.

Low disk warningsCopy Anchor Link

raft limits warns when free space on host disk or the raft-data pool falls below 5 GiB. This warning is diagnostic and does not halt container operations. Clean up dangling images, old snapshots, or unused containers to free space.

Network and forwarding issuesCopy Anchor Link

Port forward connects but returns connection refusedCopy Anchor Link

Verify that the guest service binds to 0.0.0.0 or the container's eth0 IP address. Services bound strictly to 127.0.0.1 inside the container cannot receive traffic routed through the bridge.

Workspaces cannot reach other workspaces on the same hostCopy Anchor Link

This is intended behavior. Raft configures nftables bridge filtering on rfbr0 to prevent communication between workspaces on the same bridge.

Host crash during recoveryCopy Anchor Link

If the controller crashes or SSH drops while raft recover is transferring an archive:

1. Check the Recovery handle: <location>:<name> output printed to stderr when raft recover started.
2. Inspect the reported handle with raft list --location <location> and confirm container status before removing staging.
3. Identify the exact staging file (/tmp/raft-recovery.XXXXXXXX) on the host by owner and timestamp.
4. After confirming container status, remove only that owned file:

   Copy Text

   sudo -n rm -f -- /tmp/raft-recovery.XXXXXXXX