Verification
Raft is verified using real disposable Incus workspaces, guest processes, files, Docker builds, and network connections. Tests contain no mocked services and verify Raft behavior rather than conformance with the hosted Boat service.
Verification boundaries
Tests do not certify arbitrary long-running workloads, host loss, multi-tenant security boundaries, or exact Boat image/version parity. ARM64 host reboot and physical storage capacity are not certified by the AMD64 VM test. Containers share the host kernel and do not offer VM isolation.
Storage accounting covers shared pools rather than per-box quotas. Large backup transfers hold the host lifecycle lock, which delays expiration checks.
Coverage matrix
| Implemented workflow | E2E assertion |
|---|---|
| Create/list/info/delete | Qualified handle, native state, ownership and cleanup |
| Root terminal/exec | Real PTY, workspace directory, literal arguments and exit code |
| Resource sizing | Effective CPU affinity and cgroup memory limit |
| Host capacity | Mixed-size budgets, stopped/frozen boxes, low RAM/disk and four saved slots; live create/stop/resume/destroy changes in CPU/memory allocations |
| Background jobs | Journal output, nonzero exit status, cancellation of child processes, inspection after cancellation or log rotation, and rejection of unknown job IDs |
| TTL/extend/resume | Host-enforced stop, manual gc, retained files, uninterrupted lifetime extension and 30-day resume |
| Usage | Real cgroup counters and explicitly shared-pool disk scope |
| File transfer | Binary content, paths containing spaces, stopped-workspace transfers, preservation of the destination after a failed download, and real controller write-error rejection |
| Docker | Real image build and container execution |
| Snapshot/restore/fork | File rollback, independent copy, retained network identity and deterministic concurrent-resume rejection under the real host lock |
| Backup/recovery | Stopped source, no overwrite, snapshots, content, permissions and fresh MAC |
| Private forwarding | Guest HTTP response through controller loopback tunnel |
| Desktop | WebSocket/RFB negotiation, framebuffer, pointer, keyboard, clipboard and reconnect |
| Network separation | Positive service control, negative IPv4/IPv6 peer, host and metadata probes |
| Development image | Real developer-user tool checks, immutable selection and credential-path archive rejection |
| Deployment/firewall | Repeat deployment removes stale rules without restarting Incus; failed filtering blocks normal and socket startup; boot timestamps prove filtering precedes Incus |
| Storage contract | Disposable oversized pool is rejected without automatic resizing |
| CLI options and errors | Help for all 26 commands; default location, TTL and desktop port; all six sizing profiles; global and per-location reports; invalid TTLs, sizes, required arguments, handles, paths, snapshot names, jobs and ports |
Command and option audit (2026-10-06)
Coverage, regressions and remaining limits
The audit checked help for all 26 commands and expanded the live suites to cover defaults, invalid arguments, manual garbage collection, maximum lifetimes and unknown job IDs. Two ARM64 hosts passed extended lifecycle tests, all six sizing profiles and all 31 development-tool checks.
After the fixes, live lifecycle and same-host recovery passed again. Cross-host recovery passed with a 384,827,472-byte archive. Test boxes and archives were removed without rebooting or reconfiguring the existing hosts.
The audit found and fixed two CLI bugs:
- Unknown background jobs appeared to succeed. Job inspection now distinguishes unknown IDs, loaded jobs and canceled jobs with retained journal entries. Tests cover inspection after cancellation and journal rotation.
- SSH could report success after a controller write failed. Downloads and backups now copy the stream through Python so local write errors reach the caller. Real
/dev/fulltests verify both receiving paths. The same Incus/SSH producer falsely succeeded with the previous implementation and raisedENOSPCwith the fix.
This covers implemented workflows and selected option combinations, not every possible workload or environment. A full-image export exceeded the local controller's free disk space and was interrupted. Full-sized archive validation runs on disposable CI hosts with sufficient scratch space.
The final runtime validation tests commit ac618e64. ARM64 and AMD64 both passed extended lifecycle tests with daemon restarts, capacity transitions and full development-image backup/recovery. Packaging checks passed on Python 3.11 and 3.14, and documentation checks passed.
Running verification suites
Test commands require configured hosts and immutable image fingerprints. Suites destroy only the fixtures they create.
Allocation and archive inspection:
python3 deploy/verify-limits.py: Checks allocation policies without a host. Add--location labto verify live create, stop, resume, and destroy reporting.python3 deploy/verify-image-tests.py: Inspects archive structures, including symlinks, hardlinks, FIFOs, and credential detection.
Live host suites (expand the check you need):
Lifecycle
python3 deploy/verify-live.py --location lab --extendedDaemon restart on a dedicated test host
python3 deploy/verify-live.py --location lab --restart-incusCross-host recovery
python3 deploy/verify-controls.py --source lab --target second-host --directory /path/to/archive-storageSame-host recovery
python3 deploy/verify-controls.py --source lab --target lab --directory /path/to/archive-storageFull development-image recovery
python3 deploy/verify-controls.py --source lab --target second-host --directory /path/to/archive-storage --development-imageverify-tools.py <qualified-box> executes 31 tool checks as the unprivileged developer user inside a running workspace. It compiles and runs code across installed runtimes and package managers without external agent authentication.
Extended lifecycle results (2026-10-05)
Test evidence and run details
The extended lifecycle suite and strengthened tool checks passed on two ARM64 Ubuntu hosts with Incus 6.0.6 on 2026-10-05.
Both hosts executed all six sizing combinations, all 31 developer tool checks, Docker builds, file transfers, stopped snapshots, forks, restore, jobs, and private HTTP and desktop tunnels.
Desktop testing validated keyboard input, exact clipboard synchronization, pointer coordinates, framebuffer updates, and secondary client connections. Headed Chromium testing passed with sandboxing enabled, confirming canvas rendering at 1280x720 over the noVNC interface.
Admission and backup runs
The four-saved-box admission test passed, including rejection of a fifth create, fork, or recovery attempt. Full development-image recovery succeeded with a 5,254,224,965-byte archive, verifying inventories, tool checks, permissions, fresh MACs, and snapshot rollback.
The execution wrapper reported exit 143 after the success message; independent native inventory and archive checks confirmed that all fixtures and temporary archives were removed.
Debian fixture backup and recovery passed across the following tests:
- Same-host recovery: 385,278,177-byte archive with snapshot rollback and distinct MAC/IP identities.
- Reverse cross-host recovery: 385,282,465-byte archive.
- Forward cross-host admission run: 385,272,379-byte archive (exit code 0).
All recovery fixtures were destroyed; existing operator hosts were not rebooted or reconfigured.
Review-fix validation (2026-10-06)
Test evidence and run details
On 2026-10-06, the extended suite passed on two configured ARM64 hosts after redeploying review fixes. Both ran all 31 tool checks, six sizing profiles, and deterministic race rejection when a source container started before obtaining the host lock. Cross-host recovery passed with a 385,265,470-byte archive.
Both hosts had no remaining test boxes; existing hosts were not rebooted or reconfigured.
The image inspector passed 15 test cases, including non-root SSH keys, symlinks, hardlinks, and clean controls. Network testing verified initial and repeated firewall application while preserving unrelated chains.
The review-fix native run passed all native steps on ARM64 and AMD64: development-image builds, extended lifecycle with daemon restarts, capacity transitions, and clean-image inspection. Recovery archives were 3,844,136,311 bytes on ARM64 and 3,985,872,081 bytes on AMD64.
Disposable KVM host tests
Fresh provisioning on a disposable AMD64 KVM host confirmed:
- Repeat deployment removed injected rules while leaving Incus daemon start timestamps unchanged.
- Failed firewall installation blocked both normal and socket-activated Incus startup.
- Host reboot verified firewall rules became active before Incus started.
- A disposable pool resized to 61 GiB caused deployment to fail and preserved the existing size.
ARM64 host reboot and physical disk capacity remain outside this verification.
Native image validation (2026-10-06)
Test evidence and run details
On 2026-10-06, the native CI run built ARM64 and AMD64 images on Ubuntu 24 and passed extended lifecycle tests on both. Workspaces executed all six sizing combinations, 31 tool checks, sandboxed Chromium, noVNC desktop sessions, and peer isolation checks. Timings are recorded in Boot performance.
In the release-validation run, native image jobs passed full development-image recovery on both architectures. Package inventories matched and all 31 tool checks passed. Archives measured 3,844,135,523 bytes on ARM64 and 3,985,894,517 bytes on AMD64.
Fresh-host and reboot validation
Test evidence and run details
The release run validated fresh-host setup on a disposable AMD64 KVM instance running Ubuntu 24. The test verified canonical deployment, repeat deployment, image import, and lifecycle execution.
Following a VM reboot, changed kernel boot IDs and active services were verified alongside retained files, snapshots, container states, Docker, and bridge isolation. The host test uses a sparse 100 GiB virtual disk and accessible KVM.
It preserves the production deployer's 80 GiB free-disk check but does not prove physical storage capacity. ARM64 host reboot, host loss, and cross-architecture recovery remain unverified. The VM, SSH keys, and controller configuration were disposable; existing operator hosts were not rebooted or reconfigured.
Packaging and release checks
Test evidence and run details
Package builds run via uv build. Source and wheel archives are inspected to exclude version-control files and local caches. Clean wheel installation is verified outside the worktree alongside Ruff, syntax checks, and secret scans.
Packaging workflows passed on Python 3.11 and 3.14. Guidelines for visibility changes and repository hygiene are detailed in Publishing.
Capacity command validation (2026-10-06)
Test evidence and run details
On 2026-10-06, raft limits passed read-only text and JSON inspection and live lifecycle tests on two ARM64 Ubuntu hosts. Tests verified allocation adjustments during create, stop, resume, and destroy cycles.
Allocation fixtures validated mixed sizing budgets, frozen containers, low RAM and disk conditions, host reserves, and admission limits. The four-stopped-box admission test verified that creation, forking, and recovery reject a fifth container when slots are exhausted. These are conservative allocation checks, not workload saturation benchmarks.