Handbook

Running Apiary,
and building on it.

One page split into two audiences: operators who run a Colony, and contributors who work on the codebase underneath it. Both halves link out to the primary source - ADRs, runbooks, and the README - rather than duplicating it.

For operators

Vocabulary

Two words the rest of this page uses a lot, plus the containment hierarchy they sit inside.

Colony: One or more Combs that have agreed to be a single cluster, with one of them holding leadership at any moment. A Colony can be a single standalone Comb - that's first-class, not a degraded state.
Comb: One FreeBSD host running an Apiary installation.
Cell: One thing Apiary runs - either a VM (a bhyve virtual machine) or a jail (a FreeBSD jail). Both are addressed by an ID, unique only within its type.

Each Comb runs four services:

Service	What it does	Where you see it
`apiary_raftd`	The consensus layer. Holds the Colony's replicated state.	Never directly.
`apiary_managerd`	The authority. Every state change goes through it.	Never directly.
`apiary_frontend`	The web UI.	`http://localhost:8080`
`apiary_restshimd`	A REST view of the same API, for external tools.	`http://localhost:8081`

The web UI is where most work happens. apiaryctl is the command line for what has to happen on a headless host - which is, in practice, every Comb that is not your desk.

Bringing up your first Comb

A FreeBSD host, root access, and the source to build from - nothing else.

git clone https://github.com/glenjbarber/apiary
cd apiary
make setup
make install

make setup writes this host's configuration files and generates its serving certificate and identity. Do not run it on a Comb that is going to join an existing Colony - see below for why.

Once installed, the four services live under /usr/local/etc/rc.d/:

service apiary_raftd start
service apiary_managerd start
service apiary_frontend start
service apiary_restshimd start

Then open http://localhost:8080.

The first login becomes the Admin. Apiary creates no accounts on its own: until one exists, the login page says so, and the first successful login on this Comb is granted the Admin role. Every later account is created by an Admin, with a role of their choosing. If pam_service is configured, those accounts come from PAM instead, and the same first-login rule applies to the first PAM user.

The one thing to get right: do not start apiary_raftd on a host that is going to join an existing Colony. A fresh raftd with no await_join immediately forms its own independent cluster, and that cluster then has to be discarded before the join can work.

Adding a Comb to a Colony

A two-way, human-attested handshake - the human is not a formality, it is the security property.

A join request travels from the joining Comb to the target Colony, and something has to stop a stranger from asking to join. The answer is that the target's Admin reads a value off one screen and types it into another. That transcription is the check, and no amount of automation replaces it.

Prepare the joining host. Follow Bringing up your first Comb for host preparation, but stop before starting apiary_raftd.
The joining Comb introduces itself and prints a first code.
```
/usr/local/libexec/apiary/apiaryctl join-introduce --target leader.example.com:17700
```
This prints a 6-digit first code, this Comb's certificate fingerprints, and a request ID. It submits the request to the target; it does not approve anything.
The target's Admin approves it. On the target Colony's web UI, the pending request shows the joining Comb's fingerprints. The Admin compares them against what step 2 printed, and enters the second PIN the joining Comb is showing. The join ledger (Machine page, "Open the guided join flow") shows both Combs' steps side by side, and never renders the second PIN itself - that value is released only to the joining Comb.
The joining Comb polls for approval and installs its credentials.
```
/usr/local/libexec/apiary/apiaryctl join-introduce \
    --request-id jreq-XXXX --first-code 123456 \
    --target leader.example.com:17700 --install-credentials
```
This installs the Colony's peer credentials on the joining Comb and restarts managerd and frontend so they pick them up.
If the joining Comb was given a prepared certificate, add the reservation ID the preparing Comb printed:
```
/usr/local/libexec/apiary/apiaryctl join-introduce \
    --request-id jreq-XXXX --first-code 123456 \
    --target leader.example.com:17700 --install-credentials \
    --reservation-id 7Qk2...Vi
```
The reservation ID says which preparation is being collected - it is not what authorizes the collection (the seal addressed to this Comb's own certificate is), so presenting it is safe, and an ID that does not match the join is refused.

--field prints one bare value and nothing else, so it can go straight into a form on another machine - no labels, no quoting, no trailing text to accidentally paste:

apiaryctl join-introduce --target leader:17700 --field first-code
apiaryctl join-introduce --target leader:17700 --field fingerprints
apiaryctl join-introduce --target leader:17700 --field request-id
apiaryctl join-introduce --target leader:17700 --field second-pin

Roles

Three roles decide what an account can do, attached per-Comb - the same person can be Admin on one Comb and Viewer on another.

Role	Can do
`viewer`	Read. The monitoring API, the inventory, the health of the Colony.
`operator`	Read, and run workloads. Create and manage VMs and jails.
`admin`	Everything, plus changing the Colony itself: approving a join, managing API keys and accounts, pinning trust anchors.

An RPC that is not in the role map defaults to Admin, so a missing entry fails closed rather than open.

Workloads

VMs

A VM is a bhyve virtual machine. Creating one needs a base image - an ISO or a raw disk image. Installer media (an ISO) cannot be booted from as a base image; upload it as an ISO and use it as install media instead. A compressed raw image (.raw.xz, .raw.gz, .raw.bz2, .raw.zst) can be used as a base image and Apiary decompresses it while creating the disk - used as installer media it is refused, since there is nothing to boot from it. Upload images on the Images page, then create a VM on the VMs page.

Jails

A jail is a FreeBSD jail - a process-level container sharing the host kernel. It needs a base image too, but a jail template rather than a disk image.

Networks

Defined on the Networks page. A network needs a subnet and a bridge; the bridge has to exist on the host first.

Where a workload runs

Every cell is owned by exactly one Comb, and the Comb that owns it runs it. The Cluster view on the Machine page shows which.

Scripting against it

apiary_restshimd exposes the same read-only view as REST, for external tooling - Terraform, CI, a monitoring system:

curl -H "Authorization: Bearer $APIARY_API_KEY" http://localhost:8081/v1/vms
curl -H "Authorization: Bearer $APIARY_API_KEY" http://localhost:8081/v1/monitoring/health

Endpoint	What it returns
`GET /v1/status`	This Comb's status.
`GET /v1/health`	The Colony's health verdict.
`GET /v1/monitoring/health`	The Colony's verdict plus one row per Comb, with measurements.
`GET /v1/vms`	The VM inventory.
`GET /v1/jails`	The jail inventory.
`GET /v1/networks`	The network inventory.

A measurement that was not taken is absent, not zero - a monitoring system that reads "memory available: 0" during an outage will page an operator about a healthy machine.

Certificates

Every inter-host connection runs over TLS; a Comb's own serving certificate is what its peers pin.

The everyday case

Issued automatically at install time and renewed without a restart. The Certificates page shows what each Comb holds and how long it has left. You do not normally need to touch it.

Public certificates from Cloudflare

For a certificate from a public CA - so the Comb's name resolves publicly and is trusted by browsers - set up the Cloudflare integration once with the guided walkthrough:

/usr/local/libexec/apiary/apiaryctl cert-setup setup

It asks for a Cloudflare API token (read without echo, written to a 0600 file), resolves the zone, validates the token against that zone, asks for the naming suffix this integration may issue for, then obtains the certificate. The suffix matters: an empty one is refused rather than treated as a wildcard, since it is what stops the saved integration from issuing for a name you did not mean to give this Colony.

Preparing a certificate for another Comb reuses it - no credential re-entry:

/usr/local/libexec/apiary/apiaryctl cert-setup prepare --name host2.lab2.apiary.work

That prints a reservation ID, which the joining Comb presents when it collects the certificate during the join (see Adding a Comb to a Colony, step 5).

Rotating a certificate

A Comb's pinned certificate is replaced through a trust transition, not by overwriting a file. The new certificate is staged, both are trusted briefly, and then the old one is retired. A failed transition leaves the previous certificate trusted and serving.

When something is wrong

Start with the Cluster view

The Machine page's Cluster section shows every Comb's health verdict - five states, and one of them is the point:

Verdict	Meaning
`healthy`	Observed, and nothing wrong.
`degraded`	Observed, and something is wrong.
`stale`	The observation is old.
`unknown`	There is no verdict. Not the same as healthy.
`contradictory`	The Combs disagree about this Comb.

An unreachable Comb and a Comb that was never reached both read unknown - deliberately. Absence of a verdict is not a verdict, and conflating the two would report health that was never established.

The Recovery Handbook

The Recovery Handbook page is the built-in diagnostic for a Colony that is not working. It gathers the evidence - each member's state, the assumptions register, the health signals - and produces a numbered procedure with a stop condition on each step. It is deliberately offline: it gathers what it can from this Comb and prints the procedure rather than acting on the Colony, because a recovery procedure that acts on a failing Colony is a recovery procedure that can make it worse.

Command-line triage

service apiary_managerd status     # is it running?
service apiary_raftd status
sockstat -4 -l | grep 17700        # is managerd listening?

Restarting services

The Machine page's per-service control restarts a service and confirms it came back. From the command line:

/usr/local/libexec/apiary/apiaryctl force-restart

It restarts managerd then raftd, confirming each by its own listener port, and refuses to run on the Colony's current leader - restarting raftd on the leader is the one restart a Colony cannot absorb. It also refuses when it cannot determine leadership, because an unanswered question is not a permission.

The trust store

If peers cannot verify each other, the Colony's peer trust store may be empty - which happens on a Colony built before that store existed. pin-trusted-peers backfills it:

/usr/local/libexec/apiary/apiaryctl pin-trusted-peers \
    --target leader.example.com:17700 --bundle /tmp/certs.json

Every input certificate is parsed, evaluated, and matched before the first pin is sent - a bundle whose fourth certificate is bad produces zero pins, not three.

Command reference

apiaryctl is installed at /usr/local/libexec/apiary/apiaryctl and is not on PATH - invoke it by its full path. Run apiaryctl <command> --help for the full flags of any one.

Command	What it does	Root?
`install`	Generate this Comb's configuration, certificate and identity. Reports only unless `--apply`.	Yes
`join-introduce`	Start or poll a two-way Colony join. Prints values; submits no approvals.	For `--install-credentials`
`join-authorize`	Authorize one Comb to join, by writing a single-use entry.	Yes
`force-restart`	Restart `managerd` then `raftd`, confirming each. Refuses on the leader.	—
`pin-trusted-peers`	Backfill the replicated peer trust store.	Yes
`peer-credential-backfill`	Install the Colony peer key on a Comb that predates automatic delivery.	Yes
`restart-guardrail-provision`	Generate the restart-guardrail token on a Colony that has none.	Yes
`migrate-tls-dir`	Move the serving certificate to the current default location. Reports only unless `--apply`.	Yes
`colony-ca init`	Create the Colony CA, on the first Comb of a Colony.	Yes
`colony-ca issue-raft`	Sign this Comb's raft certificate with the Colony CA.	Yes
`cert-setup setup`	Guided Cloudflare setup and certificate issuance.	Yes
`cert-setup prepare`	Prepare a certificate for a Comb that has not joined.	Yes
`cert-setup status`	Report the saved integration and the certificates held.	—

For contributors

Architecture

Language: Go, across the entire stack.
Hypervisor: bhyve, for both VMs and jails.
Storage: ZFS, replicated node-to-node via HAST.
Consensus: A dedicated raft agent (hashicorp/raft), run as a separate process from the management daemon over a schema-defined Unix domain socket protocol.
External API: RPC-style first (CreateVM, MigrateVM, ...), with a REST translation layer on top.
Frontend: A Go backend serving a JSON API, with HTMX handling interactive elements server-side.

Repository layout

cmd/: Entry points: raftd, managerd, frontend, restshimd, apiaryctl, apiaryinstall, versioncheck.
api/: Protobuf schema: api/internalpb (internal raft socket protocol) and api/rpc (external RPC API).
internal/: Core logic, one package per feature or ADR - bhyve, jail, zfs, hast, raft, manager, restshim, frontend, and dozens more.
web/: HTML templates and static assets for the frontend, embedded into the frontend binary at build time.
docs/adr/: Architecture decision records - start here for why things are built the way they are.

Building and testing

Standard Go tooling: go build ./..., go vet ./..., go test ./.... Packages that shell out to FreeBSD-only tools (internal/zfs, internal/jail, internal/hast, internal/bhyve) skip their integration tests cleanly on a non-FreeBSD dev machine - cross-compile (GOOS=freebsd GOARCH=amd64 go test -c ./internal/<pkg>) and run the resulting binary on a FreeBSD host to exercise them for real.

Proto changes require buf (buf generate); generated .go files are committed, so this isn't needed just to build. cmd/frontend links internal/pam (cgo) for real PAM login and can't be cross-compiled from macOS - build it natively on a real FreeBSD host instead. managerd/raftd/restshimd are unaffected.

Design records

Everything above is self-contained - nothing here requires reading an ADR first. For the reasoning behind a design, or the full history of a decision, the primary source is below.

Browse docs/adr Code audits Runbooks

Running Apiary,and building on it.