Runbook 01 — Provision: Proxmox foundation + all VMs¶

Goal: lay down everything the build runs on — the VLAN-aware bridge, a pool to group the lab machines, the AlmaLinux template, the Linux VM fleet (cloned by script), and the OPNsense VM (ISO). Soup-to-nuts starts here; OPNsense install is runbook 02, and the Ansible config/maintenance layer closes it out in runbook 16.

Outcome: a lab pool holding the OPNsense VM (booting the installer) and the cloned EL VMs — all on the right bridges/VLANs, addressed in 10.20.0.0/16.

0. Facts to confirm¶

Proxmox node: pve · WAN bridge: vmbr0 · LAN bridge (trunk): vmbr1 ✓
Guest storage: vmpool-enc — the encrypted SSD pool. For a clean build, stand up tang1 (runbook 24) and create the encrypted vmpool (runbook 25 §1–5) first, so guests land encrypted from the start. local-zfs (rpool) holds the pve OS + the templates; ISO storage local ✓.
VMIDs (91xx range, free): AlmaLinux VM template 9190, OPNsense 9140, fleet 9100–9130 (provision/vms.conf). LXC guests use the AlmaLinux 9 LXC template (pveam, §3b) — not a VMID.
Your SSH CA public key file (ssh_user_ca.pub) — installed into the template (§3) so every clone trusts CA-signed logins.
SSH pubkey file ~/.ssh/lab_authorized_keys (cloud-init bootstrap; or set SSHKEYS); CIUSER default almalinux (the Alma image's built-in user).

Addressing: the lab lives entirely in 10.20.0.0/16 with VLAN tags 10–90 (the tag = the subnet's third octet, a deliberate mnemonic). The firewall is .1 in every zone; DHCP pools .100–.199; static/servers .2–.99.

1. VLAN-aware LAN bridge¶

The LAN bridge must pass tagged VLANs (10/20/30/40/60/90) as a trunk. Confirm it's VLAN-aware (read-only):

grep -A6 "iface vmbr1" /etc/network/interfaces

Look for bridge-vlan-aware yes + a bridge-vids range → if present, skip. If not: Datacenter → pve → System → Network → the LAN bridge → Edit → tick VLAN aware, VLAN IDs 2-4094, Apply (ifreload -a).

2. Proxmox pool for the lab VMs¶

Group the lab machines into one pool for visibility + permission scoping:

pvesh create /pools --poolid lab

(A lab VM tag is a lighter, ACL-free alternative.) Both the script (below) and the OPNsense qm create add VMs to this pool.

3. Build the AlmaLinux 9 template (once) — VMID `9190`¶

On pve:

# 1. fetch the Alma 9 GenericCloud image (has cloud-init)
cd /var/lib/vz/template/iso
wget https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-latest.x86_64.qcow2

# 2. create the VM shell + import the disk + a cloud-init drive
qm create 9190 --name alma9-tmpl --ostype l26 --cpu host --memory 2048 \
  --scsihw virtio-scsi-single --net0 virtio,bridge=vmbr1 \
  --agent enabled=1 --serial0 socket --vga serial0
qm importdisk 9190 AlmaLinux-9-GenericCloud-latest.x86_64.qcow2 local-zfs
qm set 9190 --scsi0 local-zfs:vm-9190-disk-0 --ide2 local-zfs:cloudinit \
  --boot order=scsi0 --ciuser almalinux --sshkeys ~/.ssh/lab_authorized_keys \
  --ipconfig0 ip=dhcp
qm start 9190

SSH in (cloud-init user) and customize, then seal:

sudo dnf -y install qemu-guest-agent && sudo systemctl enable qemu-guest-agent

# trust your SSH CA: install the CA *public* key + tell sshd to honor its certs
sudo install -m644 ssh_user_ca.pub /etc/ssh/ssh_user_ca.pub
echo 'TrustedUserCAKeys /etc/ssh/ssh_user_ca.pub' | sudo tee /etc/ssh/sshd_config.d/10-ssh-ca.conf

# seal so each clone is unique
sudo cloud-init clean --logs
sudo truncate -s0 /etc/machine-id
sudo rm -f /etc/ssh/ssh_host_*
sudo poweroff

Convert to a template:

qm template 9190

SELinux stays enforcing. cloud-init regenerates SSH host keys + machine-id on each clone's first boot (no duplicate leases / key collisions). The Ansible common role also manages TrustedUserCAKeys, so the CA trust is config-as-code too (runbook 16) — the template just makes it true from first boot.

3b. Fetch the AlmaLinux 9 LXC template (once)¶

Lighter single-app guests run as LXC (see the taxonomy in §4). Download the stock CT template — the per-CT bootstrap (sshd, almalinux, CA trust) is done by lab-provision.sh, not baked here:

pveam update
pveam download local almalinux-9-default_20240911_amd64.tar.xz

(Override the pinned tag via EL_LXC in lab-provision.sh; bump deliberately.)

4. Provision the fleet (script) — VMs and LXC¶

provision/vms.conf is the lab definition. Each row's optional TYPE column (vm | lxc, default vm) picks how it's built; everything lands on STORAGE (vmpool-enc). Set site values via env or the top of lab-provision.sh (EL VM-template id, EL_LXC CT template, LAN_BRIDGE, STORAGE, POOL, CIUSER, SSHKEYS, USER_CA, NAMESERVER). Run on the Proxmox node:

./lab-provision.sh list                 # show the manifest (incl. TYPE)
./lab-provision.sh up all               # build+start the fleet (VMs + LXC)
./lab-provision.sh up caddy             # just one
./lab-provision.sh rebuild all          # destroy + recreate everything

VM-vs-LXC taxonomy (per-app IP/zone is identical either way — segmentation preserved): - VM (qm clone $EL): guests needing their own kernel / PCI+USB passthrough / an appliance image / a vTPM — OPNsense, Home Assistant, tvheadend, fileserver, FreeIPA, Nextcloud, ctrl, runner (runner executes arbitrary CI builds via rootful podman → needs a VM kernel boundary). up does: full-clone → cores/RAM + disk → tag the NIC into its zone VLAN → cloud-init sets IP/gw/DNS/user/SSH-keys + a vendor-data snippet that pre-seeds CA-trust + auth_principals (admin + the zone principal) — the VM analog of the LXC bootstrap below, so a CA-signed control/agent cert connects on the first Ansible run (cert-based, no raw key); common rewrites auth_principals per-host after → guest-agent + onboot → start. (The vendor-data snippet needs snippets content on the local storage: pvesm set local --content vztmpl,backup,iso,snippets.) - LXC (pct create $EL_LXC, unprivileged, nesting=1): the lighter native single-app guests (the 3 Caddy proxies, bind, jump) and the fixed-quadlet podman/docker hosts (obs, ntfy, authelia, forgejo, searxng, fmd, jellyfin, invidious, umami). The stock CT lacks the sshd / almalinux user / CA-trust the VM cloud-image shipped, so up reproduces them in-script: openssh-server+python3, the almalinux user, the user-CA + TrustedUserCAKeys, a pre-seeded auth_principals (so ctrl's control cert — principal admin-<zone>, not the login name — is accepted on the first run, before common rewrites it), the bootstrap key, NOPASSWD sudo. After that an LXC is identical to a VM from Ansible's view. (common skips the VM-only qemu-guest-agent on containers; size guest rows ≥ 2 GiB RAM so dnf transactions don't OOM — EL9 dnf OOM-kills at 1 GiB, on a VM as readily as an LXC.)

All guests join the lab pool. Encrypted-pool details + DR: runbook 25.

The app VMs boot now but only reach the network once OPNsense routes their VLANs (runbooks 03–04). You can up all now (they idle until then) or clone each when its app runbook calls for it — either is fine.

5. Create the OPNsense VM (ISO — not from the template) — VMID `9140`¶

Download the OPNsense dvd ISO to local. Current release is 26.1.6; images are published per series (26.1) and the installer + runbook 02's firmware update bring it to the latest patch (.6):

cd /var/lib/vz/template/iso
VER=26.1          # release series (patch 26.1.6 applied via the first update)
wget https://mirror.opnsense.org/releases/${VER}/OPNsense-${VER}-dvd-amd64.iso.bz2
bunzip2 OPNsense-${VER}-dvd-amd64.iso.bz2
# if the mirror lists a full-patch image instead, use that filename directly

Create the VM:

qm create 9140 --name opnsense --ostype l26 \
  --cpu host --cores 8 --sockets 1 --memory 8192 \
  --scsihw virtio-scsi-single --scsi0 local-zfs:32 \
  --ide2 local:iso/OPNsense-${VER}-dvd-amd64.iso,media=cdrom \
  --boot order='ide2;scsi0' --agent enabled=1 \
  --serial0 socket --vga serial0 --pool lab

Sizing for max VPN throughput: 8 vCPU (host), 8 GB. ALL traffic egresses through WireGuard here, so cores matter (kernel WG is CPU-bound). --cpu host exposes AES-NI.

Attach the two NICs (order + multiqueue matter)¶

OPNsense names by discovery order: net0 → vtnet0 (LAN), net1 → vtnet1 (WAN). Set queues= to the vCPU count so packet processing spreads across cores — without multiqueue, all WG forwarding pins to one core and caps low:

qm set 9140 --net0 virtio,bridge=vmbr1,queues=8     # LAN trunk, NO tag
qm set 9140 --net1 virtio,bridge=vmbr0,queues=8     # WAN

- No tag= on net0 — it receives the full trunk; OPNsense tags internally (runbook 03). - virtio NICs throughout (the vtnet* driver).

6. Start¶

WAN takes a DHCP transit lease (the ISP router routes the public block to it); the only public address assigned is the .69 IP-Alias VIP for the WG admin endpoint (runbook 02 §2.3). .65–.68 stay reserved.
```
qm start 9140 && qm terminal 9140     # reach the OPNsense installer
```

7. Boot orchestration — `provision/startup-order.sh`¶

A pve reboot starts the whole fleet. Left to itself that is a boot storm: guests start near-simultaneously, pve thrashes on CPU/IO/RAM, and several come up wedged — running but with sshd/agent hung. startup-order.sh gives every guest a unique startup=order=N,up=S so they start staggered, foundations first (network/DNS → monitoring → identity/storage → edge + internal planes → leaf apps → heaviest last).

# on pve, as root — idempotent (qm/pct set is declarative)
cd /root/myos/provision
DRY_RUN=1 bash startup-order.sh    # preview; changes nothing
bash startup-order.sh

Run it after provisioning any new guest. You do not have to edit it first: the guest set is derived from vms.conf, so a newly provisioned guest is picked up automatically, appended at the end, and reported — it can no longer be silently missed. Edit the in-script BOOT_ORDER list only to place a guest deliberately (i.e. something must boot before it). Order numbers are computed, so inserting one never means renumbering.

It reconciles both directions on every run, and says so out loud:

condition	what you see	what to do
guest in `vms.conf`, not in `BOOT_ORDER`	`NOTE: … auto-appended`	nothing, unless it has boot deps — then place it
name in `BOOT_ORDER`, not in `vms.conf`	`WARNING: … retired guests?`	remove the stale line
guest ordered but not on this node	`SKIP … declared, not provisioned`	nothing (normal before provisioning)

Why derived (2026-08-01). It used to be a hand-kept list of VMIDs and it had rotted silently: eight provisioned guests were missing and one retired guest (authentik) was still listed. A guest missing here still boots — with no startup config, so it starts concurrently with every other unordered guest, which is the storm this file exists to prevent. Worse, the "re-run startup-order.sh" step in runbooks 35 and 37 was a no-op for exactly the case it was written for: re-running it after provisioning did not order the new guest, because the new guest wasn't in the list. The same pass also found haos at order=5, colliding with mgmt-proxy. Guests not in vms.conf (the OPNsense and HAOS appliances, the edge VMs) still need an explicit VMID — they live in a short OFF_MANIFEST map at the top, and that map staying short is the point.

Guests with onboot unset are never autostarted, so their order is inert and the script leaves them alone. Templates carry no startup config — also correct.

Validation¶

[ ] lab pool exists; cloned EL VMs + the OPNsense VM are in it.
[ ] A cloned VM has a unique machine-id + SSH host keys, correct hostname/IP, and the right zone VLAN tag on its NIC.
[ ] OPNsense VM: net0 on vmbr1 (no tag), net1 on vmbr0, both virtio; boots to the installer console.
[ ] LAN bridge confirmed VLAN-aware.
[ ] bash startup-order.sh reports no NOTE:/WARNING: block, and every autostarting guest has a unique order: for f in /etc/pve/{qemu-server,lxc}/*.conf; do [ "$(sed -n 's/^onboot: //p' $f)" = 1 ] && sed -n 's/^startup: order=$[0-9]*$.*/\1/p' $f; done | sort -n | uniq -d prints nothing.

Then → runbook 02 (OPNsense install + base).