⚙️ CI Architecture: Hammers, Furnace & Incubator

The CI of penguins-eggs is split into three specialized pipelines, all living in .github/workflows/:

Pipeline	Workflow	Where it runs	What it produces / tests
🔨 Hammers	hammers.yml	GitHub-hosted runners (containers)	Native distribution packages (.deb, .apk, .pkg.tar.zst, .rpm)
🏭 Furnace	furnace.yml	Self-hosted runner + Proxmox VE	Bootable live ISO images generated via eggs remaster
🐣 Incubator	incubator-go.yml	Self-hosted runner + Proxmox VE	Installation validation of live ISOs onto target disks (ext4, btrfs, BIOS, UEFI)

The split follows a simple observation: packaging is user-space work and runs happily inside GitHub's containers, while remastering and installation testing need a real kernel, real mounts, storage drives, and hypervisor control — things a hardened CI container cannot provide.

---

🔨 Hammers: the Packaging Matrix

Runs on every push and pull request to main (plus manual dispatch). A fail-fast: false matrix spins up one container per target distribution:

Distro	Image	Package
Alpine	alpine:latest	.apk (via abuild)
Arch	archlinux:latest	.pkg.tar.zst
Manjaro	manjarolinux/base:latest	.pkg.tar.zst
Debian	debian:bookworm	.deb
Fedora	fedora:latest	.rpm
openSUSE	opensuse/tumbleweed	.rpm

Each leg of the matrix performs the same ritual:

1. Workshop setup: installs the native toolchain (GCC, Go, make, the distro's packaging tools) and creates the unprivileged artisan user.
2. Checkout & build: full-history checkout (tags included, used for versioning), then make compiles both oa (C) and coa (Go).
3. Native packaging: make package runs as artisan and drives the distro-specific packager (abuild, makepkg-style, dpkg, rpmbuild).
4. Live install test: the freshly built package is installed on the running container — a real smoke test of the package metadata and file layout.
5. Artifact upload: the package is published as a GitHub artifact (penguins-eggs-<distro>, 7-day retention).

Hammers therefore answers the question: does the codebase compile and package cleanly on every supported family?

---

🏭 Furnace: Remastering on Real Iron

Triggered manually (workflow_dispatch), Furnace runs on a self-hosted runner that orchestrates the Proxmox VE host (father). Each matrix entry maps a distribution to a dedicated KVM virtual machine with a pristine snapshot:

Distro	VMID	Snapshot
Alpine	301	virgin
Arch	302	virgin
Debian	303	virgin
Fedora	304	virgin

Why only these four? CLI editions of Manjaro are hard to source, and openSUSE support is currently lagging behind. The Furnace matrix will grow as those gaps close.

The flight plan of each job:

1. Secrets (air-gapped): credentials are sourced from /etc/p4/secrets.env on the runner itself — never stored on GitHub — and masked in the logs with ::add-mask::.
2. Rollback & boot: the VM is rolled back to its virgin snapshot and started (qm rollback + qm start on father). Every run begins from an identical, uncontaminated system.
3. Dynamic IP discovery: the VM's MAC address is read live from qm config, then a fast ARP sweep over the local subnet locates the assigned IP — no static leases required.
4. Wait for SSH: the job polls until the guest's SSH daemon answers.
5. Install & remaster: the latest released package for that distro is downloaded from GitHub Releases, installed natively, then sudo eggs remaster bakes the ISO on a real kernel with real mounts.
6. Export: the resulting ISO is shipped to the Proxmox storage (export iso --clean, which also prunes older versions on the server).
7. Shutdown: the VM is powered off (if: always()), leaving the hypervisor clean even on failure.

Furnace therefore answers the question: does the full remastering chain actually produce a bootable ISO on every supported family?

---

🐣 Incubator: Live ISO Installation Testing

Triggered manually (workflow_dispatch via incubator-go.yml), Incubator leverages incubator-go — a high-performance, zero-dependency Go orchestration tool designed to automatically deploy, install, and rigorously test Linux ISOs generated by penguins-eggs on Proxmox VE.

By combining Proxmox VE and the QEMU Guest Agent, Incubator acts as an automated QA inspector: it boots the live ISO, executes a fully unattended installation via krill, reboots, and extracts system telemetry to verify the integrity of the newly minted OS without human intervention.

🏗️ Architecture & Key Features

• Worker Pool (Semaphore Pattern): Uses Go routines to test multiple ISOs concurrently. Keeps the Proxmox host at maximum efficiency without overloading disk I/O, safely queuing tasks and managing hardware limits.
• Zero External Dependencies: Compiled as a single standalone Go binary. No Python virtual environments or Node.js runtime setup required on self-hosted runners.
• Bulletproof Guest Control: Communicates exclusively via the QEMU Guest Agent. Includes advanced anti-hang countermeasures ("Ghost Finger" and "Guillotine" fallbacks) to prevent zombie VMs on test failures.
• Universal Topology Validation: Tests complex boot topologies including UEFI/BIOS, Btrfs subvolumes, ext4 root partitions, and strict SELinux auto-relabeling workflows.
• Automated Telemetry: Dynamically extracts fstab and fastfetch/neofetch diagnostic reports directly from the installed target disk.

🎛️ Test Matrix & Workflow Execution

Parameter	Options	Description
fstype	all, ext4, btrfs	Target filesystem to partition, format, and deploy.
firmware	bios, uefi	Target VM firmware environment.
vmid	150 (ext4/bios), 160 (ext4/uefi), 170 (btrfs/uefi)	Dedicated Proxmox test VM instances.

1. Matrix setup: dynamically configures the matrix based on selected filesystem (ext4, btrfs, or all).
2. Remote execution: triggers incubator-go on the Proxmox host via SSH to boot the ISO, run the installer (krill), and verify disk deployment.
3. Report aggregation: collects Markdown summaries generated by incubator-go, uploads them as GitHub artifacts, and appends the test matrix table directly to $GITHUB_STEP_SUMMARY.

Incubator therefore answers the question: can an ISO produced by penguins-eggs be successfully installed and booted on target disks across ext4, btrfs, BIOS, and UEFI?

For the host/guest configuration behind this setup (VirtFS, QEMU Guest Agent, snapshots), see proxmox.md.