sneak preview: uv, Click, Rich, tui; flavors

[rpardini]

sneak preview: uv, Click, Rich, tui; flavors

• 🌵 captain: fix .isEnabledFor(logging.DEBUG) calls
• 🐸 captain: force type console in init.py
• 🌴 captain: catch required arg and loosen type (WiP: other way around.)
• 🌿 captain: util: Rich rule's separating run() output
• 🌱 captain: better logging; in-docker: prefix
• 🌳 captain: add .editorconfig matching ruff
  • .editorconfig for shell scripts and shell templates
• 🍃 docker/mkosi: mount a Docker Volume at /work, so mkosi can cache the tools tree across invocations
• 🍀 util: use a Panel to log the full subprocess command line
• 🌵 captain: introduce Config build_kernel boolean
  • this will allow using a generic/distro kernel instead of building one
• 🐸 WiP: use linux-image-generic distro kernel
  • hack until it uses the mkosi-supplied kernel for everything
  • artifacts: look for mkosi-supplied vmlinuz first
• 🌴 gha: skip all kernel stuff
  • self-built kernel from source will come back at a later stage, as a .deb package
• 🌿 gha: single matrix except for combined amd64/arm64 image at the end
  • just do everything at once
• 🌱 mkosi.conf: just use tiny-initramfs instead of blocking the others (dracut/initramfs-tools)
• 🌳 mkosi.finalize: export DTBs for arches that have them
  • also: show info about modules and full rootfs
• 🍃 artifacts: collect dtb mkosi.output folder into out/
• 🍀 publish: publish DTBs in OCI image as a directory
• 🌵 kernel: drop all kernel-build related code
  • later to be reborn as standard apt package
• 🐸 mkosi.finalize: drop kernel sound/media/net-wireless modules
  • mkosi.finalize: clean up logging a bit
  • mkosi.postinst: debugs
• 🌴 gha: introduce FLAVOR_ID (nee KERNEL_VERSION) gha: fix
• 🌿 release: include cfg.kernel_version as suffix to all tags
• 🌱 gha: bump external actions so GHA stops complaining about Node 24
• 🌳 gha: pass DEFAULT_FLAVOR_ID as KERNEL_VERSION for publish-combined gha: pass DEFAULT_FLAVOR_ID as KERNEL_VERSION for publish-combined - retry
• 🍃 captain/gha: KERNEL_VERSION (et al) is now FLAVOR_ID
  • --flavor-id
• 🍀 captain: cleanups / doc updates WiP (drop kernel build-related)
• 🌵 captain: DEFAULT_FLAVOR_ID = "trixie-full"
• 🐸 flavors: introduce flavors but logging sucks and no dataclass better, logging still sucks for Full -> Common much better in-package this sucks less cosmetics some static and template rendering and hardcoded cleanup kinda-works kinda works
• 🌴 gha: add trixie-rockchip64 flavor
• 🌿 stages: show what tools_mode is running
• 🌱 gha: pass --arch to build.py build
• 🌳 common_debian: add 01nopty with Dpkg::Use-Pty "0"; to mkosi sandbox tree
• 🍃 captain: add support for mkosi skeleton tree
  • gha: still trying to fix dpkg/apt output in gha
  • this went nowhere, but hindsight is 20/20
• 🍀 common_debian: common bash header with logging, sprinkle some dust on postinst and finalize
• 🌵 mkosi.conf: output a JSON manifest
• 🐸 mkosi.conf: force tools tree to be Debian Trixie
  • mkosi defaults to 'testing'
• 🌴 mkosi.conf: use sandbox tree also for tools tree
• 🌿 flavor: ensure flavor supports requested architecture
• 🌱 flavors: cleanup a bit
• 🌳 flavors/gha: introduce meson64 flavor
• 🍃 docker: never interactive, never a terminal
• 🍀 cli/flavor: pass Flavor down to build and initramfs cmds; introduce BaseFlavor::has_iso()
  • so only the fully UEFI/ACPI flavor gets ISOs by default
  • they don't make sense for DeviceTree flavors
• 🌵 gha/docker: set and pass down FORCE_COLOR=1
• 🐸 captain: don't shorten out logging record name
• 🌴 buildah: take env BUILDAH_INSECURE=1 for --tls-verify=false
  • so one can use an HTTP local registry:2 for testing pushes
• 🌿 release: pass flavor down to release, avoid releasing iso for !has_iso
• 🌱 cli: new cli under click_cli
  • add to pyproject.toml, thus uv run captain ...
• 🌳 click: add tools command
  • using captain.cli._stages
• 🍃 click: some sugar, list available flavors via reflection
• 🍀 docker: build_builder(): more info
• 🌵 click: kill old tools cli (keep stage) click: kill old cli completely
• 🐸 click: add iso command back
  • as it is called internally via docker
• 🌴 gha: switch to click cli
• 🌿 gha: release-publish
• 🌱 captain: rework obtain_builder
• 🌳 gha: separate build-dockerfile arch-based matrix job
• 🍃 captain: adapt to options moved to common
• 🍀 captain: simplify Rich logging and trace handler captain: simplify Rich logging and trace handler; let's not have two click packages
• 🌵 gha: force tools to run native
• 🐸 docker/Dockerfile: consolidate single Dockerfile; split and balance layers to optimize for parallel pulls
• 🌴 docker: rework re-launching inside docker & docker envs
• 🌿 docker/mkosi: get rid of tools tree; back to Debian's trixie system Python + pip for mkosi
  • mkosi re-launches itself and ignores uv's env without a tools tree
  • uv instead does use the system Python if it matches constraints
  • build in /work in Dockerfile, allows to reuse .venv
  • don't mount a volume in /work, instead use the one provided by the Dockerfile
• 🌱 captain: honor FORCE_COLOR=1 for internal logging
• 🌳 flavors: tighten semantics, refactor
  • refactor add_static_dir() into BaseFlavor
  • make has_iso() abstractmethod at BaseFlavor level
  • make kernel_packages() abstractmethod at common-debian level
• 🍃 flavors: introduce common-acpi, move acpi/impi stuff there
  • so it's common-debian > common-acpi > trixie-full
  • in the future: common-debian > common-acpi > trixie-mainline
  • common-acpi implements has_iso() as True
• 🍀 click: introduce CliContext and reuse common options via @click.Group
  • CliContext is a Config factory, taking both common and specific options
  • actually implement --verbose, thus new default logging level is INFO
• 🌵 captain: make Rich richer
  • show Locals
  • show all frames in traces
  • use Highlights for FORCE_COLOR=1
• 🐸 captain: introduce Trogon(/Textual) to auto-create a TUI from Click
  • See https://github.com/Textualize/trogon and https://github.com/Textualize/textual
  • this cashes out on the Click / Rich investment
• 🌴 gha: use envvars, not --options
  • reduce suffering while I juggle Click
• 🌿 captain: even Richer Rich
  • uv: split Config::verbose_uv from --verbose/Logging.DEBUG; use --quiet otherwise
  • use shutil.get_terminal_size to obtain and pass-down-Docker COLUMNS
  • nicer logging format, use a whale emoji for in-Docker logs
  • show Rich Table with Docker environment vars if --verbose
  • show Rich Panel and Rich Syntax for util.run() if --verbose
• 🌱 gha: don't upload .iso as part of initramfs artifact
  • we've a separate artifact for .iso
• 🌳 flavor/gha: add trixie-rockchip64-vendor and trixie-armbian-rpi flavors
  • also build them
  • trixie-armbian-rpi strays from standard naming as I think we might need to build our own kernel for RPi to achieve small-enough-for-eeprom-netboot eventually
• 🍃 captain: tink-agent-setup Wants/After time-set.target
  • so time is set to something more reasonable, otherwise pulling fails with date-related TLS issues
• 🍀 captain: move systemd enablement/disablement from postinst to systemd preset files
  • doing this in postinst has no effect, as mkosi applies presets after postinst
  • it could be done in finalize, but that would un-do the presets applied by mkosi
  • so instead do it all in two preset files:
    • one for generic OS-related enablement and disablement (20-captainos-base)
    • one for tink-agent related stuff (10-captainos-tink)
• 🌵 captain: loosen systemd-networkd-wait-online to accept any interface and 15s timeout
  • otherwise hangs and delays on machines with multiple interfaces
• 🐸 captain: tink-agent-setup Wants/After time-sync.target
  • time-set.target is not enough for machines without an RTC and working battery
  • time-sync.target requires timesyncd to be fully sync'ed
  • so time is set to something more reasonable, otherwise pulling fails with date-related TLS issues
• 🌴 kernel: bring back pre-Click kernel impl
• 🌿 util: show run()'s env vars in a Rich Table
• 🌱 docker: rework run_in_builder() with both command_and_args and extra_docker_args
  • also introduce cfg.verbose_docker for setting env CAPTAIN_VERBOSE=1 when relaunching
• 🌳 captain/cli: --verbose sets root level as well as current logger to DEBUG
• 🍃 cli: re-introduce shell subcommand
  • which runs bash interactively under Docker
  • using same envs / volumes / mounts as regular run_in_builder()
• 🍀 kernel: rework with kernel's own make bindeb-pkg into a .deb output
  • kernel packaging code already knows how to:
    • strip modules
    • zstd modules
    • handle Image name differences
    • handle depmod
    • produce a .deb package
  • but we gotta go 2-deep as .deb's are output in parent directory by kbuild
  • Dockerfile: add layers with deps:
    • rust compiler layer (inevitable after 7.0.y)
    • debhelper + etc needed for kernel deb packaging
• 🌵 flavor: introduce BaseFlavor::pre_mkosi_stage()
  • so flavors can assert their requirements are built/met
• 🐸 mkosi: show Rich.Syntax'ed mkosi.conf before running mkosi
  • ... since we already paid the price of Pygments...
• 🌴 captain: allow building amd64 kernel on arm64
  • specify a CROSS_COMPILE for x86_64
• 🌿 common_debian: introduce package_directories() for mkosi's PackageDirectories=
  • this allows flavors to add .deb's in directories as a local apt repo
  • .deb's can then be installed by name
  • mkosi handles the repo creation and usage directly (via apt-utils)
• 🌱 builder: Dockerfile: add cross-arch libssl-dev and native apt-utils
  • cross libssl-dev needed for make bindeb-pkg to work cross-arch
  • apt-utils needed for mkosi's PackageDirectories=
• 🌳 cli: expose stages (nee _stages) module
  • this skews package protection to allow flavors to reuse code
  • also expose function build_kernel_stage()
• 🍃 flavors: introduce trixie-slim flavor, using captainos kernel
  • this one uses the captain-build-from-source kernel .deb
  • let's violate some package boundaries:
    • cli acrobatics: delegates to kernel build stage before initramfs build
    • impls package_directories() by calling kernel
• 🍀 kernel: introduce --config for round-trip menuconfig / savedefconfig
  • interactive, handle when re-launching in Docker
  • just "Exit" to update a defconfig with no changes
  • eg uv run captain --verbose --arch amd64 kernel --config
• 🌵 kernel: 6.18.y.amd64: defconfig standard
• 🐸 kernel: 6.18.y.amd64: CONFIG_MODULE_COMPRESS_ZSTD=y
• 🌴 kernel: 6.18.y.arm64: defconfig standard
• 🌿 kernel: 6.18.y.arm64: defconfig: CONFIG_MODULE_COMPRESS_ZSTD=y
• 🌱 gha: reintroduce kernel build jobs
  • using the Docker builder, so depend on that
  • abuse GHA caches and artifacts
  • the whole matrix will wait on kernel builds
    • but only a few will actually download and use the artifact
• 🌳 gha: split matrix to add kernel job dependency for trixie-slim jobs
  • add gha builds for trixie-slim flavors (using captainos-kernel)
  • we've two kinds of build-captainos jobs:
    • ones that depend on kernel-build (trixie-slim)
    • ones that don't (all the others)
  • use YAML anchor trick to avoid duplicating 95% of code (envs: and steps: are defined only once)
  • matrix and needs (dependencies) are the only difference
  • artifact download is common but conditional on matrix values
• 🍃 gha: make trixie-slim the default flavor
  • so it is the one included in the combined images
  • this does not change the CLI defaults, ofc
  • for trixie-slim, combined should depend on build-captainos-with-kernel
  • better names for matrix jobs
• 🍀 gha: don't push OCI images for pull request workflows
  • (Docker) builder will end up being built multiple times for PR workflows, but such is life
    • it's also too huge for GHA artifact or GHA cache
  • don't push any captainos artifacts either (simple or combined)
    • also: only push those when building main branch
• 🌵 cli: bring back qemu cli, with extra support for arm64 & UEFI/OVMF
  • arm64 qemu requires OVMF/AAVMF firmware (edk2) via a pflash
  • use q35 machine type for amd64 (it is still not a default)
  • add kvm (Linux) and hvf (Darwin) acceleration if present and matching host/target arch
  • add systemd.journald.forward_to_console=1 as default prefix kernel cmdline
  • remove audit=0 as cmdline prefix default
  • run with eg uv run captain --verbose --arch=amd64 --flavor-id=trixie-slim qemu -- tink_worker_image=... <...>
    • extra args after -- are passed directly as kernel cmdline
    • maybe restore option/env parsing with Click later (kept TODO'ed _TINK_PARAMS) for ergonomics
  • qemu: more systemd logging in default kernel cmdline
  • qemu: run x86 also under OVMF (UEFI), not BIOS emulation
    • this more accurately emulates real environment
      • real EFI implementation, with efivars etc
    • unify accel handling (accel only when host arch matches target arch)
      • but then it's the same across arches: hcf for Apple, kvm for Linux
    • refactor find_ovmf_firmware() as it's common
• 🐸 kernel: hash defconfig + kernel.py code and use as cache key; clean output when cache missed
  • that way any changes to defconfig or kernel-building logic should force a new build
  • on new builds, clean up old artifacts so caches are kept clean (eg: GHA's)
• 🌴 gha: drop DEFAULT_FLAVOR_ID; make publish-combined a matrix (for both trixie-slim and trixie-full)
  • so we've two combined images now, and no longer a default one
• 🌿 timesyncd: retry connections every 2s
  • seems like timesyncd is interrupted by networking changes constantly during bootup
  • each change triggers a retry timeout, which is 30s by default
• 🌱 util: debug: quote run() params with spaces or specials in them
• 🌳 flavors: flavor_packages() hierarchy, include_apt(), include_hwdb()
  • include_apt(): if true, includes a working apt install
  • include_hwdb(): if true, leaves hwdb data in
  • move acpid package to common_acpi flavor
  • also always include curl, pciutils, pv
• 🍃 flavors: introduce trixie-genio for Armbian's linux-image-edge-genio
• 🍀 gha: add trixie-genio build
• 🌵 kernel: 6.18.y amd64: CONFIG_BE2NET=y (from kernel: enable be2net network driver for amd64 #59)
  • via uv run captain --arch amd64 kernel --config, then pick CONFIG_BE2NET
• 🐸 docker: pass --load to docker buildx build
  • from Improve Docker build reliability and add build idempotency: #64
• 🌴 release-publish: fix to use run_captain_in_builder()
• 🌿 initramfs/iso: introduce content-hashing for idempotency
  • flavor can produce a hash of contents
  • incl static + templated files
  • does NOT include extratrees or packagedirs yet (kernel/tools)
  • does NOT clean up leftovers yet
• 🌱 gha: set CAPTAIN_VERBOSE=1 & allow workflow_dispatch
• 🌳 gha: add GHA caches for initramfs and iso
• 🍃 tools: default to native mode (avoid pulling builder for cache hits)
• 🍀 publish: more determinism
  • deterministic tar for dtb layer too
  • use epoch as created date, so manifests also hit
• 🌵 flavors: common-debian: reduce postinst spurious debug logging
• 🐸 captain: split mkosi.input from mkosi.output
  • tools and kernel are really inputs to mkosi, not outputs
• 🌴 gha: split mkosi.input from mkosi.output
  • tools and kernel are really inputs to mkosi, not outputs
• 🌿 publish: let flavor decide which files/dirs are published in OCI images
  • BaseFlavor:
    • list_arch_artifacts() handles vmlinuz, initramfs, and iso directly
      • delegates to overridable add_arch_dtb_artifacts()
        • defaults to including dtbs for aarch64
        • but not for trixie-slim (to keep it, well, "slim")
  • introduce artifacts.py OutputArchArtifactType and OutputArchArtifact
    • just to type-enforce
  • _collect_arch_artifacts() is dead, long live _checksum_files()
  • rework logic in oci._publish()
  • this is to prepare for extra per-flavor assets, eg "rpi eeprom firmware" for rpi flavor
• 🌱 flavor: introduce BaseFlavor::post_mkosi_stage() and BaseFlavor::post_artifact_collect()
  • for custom stuff done after initramfs build / after artifact collection (mkosi.output/... -> out/)
• 🌳 trixie-armbian-rpi: implement ready-to-go RPi firmware in firmware-rpi dir
  • this is targetting (eeprom-based) RPi Netboot (without u-boot)
  • implement list_arch_artifacts(), post_mkosi_stage(), post_artifact_collect()
• 🍃 util: introduce symlink_relative()
  • with mixes Path from pathlib and os.path for non-subpaths
• 🍀 trixie-armbian-rpi: include kernel8.img and initramfs8 symlinks
  • refactor to use util.symlink_relative()
• 🌵 trixie-slim: bump kernel 6.18.22 -> 6.18.26 (copy.fail etc)
  • comment-change in mkosi.conf to force hash to change and thus all to rebuild
• 🐸 common-debian: enable Networking for scripts; fix postinst docs and use mkosi-chroot
  • set [Build].WithNetwork=yes so scripts can hit the network if needed
  • mkosi.postinst is not mkosi.postinst.chroot
    • See https://github.com/systemd/mkosi/blob/main/mkosi/resources/man/mkosi.1.md#scripts
• 🌴 release: rework into click Group; bring back pull and tag
  • publish is a release subcommand now
  • logging fixes, don't show locals, suppress Docker traces
  • TODO: BUILDAH_INSECURE etc
  • center on Config (release_ prefix), Docker repasses
  • drop git describe, always use v0.0.0-{sha}
• 🌿 gha: adapt Release to new group, don't fetch-depth:0 even for CI
• 🌱 captain: implement REGISTRY_INSECURE=1 env var for talking to insecure registries
  • nee BUILDAH_INSECURE, also used for skopeo
  • mostly for development vs registry:2
    • eg docker run -p 5001:5000 -it registry:2
    • and REGISTRY_INSECURE=1 uv run captain ... --registry 192.168.66.10:5001 publish
• 🌳 captain: simple end-user customization via custom folder
  • allows for
    • custom/packages.txt: extra packages to be installed
    • custom/mkosi-postinst and custom/mkosi-finalize folders: extra scripts
    • custom/files: extra files in root filesystem
  • when rendering templates, force newline at the end of each
  • pass custom dir down to Docker
• 🍃 custom: example customization possibilities (packages, scripts, extra files)

[rpardini]

See GHA runs for CI and Release workflows on my fork.