Multi-Line SSH Keys in `config.toml` Works for `--type qcow2` but Crash when `--type bootc-installer`

[i448]

Description:

When using a multi-line TOML string (""") to define multiple SSH keys for a user in config.toml, the build succeeds for --type qcow2 but creates a broken installer for --type bootc-installer.

Discovered this while trying to add a user called podman, there are 2 head administrators users so I added 2 SSH keys. I got a error from kickstart in --type bootc-installer version while the generated --type qcow2 works fine.

The resulting ISO boots, but the Anaconda installer crashes during the "starting automated install" phase. This is because the multi-line TOML string is being translated into a multi-line entry in the Kickstart file (ks.cfg), which is invalid syntax for the sshkey command.

Here's an attached image from me debugging the error in a VM:

Here's the image where it couldn't find ks.cfg as expected:

Environment:

• Host OS (Container Building): Fedora 44 Silverblue
• Source OCI Image: quay.io/fedora/fedora-bootc:44

Reproduction Steps:

1. Create a config.toml with a multi-line SSH key block:

   [[customizations.user]]
   name = "podman"
   key = """
   ssh-ed25519 AAA... johndoe@coldmail.com
   ssh-ed25519 AAA... janedoe@coldmail.com
   """
   

2. Build a QCOW2 image: --type qcow2. (Success: System boots and keys are injected correctly).

3. Build a Bootc-Installer image: --type bootc-installer --installer-payload-ref <your-ref-here>. Side Note: Why documentation say --bootc-installer-payload-ref?

4. Boot the resulting ISO in QEMU.

Expected Result:

The installer should parse the multiple keys correctly (either by flattening them into a single line or generating multiple sshkey entries) and complete the installation.

Actual Result:

The installer crashes with a Python traceback from shlex:

ValueError: No closing quotation

Followed by:

Kickstart file /run/install/ks.cfg is missing.

This indicates that the ks.cfg generated by bootc-image-builder contains an unclosed quote because the SSH key string spans multiple lines.

Image from before:

Here's what is suppose to happen:

Additional Context:

The issue seems to be in how the TOML customizations are serialized into the Kickstart format. While the osbuild stages for disk images handle multi-line strings correctly, the Kickstart generator for the installer needs to sanitize or flatten these strings to ensure they remain single-line commands. As seen above.

Here's the source I used for building the Container:

Containerfile

FROM quay.io/fedora/fedora-bootc:44


Enable faster downloads.
RUN mkdir /etc/dnf/dnf.conf && \
    echo -e "fastestmirror=True\nmax_parallel_downloads=10" >> /etc/dnf/dnf.conf


# Install Anaconda for initializing the system upon first boot.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf install -y \
        anaconda \
        anaconda-install-env-deps \
        anaconda-dracut \
        dracut-config-generic \
        dracut-network \
        net-tools \
        squashfs-tools \
        grub2-efi-x64-cdboot \
        python3-mako \
        lorax-templates-* \
        biosdevname \
        prefixdevname


# Fedora EFI.
RUN mkdir -p /boot/efi && cp -ra /usr/lib/efi/*/*/EFI /boot/efi


# For Larax ISO builder.
RUN mkdir -p /var/mnt


# Installs wifi drivers.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf -y install \
    linux-firmware \
    NetworkManager-wifi

RUN systemctl enable NetworkManager


# Add `machinectl` for Quadlets.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf -y install systemd-container


# Set up SSH Server to only accept SSH key logins.
ADD config/override_pubkey_authentication.conf /etc/ssh/ssh_config.d/override_pubkey_authentication.conf


# Change shell to Nushell.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf -y install 'dnf5-command(copr)' && \
    dnf -y copr enable atim/nushell && \
    dnf -y install nushell

RUN echo "/usr/bin/nu" >> /etc/shells
ADD config/useradd /etc/default/useradd


# Install BLAKE 3.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf -y install b3sum


# Install Zoxide.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf -y install zoxide

RUN mkdir -p /usr/share/nushell/vendor/autoload && \
    zoxide init nushell > /usr/share/nushell/vendor/autoload/zoxide.nu


# Add Terra Repository.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf -y install --nogpgcheck --repofrompath 'terra,https://repos.fyralabs.com/terra$releasever' terra-release && \
    curl -fsSL https://github.com/terrapkg/subatomic-repos/raw/main/terra.repo | tee /etc/yum.repos.d/terra.repo

RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf -y install television


# Enable Free and Non-Free Repositories.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf -y install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm && \
    dnf -y install https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm


# Add p7zip.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf -y install p7zip


# BUG: Atuin haven't updated Nushell integration! Code will fail in runtime!
# # Add Atuin.
# RUN --mount=type=cache,target=/var/cache/libdnf5 \
#     dnf -y install atuin
# RUN mkdir -p /usr/share/nushell/vendor/autoload && \
#     HOME=/tmp atuin init nu --disable-up-arrow > /usr/share/nushell/vendor/autoload/atuin.nu


# Override `KillUserProcesses` to `no`.
ADD config/override_kill_user_processes.conf /etc/systemd/logind.conf.d/override_kill_user_processes.conf


# Add network configuration.
ADD config/FREEWIFI.nmconnection /etc/NetworkManager/system-connections/FREEWIFI.nmconnection


# Add Carapace to Nushell and enble bridges for other shells.
ADD config/fury.repo /etc/yum.repos.d/fury.repo

RUN --mount=type=cache,target=/var/cache/libdnf5 \
    dnf -y install carapace

RUN mkdir -p /usr/share/nushell/vendor/autoload && \
    carapace _carapace nushell > /usr/share/nushell/vendor/autoload/carapace.nu

RUN mkdir -p /usr/share/nushell/vendor/autoload/env && \
    echo '$env.CARAPACE_BRIDGES = "zsh,fish,bash,inshellisense"' \
        > /usr/share/nushell/vendor/autoload/env/carapace_bridges.nu


# Install `sbctl`.
RUN --mount=type=cache,target=/var/cache/libdnf5 \
    sudo dnf -y copr enable chenxiaolong/sbctl && \
    sudo dnf -y install sbctl


# Updates packages.
RUN dnf -y update

config.toml

[customizations.kernel]
append = "consoleblank=60"

[customizations.timezone]
timezone = "Asia/Kuching"
ntpservers = ["time.cloudflare.com", "time.google.com"]

[[customizations.user]]
name = "ben"
password = "$6$..."
key = "ssh-ed2559 ..."
groups = ["wheel"]

[[customizations.user]]
name = "kp"
password = "$6$..."
key = "ssh-ed25519 ..."
groups = ["wheel"]

[[customizations.user]]
name = "podman"
password = ""
key = """
ssh-ed25519 ...
ssh-ed25519 ...
"""
groups = []

[i448]

Also doesn't work at the new image-builder-cli. Should I make a new issue over there too?

[supakeen]

No, just the issue here is fine @i448 we'll probably migrate the issues at some point and I /might/ move this issue over to images which is the underlying library for both tools 🙂