Skip to content

architecture.md

Latest commit

 

History

History
280 lines (196 loc) · 13.2 KB

architecture.md

File metadata and controls

280 lines (196 loc) · 13.2 KB
Raw

MicroShift Architecture

This document describes how MicroShift is built and how its components interact at runtime. For the design goals, principles, and reasoning behind architectural decisions, see design.md.

Repository Layout

Directory Purpose
cmd/microshift/ Main binary entry point (Cobra CLI)
etcd/ Etcd subprocess manager (separate binary)
pkg/ Core library packages (see Package Structure)
assets/ Kubernetes manifests for core components
assets/optional/ Manifests for optional components (OLM, Gateway API, cert-manager, SR-IOV, etc.)
packaging/ RPM spec, systemd units, CRI-O config, SELinux policy, Containerfiles
test/ E2E test framework (Robot Framework), scenarios, image blueprints
scripts/ Build tooling, verification, rebase automation
docs/ User and contributor documentation
vendor/ Standard Go vendor directory for third-party dependencies (managed by make vendor)
deps/ Vendored copies of openshift/kubernetes and k8s.io/klog with rebase patches applied, referenced via go.mod replace directives. Exists so that vendor checks in ART (the release build system) pass cleanly

Package Structure

Control Plane Lifecycle

Package Purpose
pkg/cmd/ Cobra CLI commands: run, version, show-config, backup, restore, healthcheck, c2cc-probe
pkg/controllers/ Kubernetes control plane service wrappers — etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet, OpenShift controllers
pkg/servicemanager/ Dependency-aware service lifecycle manager with startup recording
pkg/admin/ Backup/restore, data management, pre-run checks (version metadata, feature gates, health verification)

Post-Startup Components

Package Purpose
pkg/components/ Infrastructure services deployed after control plane is ready: DNS, ingress router, storage (CSI), service CA, CNI, Multus CNI
pkg/kustomize/ Kustomization engine for manifest deployment from configured paths

Configuration and Runtime

Package Purpose
pkg/config/ Configuration parsing and validation — maps config.yaml to Go structs covering apiserver, etcd, DNS, ingress, networking, storage, C2CC, and other settings
pkg/release/ Release version tracking and component image references (auto-generated by rebase)
pkg/version/ Version metadata (major/minor/patch, build timestamp, variant)
pkg/node/ Node initialization and kubelet configuration
pkg/healthcheck/ Cluster health verification, including optional workload awareness

Supporting Packages

Package Purpose
pkg/mdns/ mDNS service discovery for edge networking
pkg/loadbalancerservice/ LoadBalancer service type support
pkg/gdp/ Graceful Degraded Performance handling
pkg/sysconfwatch/ System configuration file watcher
pkg/telemetry/ OpenShift telemetry integration (enterprise variant only)
pkg/apis/ MicroShift-specific API types (CRDs)
pkg/generated/ Generated clientsets, informers, and listers for MicroShift APIs
pkg/util/ Utilities — crypto/certificates, signal handling, file operations

Startup Sequence

MicroShift runs as a single systemd unit (microshift.service). The main binary executes RunMicroshift(cfg) which follows this sequence:

1. Pre-Flight Checks

  • Verify root privileges
  • Load and validate configuration from /etc/microshift/config.yaml
  • Set logging verbosity
  • Manage data backups (pre-run)

2. Environment Setup

  • Configure NO_PROXY environment
  • Create data directories under /var/lib/microshift/
  • Process version metadata and feature gates
  • Generate or rotate cryptographic materials (certificates, keys)

3. Service Startup

The ServiceManager starts embedded services in dependency order. Each service registers:

  • Name() — service identifier
  • Dependencies() — list of services that must start first
  • Run() — the goroutine that runs the service

Services start concurrently, respecting the dependency graph. The core services are:

  1. etcd (separate process, managed by main binary)
  2. kube-apiserver
  3. kube-controller-manager
  4. kube-scheduler
  5. kubelet
  6. OpenShift controllers (route, policy, etc.)

4. Component Deployment

After the control plane is ready, StartComponents() deploys infrastructure services in this order:

  1. Service CA
  2. CSI snapshot controller (if enabled in config)
  3. CSI plugin (LVMS)
  4. Ingress router (openshift-ingress)
  5. DNS controller (openshift-dns)
  6. CNI plugin (OVN-Kubernetes)
  7. Multus CNI (if enabled in config)
  8. Certificate authority (multinode mode only — unsupported, testing only)

5. Graceful Shutdown

Signal handling (SIGTERM, SIGINT) triggers ordered service termination with a 15-second timeout.

Deployment Modes

MicroShift supports two deployment models:

  • ostree (RPM-based): MicroShift is installed as RPM packages on an rpm-ostree system (RHEL for Edge). Updates are delivered as new ostree commits that atomically replace the entire OS and application layer.
  • bootc (image mode): MicroShift is embedded in a bootable container image. The entire system — OS, MicroShift, and configuration — is delivered as a single OCI image. Updates replace the image atomically.

Both models support atomic rollback. The choice between them affects packaging, delivery, and the CI test infrastructure (see CONTRIBUTING.md).

Greenboot Integration

MicroShift integrates with greenboot for automatic health checking after updates. On boot, greenboot runs health check scripts. If MicroShift's health check fails (control plane not ready within timeout), greenboot triggers an automatic rollback to the previous ostree commit or bootc image.

The health check script verifies that MicroShift's core workloads and any installed optional workloads are running. See pkg/healthcheck/ for the implementation.

For testing details, see greenboot testing.

Backup and Restore

MicroShift supports backup and restore of its data directory via microshift backup and microshift restore CLI commands. Backups capture etcd state, certificates, and cluster metadata. This is implemented in pkg/admin/ and runs as a pre-flight step before updates to enable recovery.

For user-facing documentation, see auto-recovery.

Data Directory

MicroShift stores all runtime state under /var/lib/microshift/:

  • certs/ — TLS certificates and keys
  • etcd/ — etcd database
  • kubelet-plugins/ — kubelet plugin data
  • resources/ — cluster resource state (kubeconfigs, API server secrets, scheduler/kubelet configs)
  • version — data version metadata

Backups are stored separately at /var/lib/microshift-backups/.

This directory must persist across reboots. Its contents are version-dependent — MicroShift validates the data version on startup and refuses to start if there is a version mismatch (handled by pkg/admin/prerun/).

Optional Packages

The base microshift RPM requires three subpackages: microshift-selinux (SELinux policy), microshift-networking (OVN-Kubernetes + Open vSwitch), and microshift-greenboot (health check scripts). These are not optional.

Additional functionality is delivered as optional RPM subpackages:

Available Optional Components

RPM Package Delivery Mechanism Functionality
microshift-olm manifests.d/001-microshift-olm Operator Lifecycle Manager
microshift-gateway-api manifests.d/000-microshift-gateway-api Gateway API (Service Mesh)
microshift-cert-manager manifests.d/060-microshift-cert-manager Certificate Manager
microshift-sriov manifests.d/070-microshift-sriov SR-IOV Network Operator
microshift-observability manifests.d/003-microshift-observability + systemd service OpenTelemetry Collector
microshift-ai-model-serving manifests.d/010-*-kserve, 050-*-runtimes AI Model Serving (KServe)
microshift-multus config file in /etc/microshift/config.d/ + CRI-O config Multus CNI (enables via config, not manifests)
microshift-low-latency tuned profile Baseline low-latency workload configuration

Most optional components also have a -release-info RPM containing container image references used by image builder blueprints.

Discovery Mechanism

Optional components are discovered via the manifest path system configured in config.yaml:

manifests:
  kustomizePaths:
    - /usr/lib/microshift/manifests
    - /usr/lib/microshift/manifests.d/*
    - /etc/microshift/manifests
    - /etc/microshift/manifests.d/*

The glob pattern /usr/lib/microshift/manifests.d/* automatically picks up any installed optional RPM. Installing the RPM is sufficient to enable the component; no configuration change is needed.

Config-Based Control

Some components have explicit enable/disable settings in config.yaml:

  • Multus CNI (pkg/config/network.go): .network.multus.status — Enabled or Disabled (default: Disabled)
  • CSI Snapshot Controller (pkg/config/storage.go): .storage.optionalCsiComponents — list of optional CSI components to deploy

Health Check Integration

The health check system (pkg/healthcheck/microshift_optional_workloads.go) is optional-aware. It checks whether each optional component's directory exists on disk AND is included in the configured kustomization paths before expecting its workloads to be running.

Asset Management

Core Assets

Kubernetes manifests for built-in components live in assets/, organized by function:

  • assets/core/ — core cluster resources (namespaces, kubelet config, API service registrations, priority classes)
  • assets/components/ — infrastructure service manifests (DNS, router, CSI, service CA, OVN, Multus, C2CC, CSI snapshot controller)
  • assets/controllers/ — control plane controller manifests
  • assets/crd/ — custom resource definitions
  • assets/release/ — release metadata
  • assets/version/ — version tracking
  • assets/embed.go — Go embed directives to bundle assets into the binary

These are embedded in the binary at build time and deployed via pkg/components/ and pkg/kustomize/.

Optional Assets

Manifests for optional components live in assets/optional/, organized by feature:

  • assets/optional/operator-lifecycle-manager/
  • assets/optional/gateway-api/
  • assets/optional/cert-manager/
  • assets/optional/sriov/
  • assets/optional/observability/
  • assets/optional/ai-model-serving/

These are packaged into the corresponding optional RPMs and installed to /usr/lib/microshift/manifests.d/.

Rebase-Managed Files

The following are maintained by the rebase automation (scripts/auto-rebase/). See rebase overview for details on the process.

Files Updated by
assets/ manifests rebase.sh manifests
pkg/release/release_*.go (image refs) rebase.sh images
Makefile.version.*.var, Makefile.kube_git.var rebase.sh buildfiles
vendor/, deps/ rebase.sh go.mod + make vendor

Configuration Model

MicroShift is configured through /etc/microshift/config.yaml. The configuration is strictly declarative and uses minimal options with sensible defaults.

The config file maps to Go structs in pkg/config/:

Config Section Go Type File
.apiServer ApiServer pkg/config/apiserver.go
.dns DNS pkg/config/dns.go
.etcd EtcdConfig pkg/config/etcd.go
.ingress IngressConfig pkg/config/ingress.go
.manifests Manifests pkg/config/manifests.go
.network Network pkg/config/network.go
.node Node pkg/config/node.go
.storage Storage pkg/config/storage.go
.debugging Debugging pkg/config/debugging.go
.telemetry Telemetry pkg/config/telemetry.go
.kubelet map[string]any pkg/config/config.go
.genericDevicePlugin GenericDevicePlugin pkg/config/gdp.go
.clusterToCluster C2CC pkg/config/c2cc.go

The default configuration is maintained in packaging/microshift/config.yaml. Changes to config structs must be reflected in the default config file, and vice versa.

Build System

MicroShift uses Makefile-based builds with OpenShift build machinery (openshift/build-machinery-go).

Key binaries:

  • microshift — main binary (all control plane + worker functionality)
  • microshift-etcd — etcd subprocess
  • generate-config — configuration generation utility

The build supports two variants controlled by build tags:

  • enterprise (default) — includes OpenShift telemetry
  • community — no telemetry, suitable for OKD builds

Version information is tracked in:

  • Makefile.version.x86_64.var / Makefile.version.aarch64.var — OCP version per architecture
  • Makefile.kube_git.var — Kubernetes git version metadata