diff --git a/docs/rancher/cloud-provider.md b/docs/rancher/cloud-provider.md index 90830420b8..8141af7191 100644 --- a/docs/rancher/cloud-provider.md +++ b/docs/rancher/cloud-provider.md @@ -15,32 +15,469 @@ description: The Harvester cloud provider used by the guest cluster in Harvester -[RKE2](./node/rke2-cluster.md) clusters can be provisioned in Rancher using the built-in Harvester Node Driver. Harvester provides [load balancer](#load-balancer-support) and Harvester cluster [storage passthrough](./csi-driver.md) support to the guest Kubernetes cluster. +## Introduction + +You can provision [RKE2](./node/rke2-cluster.md) clusters in Rancher using the built-in **Harvester Node Driver**. This component provides [load balancer](#load-balancer-support) and [storage passthrough](./csi-driver.md) capabilities to guest Kubernetes clusters. In this page we will learn: -- How to deploy the Harvester cloud provider in an RKE2 cluster. +- The main functionality and key configuration parameters of the **Harvester Cloud Provider**. +- How to deploy the **Harvester Cloud Provider** in an RKE2 cluster. - How to use the [Harvester load balancer](#load-balancer-support). +### Main Functionalities + +The **Harvester Cloud Provider** (**HCP**, also known as **Cloud Controller Manager** or **CCM**) implements a subset of the `cloudprovider` interface defined by `k8s.io/cloud-provider`. + +- **[Node Instance Metadata Support](#node-instance-metadata-support)**: Dynamically discovers and reports node metadata (such as node names, regions, zones, and IP addresses), acting as a critical bootstrap component for the guest cluster. + +- **[Load Balancer Support](#load-balancer-support)**: Automatically provisions and configures load balancers for Kubernetes `Service` objects (type: `LoadBalancer`), routing external traffic to the correct guest nodes. + +:::important + +Harvester-cloud-provider operates in a strictly cloud-native manner by querying and managing resources directly through the Harvester Kubernetes API server: + +- **API-Driven Discovery & Deterministic Node Instance Metadata**: + It retrieves node instance metadata (such as `ProviderID`, region/zone topology, and IP addresses) by observing the `VirtualMachineInstance` (VMI) object status directly from the Harvester API server—rather than executing host-level commands (such as `ip addr`) or guessing and hard-coding specific network interfaces (NICs) or ports. Relying on the VMI status as the single source of truth guarantees consistent and deterministic metadata reporting across all nodes, providing a reliable foundation for downstream node lifecycle synchronization and load balancer traffic routing. + +- **Centralized Management**: + Running a defined number of Pods (as a Deployment rather than a per-node DaemonSet), it centrally queries and watches metadata for every node (VM) across the guest cluster via standard K8s API interactions. + +- **CCM Framework Alignment & Flag-Based Control**: + Built directly on top of the Kubernetes Cloud Controller Manager (CCM) framework, it uses standard public flags alongside extended Harvester flags (`--flag`) to modularly toggle controllers (such as standard CCM controllers or the custom VMI controller) and fine-tune networking behavior to match complex cluster setups. + +::: + ### Backward Compatibility Notice :::note -Please note a known backward compatibility issue if you're using the Harvester cloud provider version **v0.2.2** or higher. If your Harvester version is below **v1.2.0** and you intend to use newer RKE2 versions (i.e., >= `v1.26.6+rke2r1`, `v1.25.11+rke2r1`, `v1.24.15+rke2r1`), it is essential to upgrade your Harvester cluster to v1.2.0 or a higher version before proceeding with the upgrade of the guest Kubernetes cluster or Harvester cloud provider. - For a detailed support matrix, please refer to the **Harvester CCM & CSI Driver with RKE2 Releases** section of the official [website](https://www.suse.com/suse-harvester/support-matrix/all-supported-versions/). ::: ## Deploying +Harvester-cloud-provider is packaged as an official Helm chart and natively integrated into the Rancher & RKE2 ecosystem. + +- **Automatic Deployment (RKE2)**: When provisioning an RKE2 guest cluster on **Harvester Node Driver**, `harvester-cloud-provider` is **automatically** deployed into the guest cluster during cluster initialization as an **RKE2 bootstrap chart**. + +- **Manual / Custom Deployment**: You can also [manually install](#deploying-to-the-k3s-cluster-with-harvester-node-driver-experimental), upgrade, or customize the provider's configuration settings using the official Helm chart directly. + +### Chart Configuration Reference + +For a complete reference of all configurable options, refer to the chart's [`values.yaml`](https://github.com/harvester/charts/blob/release/charts/harvester-cloud-provider/values.yaml). + +The following table lists the most common parameters: + +| Parameter | Description | Default | First Available Version | +| --- | --- | --- | --- | +| `extraArgs` | Additional CLI flags passed to the cloud provider container | `[]` | 0.2.12 | +| `cloudConfigPath` | Legacy host path to the cloud-config file (default: `"/etc/kubernetes/cloud-config"` in 0.2.3–0.2.9) | `"/var/lib/rancher/rke2/etc/config-files/cloud-provider-config"` | 0.2.10 | +| `cloudConfig.secretName` | Kubernetes Secret name containing cloud-config data | `""` | 0.2.12 | +| `cloudConfig.secretKey` | Secret key containing the configuration data | `"cloud-config"` | 0.2.12 | +| `cloudConfig.hostPath` | Fallback host path for cloud-config | `"/var/lib/rancher/rke2/etc/config-files/cloud-provider-config"` | 0.2.12 | +| `global.cattle.clusterName` | Target cluster name in Rancher | `""` | 0.2.3 | +| `kube-vip.enabled` | Enable embedded `kube-vip` for LoadBalancer VIPs | `true` | 0.2.7 | +| `kube-vip.env.svc_enable` | Enable `kube-vip` Service controller | `"true"` | 0.2.7 | +| `kube-vip.env.svc_election` | Enable leader election for Services (required for `externalTrafficPolicy: Local`) | `"false"` | 0.2.12 | + + +#### Rancher UI Configuration Options + +The Rancher UI renders dynamic configuration forms driven by the chart's `questions.yaml` file. While the form exposes the most common options directly, you can click **Edit as YAML** of the Rancher UI form view to configure any additional parameters supported in `values.yaml`. + +Below are the default options rendered in the Rancher Manager UI for `harvester-cloud-provider` v0.2.12: + +![Rancher UI options for Harvester Cloud Provider](/img/v1.9/rancher/hcp-ui-options.png) + +### Detailed Configuration Guides + +#### 1. Cluster Identifier + +The `global.cattle.clusterName` parameter configures a unique cluster identifier used by `harvester-cloud-provider` to tag and track resources allocated in the underlying Harvester cluster. + +```yaml +global: + cattle: + clusterName: "cgc" +``` + +:::note + +- **Rancher Provisioned Clusters**: If you deploy an RKE2 guest cluster through the Rancher Manager UI, Rancher automatically injects `global.cattle.clusterName` with the cluster's unique name. + +- **Manual Chart Deployments**: If you deploy or manage the `harvester-cloud-provider` chart manually, you must explicitly set this field to a unique, unified name for your cluster. + +::: + +For **Rancher-provisioned RKE2 clusters**, Rancher automatically embeds the `harvester-cloud-provider` chart configuration within the `Cluster` custom resource (`provisioning.cattle.io/v1`). + +To customize these parameters, edit the `Cluster` resource directly in the Rancher UI or update it via **Edit as YAML**: + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +metadata: + name: cgc +spec: + rkeConfig: + chartValues: + harvester-cloud-provider: + cloudConfigPath: /var/lib/rancher/rke2/etc/config-files/cloud-provider-config + global: + cattle: + clusterName: cgc +``` + +![HCP configure nested in Rancher cluster object](/img/v1.9/rancher/hcp-chart-config-path.png) + +:::tip + +**YAML Structure & Indentation**: All HCP-related configuration parameters must be nested directly under the `harvester-cloud-provider`: key. Pay close attention to YAML indentation—incorrect indentation may cause Helm to ignore or misinterpret your chart parameters. + +When using **Edit as YAML** in the Rancher UI, the built-in editor automatically performs syntax checking and highlights indentation errors before you save. + +::: + + +:::warning + +**Resource Allocation & Leakage Risks** + +If `global.cattle.clusterName` is missing, the underlying Cloud Controller Manager framework defaults to using `kubernetes` as the cluster name. + +Because Harvester manages multi-tenant and multi-cluster environments, using the generic fallback name prevents Harvester from effectively identifying which guest cluster owns specific backing resources (such as Load Balancers). This can cause resource tracking conflicts, resource leaks, or unexpected resource exhaustion across clusters sharing the same Harvester installation. + +Starting with `HCP v0.2.12` and `Harvester v1.9.0`, both `harvester-cloud-provider` (running in the guest cluster) and `harvester-load-balancer` (running in the Harvester cluster) will emit warning logs whenever the cluster name is missing or defaults to `kubernetes`. If you observe these warning logs in either location, inspect your Helm chart parameters immediately and update them. + +::: + +##### Parameter Alignment & Terminology Mapping + +Due to legacy design evolution and the need to preserve backward compatibility across different layers, the **cluster identifier** is referenced using different parameter names depending on the context: + +| Context / Location | Parameter Term | Notes & Alignment | +| :--- | :--- | :--- | +| **Rancher UI** | `Cluster Name` | Set during guest cluster creation in the UI. | +| **HCP Helm Chart** | `global.cattle.clusterName` | Configured in `values.yaml`. Injected automatically by Rancher UI for RKE2 guest clusters; must be set manually in all other cases. | +| **HCP Deployment Flag** | `--cluster-name` | Internal container argument injected into the HCP deployment. The Helm chart template automatically converts `global.cattle.clusterName` to `--cluster-name`. When troubleshooting, check the Deployment manifest to verify this parameter is set correctly, but **do not edit it directly**. | +| **Cloud Config Generation** | `serviceAccountName` | Parameter specified when generating the `cloud-config` payload. | + +:::warning + +Critical Alignment Requirement: + +Although different components use different parameter names (`Cluster Name`, `global.cattle.clusterName`, `--cluster-name`, and `serviceAccountName`), **they all represent the exact same cluster identifier and must strictly match**. + +If these values do not match, `harvester-cloud-provider` will fail to authenticate or properly manage resources in Harvester. + +::: + +#### 2. Remote Harvester Cloud Configuration + +Harvester-cloud-provider requires a [cloud-config](#generate-the-cloud-config-for-harvester-cloud-provider) payload to connect to the remote Harvester cluster to manage VM metadata and LoadBalancers. You can configure this using either `legacy host-path mounts` or `Kubernetes Secrets`. + +##### Legacy File-Based Cloud-Config + +This approach relies on mounting the configuration file directly from the host node's filesystem. + + +- Default Path (`cloudConfigPath`): RKE2 automatically injects the cloud configuration into `/var/lib/rancher/rke2/etc/config-files/cloud-provider-config` for the cloud-provider container to access. + +- Configuration Syntax: + ```yaml + cloudConfigPath: "/var/lib/rancher/rke2/etc/config-files/cloud-provider-config" + ``` + +- Host Path Fallback (`cloudConfig.hostPath`): + + Maintained as a deprecation placeholder. This is only evaluated if `cloudConfig.secretName` is empty and `cloudConfigPath` is removed. + +##### New Secret-Based Cloud-Config (Recommended) + +__Available as of Harvester-cloud-provider v0.2.12__ + +This modern approach uses a Kubernetes Secret to store and manage the configuration payload natively inside the cluster. + +```yaml +cloudConfig: + secretName: "" + secretKey: "cloud-config" + hostPath: "/var/lib/rancher/rke2/etc/config-files/cloud-provider-config" +``` + +Configuration steps: + +1. **Generate and copy the cloud-config content** + + Generate the `cloud-config` by following [Generate the Cloud Config for Harvester Cloud Provider](#generate-the-cloud-config-for-harvester-cloud-provider). Copy the full configuration content string under the `content:` key (note that this value is already Base64-encoded). + + :::important + + The `namespace` and `serviceAccountName` used when generating the `cloud-config` must strictly match the guest cluster deployment settings: + + * **`namespace`**: Must be the exact namespace where the guest cluster is deployed. + * **`serviceAccountName`**: Must match the exact `cluster name` of the guest cluster. + + If these values do not match, Harvester Cloud Provider (HCP) will be unable to connect to Harvester. + + ::: + + ![Cloud Config Secret Content](/img/v1.9/rancher/hcp-config-secret-content.png) + +2. **Paste the Secret manifest during cluster provisioning** + + When provisioning the new guest cluster in the Rancher UI, paste the generated Secret definition into the **Additional Manifests** section under **Cluster Configuration**. + + ```yaml + apiVersion: v1 + kind: Secret + metadata: + name: hcp-cloud-config + namespace: kube-system + type: Opaque + data: + cloud-config: + ``` + + :::important + + **namespace**: The `harvester-cloud-provider` is deployed in `kube-system` by default, so this **Secret** must be created in the same namespace. + + **Secret Key (data.cloud-config)**: The default key name expected by the provider is `cloud-config`. If you choose a different key name in data, make sure to update the Cloud Config Secret Key field (`cloudConfig.secretKey`) in the Rancher UI to match your custom key. + + ::: + + ![Rancher UI Additional Manifests Field](/img/v1.9/rancher/hcp-config-additional-manifest.png) + +3. **Reference the Secret name in the Harvester Cloud Provider addon** + + Under **Addon: Harvester Cloud Provider**, enter the name of your created Secret into the **Cloud Config Secret Name** field. + + ![Rancher UI Cloud Config Secret Name Field](/img/v1.9/rancher/hcp-config-sc-name.png) + + +:::important + +In SELinux-enabled clusters, container runtimes enforce strict security context labeling. The `harvester-cloud-provider` container is blocked from mounting or reading local host files directly under `/var/lib/rancher/rke2/...` because host path access triggers SELinux permission violations (`EACCES: Permission denied`). + +Using a **Kubernetes Secret** eliminates host filesystem dependency, allowing the pod to securely mount the configuration via standard Kubernetes volume mechanisms without SELinux denial issues. + +::: + +Configuration Precedence + +1. `cloudConfig.secretName`: Highest Priority. + When explicitly set, it mounts a Kubernetes Secret and overrides all other settings. + - `cloudConfig.secretKey`: The key inside the Secret containing the config payload (defaults to "cloud-config"). + +1. `cloudConfigPath`: Legacy Default. + Maintained for backward compatibility. Evaluated if `secretName` is empty. + +1. `cloudConfig.hostPath`: (Fallback). + Only used as a fallback when `secretName` is empty and `cloudConfigPath` is explicitly set to "". + + +#### 3. Extra Arguments + +`harvester-cloud-provider` implements features on top of the [Kubernetes Cloud Controller Manager](https://kubernetes.io/docs/concepts/architecture/cloud-controller/) (CCM) framework. To cooperate with the CCM framework, it supports standard upstream public flags as well as Harvester-specific extended runtime flags. The CCM framework supports a wide variety of flags to flexibly tune controller features, networking logic, and system behavior. + +__Available as of Harvester-cloud-provider v0.2.12__ + +:::note + +* **Prior to v0.2.12**: Passing custom flags via `extraArgs` directly on the `harvester-cloud-provider` Deployment or Pod spec was non-persistent and would be overwritten during guest cluster upgrades or redeployments. + +* **Starting in v0.2.12**: Chart-level flag configurations are supported, ensuring all custom arguments persist across cluster upgrades and redeployments. + +::: + +Supported flags overview: + +1. **Framework & Standard CCM Controllers Flags**: + + - `--controllers`: List of controllers to enable (e.g., cloud-node-controller, node-route-controller). Omit to disable. + + - `--v`: Logging verbosity level (e.g., --v=5 for debug logging). + +2. **Harvester Extended Controllers and Flags**: + + - `--disable-vmi-controller`: Set to `true` to disable Harvester's custom VMI controller. + - **Context**: The standard CCM framework `--controllers` flag only toggles upstream CCM controllers. Harvester introduces a dedicated VMI controller to continuously watch Harvester `VirtualMachineInstance` (VMI) resources and dynamically sync topology changes (such as region and zone updates) back to guest `Node` objects. + - **Recommendation**: Leave this enabled (`false`) to ensure dynamic topology sync works seamlessly as detailed in [Node Instance Metadata Support](#node-instance-metadata-support). + + - `--show-full-help-on-error`: Set to true to print full CLI help if a flag parsing error occurs at startup. + - **Context**: By default, the upstream CCM framework dumps several hundred lines of help text on any CLI parsing error, making startup logs noisy and hard to read. `harvester-cloud-provider` silences this verbose help by default; setting this flag to `true` restores the upstream behavior. + +3. **Harvester Networking & IP Selection Flags**: + + - `--management-network`: Defines the [target VM network name](../networking/harvester-network.md#create-a-vm-network) for network selection in multi-network environments where guest cluster nodes (Harvester VMs) are attached to multiple VM networks. When set, the provider will allocate LoadBalancer IPs and report node IP addresses specifically from this network (bypassing the default "first-hit" network selection logic). + + - `--node-ip-cidr`: CIDR range for exact IP selection in multi-IP environments. + + - `--node-exclude-ip-ranges`: Comma-separated blacklist of IPs/subnets to exclude from node status reports. + + - `--disable-annotation-alpha-provided-ip-addr`: Set to true to disable legacy alpha annotations and rely solely on CIDR logic. + +:::note + +**Minimum Harvester version requirement**: The target Harvester cluster must be running **v1.9.0 or higher** to support the `--management-network` flag for LoadBalancer services, designating it as the load balancer target network. Earlier Harvester versions fall back to `first-fit` resolution to select the target network. For details, see [Guest Cluster Load Balancer Network Resolution](../networking/ippool.md#guest-cluster-load-balancer-network-resolution). + +::: + + +The following examples demonstrate how to configure `extraArgs` for various deployment scenarios: + +**Example 1: Disable LoadBalancer Controller** + +Use this configuration if you deploy an alternative load balancer controller and need to disable the default LoadBalancer controller provided by `harvester-cloud-provider`: + +```yaml +extraArgs: + - "--controllers=cloud-node-controller,cloud-node-lifecycle-controller,node-route-controller" +``` + +:::info + +Disabling embedded `kube-vip`: + +Because `kube-vip` is embedded within `harvester-cloud-provider` to advertise LoadBalancer IP addresses, disabling the upstream service controller typically means you should also disable kube-vip in your Helm values: + +If you plan to keep `kube-vip` enabled while using an alternative load balancer controller, that third-party controller must be capable of cooperating with `kube-vip`. Integrating third-party load balancer controllers with kube-vip is outside the scope of `harvester-cloud-provider`. + +::: + +**Example 2: Multi-Network / Multi-IP Configuration with Exclusions (Recommended)** + +When guest cluster nodes are booted with multiple networks, dual-stack IP addresses, or multiple IPv4 addresses on a single interface, default `first-hit` selection logic can cause non-deterministic IP reporting. + +The following scenarios demonstrate how to use `extraArgs` to handle complex networking setups: + +**Example 2.1 : Multi-Network Environments (Network Selection)** + +When cluster nodes are attached to multiple Harvester VM networks (e.g., `default/vlan-100` and `default/vlan-200`), `harvester-cloud-provider` might randomly select an interface. Setting `--management-network` forces the provider to only report node IPs from the specified network (e.g., `default/vlan-100`). Additionally, the provider will allocate LoadBalancer IPs from this designated network. + +```yaml +extraArgs: + - "--management-network=default/vlan-100" +``` + +**Example 2.2: Dual-Stack Interfaces in Single-Stack (IPv4-Only) Clusters** + +When cluster nodes receive both IPv4 and IPv6 addresses, harvester-cloud-provider may default to reporting both of the IP addresses as the node's `InternalIP`. Setting `--node-ip-cidr` instructs the provider to select the matching IPv4 address as the primary `InternalIP` (relegating the IPv6 address to `ExternalIP`). + +```yaml +extraArgs: + - "--node-ip-cidr=192.168.1.0/24" +``` + +**Example 2.3: Excluding Secondary IP Ranges on the Same Network (e.g., Split Subnets)** + +When cluster nodes have multiple IPv4 addresses configured on the same management network interface (`default/vlan-100`), such as `192.168.100.0/25` for node management and `192.168.100.128/25` reserved strictly for cluster-internal usage (like internal traffic or storage), the provider defaults to assigning the first IPv4 as `InternalIP` and automatically publishing the second IPv4 as `ExternalIP`. + +To enforce your network design and prevent internal secondary IPs on the same interface from leaking into Kubernetes node status as `ExternalIP`, combine all three flags: + +1. `--management-network`: Selects the target network interface. + +1. `--node-ip-cidr`: Locks the primary InternalIP selection to the node management subnet range. + +1. `--node-exclude-ip-ranges`: Excludes the secondary IP range from being reported as `ExternalIP`. + + +```yaml +extraArgs: + - "--management-network=default/vlan-100" + - "--node-ip-cidr=192.168.100.0/25" + - "--node-exclude-ip-ranges=192.168.100.128/25" +``` + +Combined Production Example: + +Combining these flags ensures strict, predictable InternalIP selection and prevents secondary networks or split subnets from leaking into ExternalIP status: + +```yaml +extraArgs: + - "--management-network=default/vlan-100" + - "--node-ip-cidr=192.168.100.0/25" + - "--node-exclude-ip-ranges=192.168.100.128/25" + - "--disable-annotation-alpha-provided-ip-addr=true" + - "--show-full-help-on-error=true" +``` + +:::note + +A Known Limitation: Rancher Manager UI IP Synchronization: + +`harvester-cloud-provider` correctly applies these network flags and updates the Kubernetes Node object status (`InternalIP / ExternalIP`). However, Rancher Manager UI does not dynamically re-sync node IP changes in its UI if they are updated after initial node registration (for more details, see [Harvester Issue 10381](https://github.com/harvester/harvester/issues/10381#issuecomment-5264412173)). + +- **Example Scenario**: On an IPv4-only cluster where nodes initially report both IPv4 and IPv6 addresses as InternalIP, specifying --node-ip-cidr enables harvester-cloud-provider to successfully filter the Kubernetes Node status down to IPv4 only. However, the Rancher UI may continue displaying the unsynced IP information in its dashboard. + +- **Workaround**: Set these network parameters in `extraArgs` during initial cluster bootstrapping. If applying these flags to an existing cluster, a cluster redeployment is required for Rancher UI to reflect the updated node metadata. + +::: + +#### 4. Embedded Kube-vip Integration + +`harvester-cloud-provider` integrates with `kube-vip` to provision and manage Virtual IPs for Kubernetes `LoadBalancer` services. + +##### Disabling embedded kube-vip + +If you want `harvester-cloud-provider` to retain its LoadBalancer IP allocation and management logic (such as pool-based IP assignment), but prefer using another LoadBalancer tool or external BGP/ARP speaker to handle VIP traffic routing, you can disable the embedded `kube-vip` sub-chart: + +```yaml +kube-vip: + enabled: false +``` + +##### Support service `externalTrafficPolicy: Local` + +By default, kube-vip runs exclusively on control-plane nodes. To support `externalTrafficPolicy: Local` for LoadBalancer services, traffic must be routed directly to nodes hosting workload pods. This requires two configuration changes: + +1. Enable service leader election by setting `svc_election: "true"`. + +1. Expand `kube-vip.affinity` rules so kube-vip pods run on worker nodes in addition to control-plane nodes. + +```yaml +kube-vip: + env: + svc_election: "true" + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/control-plane + operator: Exists + - matchExpressions: + - key: node-role.kubernetes.io/worker + operator: Exists +``` + +:::important + +Best Practice: Ensure the node coverage for `externalTrafficPolicy: Local`. + +Deploying `kube-vip` across all nodes in the guest cluster is strongly recommended when using `externalTrafficPolicy: Local`. + +Because `kube-vip` requires a co-located workload pod on the same node to elect a leader and advertise the VIP: + +- **No common nodes**: If `kube-vip` runs only on `control-plane` nodes while workload pods run exclusively on `worker` nodes, `kube-vip` cannot elect a leader and will not advertise the VIP for that service. + +- **Partial overlap**: If `kube-vip` runs on a subset of worker nodes (e.g., `node1`, `node2`) while workload pods run on `node2` and `node3`, only the overlapping node (`node2`) can be elected to advertise the VIP. + +In both cases, the `high availability (HA)` of the LoadBalancer service is significantly compromised or lost entirely. + +::: + ### Prerequisites -- The Kubernetes cluster is built on top of Harvester virtual machines. + +- The guest Kubernetes cluster is built on top of Harvester virtual machines. - The Harvester virtual machines run as guest Kubernetes nodes are in the same namespace. :::info important -Each Harvester VM must have the `macvlan` kernel module, which is required for the `LoadBalancer` services of the **DHCP** IPAM mode. +If `harvester-cloud-provider` `kube-vip` is not disabled explicitly, each Harvester VM must have the `macvlan` kernel module, which is required for the `LoadBalancer` services of the **DHCP** IPAM mode. To check if the kernel module is available, access the VM and run the following commands: @@ -185,7 +622,7 @@ When spinning up a K3s cluster using the Harvester node driver, you can perform bootstrap: true repo: https://charts.harvesterhci.io/ chart: harvester-cloud-provider - version: 0.2.2 + version: 0.2.12 helmVersion: v3 ``` @@ -239,8 +676,8 @@ You can send `POST` and `GET` requests to the Harvester API endpoint `/v1/harves | Parameter | Type | Description | Example | | :--- | :--- | :--- | :--- | -| `namespace` | String | Target Kubernetes namespace | `gc-test` | -| `serviceAccountName` | String | Service account name | `gc4` | +| `namespace` | String | Target namespace in Harvester where the guest cluster is deployed. | `gc-test` | +| `serviceAccountName` | String | Name of the ServiceAccount created for `cloud-config` generation. **Must strictly match the guest cluster name.** | `gc4` | | `clusterRoleName` | String | ClusterRole to bind to the service account (optional) | `harvesterhci.io:cloudprovider` (only supported value) | | `outputFormat` | String | Output format | `yaml` (only supported value) | @@ -417,6 +854,42 @@ For more information, see [this GitHub issue comment](https://github.com/harvest ::: +## Node Instance Metadata Support + +When registering and updating guest nodes, `harvester-cloud-provider` queries the Harvester API server to inspect the underlying `VirtualMachine` (VM) and `VirtualMachineInstance` (VMI) objects. It constructs the standard `cloudprovider.InstanceMetadata` struct by targeting three key metadata elements from the VMI: + +### 1. Provider Identifier (`ProviderID`) + +Sets a globally unique identifier for the guest node based on the underlying Harvester VM's UID: + +* **Format**: `harvester://` +* **Purpose**: Allows Kubernetes to deterministically map the guest `Node` object back to its physical Harvester VM resource. + +### 2. Topology Metadata (`Region` & `Zone`) + +Reads topology annotations set on the `VMI` object to establish placement context for Kubernetes scheduling: + +* **Region**: Extracted from the `topology.kubernetes.io/region` annotation. +* **Zone**: Extracted from the `topology.kubernetes.io/zone` annotation. +* **Fallback**: If the VMI lacks topology annotations, `ProviderID` is still reported while region/zone fields remain unset. + +### 3. Node Addresses (`NodeAddresses`) + +Constructs the complete address list (`[]v1.NodeAddress`) for the guest node by combining hostname mapping with VMI status discovery: + +- **Host Name (`NodeHostName`)**: + Sets `v1.NodeHostName` directly using the target `node.Name`. + +- **IP Discovery (`InternalIP` & `ExternalIP`)**: + Evaluates active network interface IPs reported directly in the VMI status alongside provider configuration—eliminating the need for host-level probes or hardcoded NIC assumptions. + +- **Deterministic Mapping**: + Assigns detected IPs to `InternalIP` and `ExternalIP` types, providing a consistent source of truth for downstream intra-cluster networking and load balancer traffic routing. + +:::note +To customize or filter which VM network interfaces and IP ranges are used when reporting node addresses, see [#### 3. Extra Arguments (extraArgs)](#3-extra-arguments). +::: + ## Load Balancer Support Once you've deployed the Harvester cloud provider, you can leverage the Kubernetes `LoadBalancer` service to expose a microservice within the guest cluster to the external world. Creating a Kubernetes `LoadBalancer` service assigns a dedicated Harvester load balancer to the service, and you can make adjustments through the `Add-on Config` within the Rancher UI. @@ -446,6 +919,12 @@ Harvester's built-in load balancer offers both **DHCP** and **Pool** modes, and - `cloudprovider.harvesterhci.io/ipam: "ippool"` - `cloudprovider.harvesterhci.io/network: "default/mgmt-vlan1"` + :::note + + The remote Harvester version needs to be v1.9.0 or higher to support the customized `cloudprovider.harvesterhci.io/network` usage on Harvester-cloud-provider. + + ::: + When a guest cluster uses multiple networks, or when multiple guest clusters with distinct networks share a single namespace, configuring the correct network parameters is critical. For details on how the system automatically determines the network, refer to [Guest Cluster Load Balancer Network Resolution](../networking/ippool.md#guest-cluster-load-balancer-network-resolution). ![](../../static/img/v1.9/rancher/guest-cluster-load-balancer-pool.png) diff --git a/static/img/v1.9/rancher/hcp-chart-config-path.png b/static/img/v1.9/rancher/hcp-chart-config-path.png new file mode 100644 index 0000000000..e1a6bbc9ae Binary files /dev/null and b/static/img/v1.9/rancher/hcp-chart-config-path.png differ diff --git a/static/img/v1.9/rancher/hcp-config-additional-manifest.png b/static/img/v1.9/rancher/hcp-config-additional-manifest.png new file mode 100644 index 0000000000..438bc2f955 Binary files /dev/null and b/static/img/v1.9/rancher/hcp-config-additional-manifest.png differ diff --git a/static/img/v1.9/rancher/hcp-config-sc-name.png b/static/img/v1.9/rancher/hcp-config-sc-name.png new file mode 100644 index 0000000000..d2267515a1 Binary files /dev/null and b/static/img/v1.9/rancher/hcp-config-sc-name.png differ diff --git a/static/img/v1.9/rancher/hcp-config-secret-content.png b/static/img/v1.9/rancher/hcp-config-secret-content.png new file mode 100644 index 0000000000..3a41973d1e Binary files /dev/null and b/static/img/v1.9/rancher/hcp-config-secret-content.png differ diff --git a/static/img/v1.9/rancher/hcp-ui-options.png b/static/img/v1.9/rancher/hcp-ui-options.png new file mode 100644 index 0000000000..cdc7e12eb5 Binary files /dev/null and b/static/img/v1.9/rancher/hcp-ui-options.png differ