fix: add talos support by hydazz · Pull Request #695 · kubernetes-sigs/dra-driver-nvidia-gpu

hydazz · 2025-10-19T00:12:32Z

This is a starter PR to add support for Talos OS's different nvidia paths.

Tested the gpu component with the changes here in my environment and it works.

Feedback is needed, as i'm unsure how to add the usr/local/glibc path to CDI nicely, I don't believe getTalosLibrarySearchPaths will cut it globally...

copy-pr-bot · 2025-10-19T00:12:35Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

asymingt · 2025-11-14T15:48:37Z

I can independently confirm that this works on my Talos cluster. After installing with helm I finally see resource slices being made available on a machine with five RTX A4000 GPUs. Thank you, @hydazz!

$ kubectl get resourceslices
NAME                                 NODE            DRIVER           POOL            AGE
talos-pxs-ia1-gpu.nvidia.com-gsvft   talos-pxs-ia1   gpu.nvidia.com   talos-pxs-ia1   11m

For repeatability, you will need a container image to be built. I have pushed one to asymingt/k8s-dra-driver-gpu. You will need to modify this line to asymingt/k8s-dra-driver-gpu:v25.8.0-dev before installing the chart this way:

helm upgrade -i nvidia-dra-driver-gpu ./k8s-dra-driver-gpu/deployments/helm/nvidia-dra-driver-gpu   \
   --create-namespace  --namespace drivers  \
   --set gpuResourcesEnabledOverride=true   \
   --set resources.gpus.enabled=true  \
   --set resources.computeDomains.enabled=false 
   --wait

To optionally rebuild the container image, install docker + qemu-binfmt + buildx, checkout this code and run:

export IMAGE_NAME=<your_docker_hub>/k8s-dra-driver-gpu
export VERSION=v25.8.0-dev
export PUSH_ON_BUILD=true
export BUILD_MULTI_ARCH_IMAGES=true

make -f deployments/container/Makefile build

hydazz · 2025-11-15T10:48:32Z

I believe this is set to be fixed on the Talos side, by them installing the nvidia stuff where this expects it to go, not the other way around

asymingt · 2025-11-16T13:50:35Z

While we wait for Talos to update its driver install location, I've been trying to get MPS working on Talos using this PR branch and the following helm values.

gpuResourcesEnabledOverride: true
resources:
  gpus:
    enabled: true
  computeDomains:
    enabled: false
featureGates:
  MPSSupport: true

Looks like the mps-control-daemon keeps restarting with the following error:

$ k logs mps-control-daemon-49e0f7b0-e884-4ab0-ac35-b50bca50f681-e4dlqgl -n drivers
chroot: can't execute 'sh': No such file or directory

It's probably related to this issue: #469

I've opened a PR to fix it on your branch: hydazz#1

klueska · 2025-11-24T13:56:37Z

@hydazz given your comment about Talos adjusting themselves to accommodate the existing search paths, how would you propose moving forward with this PR?

hydazz · 2025-11-25T07:00:15Z

@hydazz given your comment about Talos adjusting themselves to accommodate the existing search paths, how would you propose moving forward with this PR?

@klueska I don't have definitive knowledge, I just inferred that conclusion based on:
https://discord.com/channels/673534664354430999/942576972943491113/1434096797562703983

if we could move this to a github discussion under extensions repo, we could collaborate more (I believe /usr/local/glibc/usr/lib was a wrong choice of path from first place [thinking about merged /usr/ though it was kind of done to fix issues with musl libs co-existing), but let's do a discussion on a good path moving forward and trying to use the operator and dra plugins as much as possible without platform specific hacks

(I could not find such referenced discussion)

siderolabs/extensions#836
siderolabs/extensions#476
#605

I don't know if there is talks between nvidia/talos, or whats outside of linked above, but it could easily be fixed here, just with something better than getTalosLibrarySearchPaths (pretty easily?), or on the extension side, but thats probably a larger change.

Perhaps @frezbo would have more insight?

klueska · 2025-12-02T10:23:42Z

+func getTalosLibrarySearchPaths() []string {
+	return []string{
+		"/driver-root/usr/local/glibc/usr/lib",
+		"/driver-root/usr/local/glibc/lib",
+		"/driver-root/usr/local/glibc/lib64",
+	}
+}


@elezar is this something we would want to add directly to nvcdi in the nvidia-container-toolkit as a standard search path?

I don't think there's a problem in adding this to the toolkit. At the moment the defaults are defined at quite a low level (which is where @hydazz has added them in NVIDIA/nvidia-container-toolkit#1621) and we may want to consider making these easier to specify at a higher level.

frezbo · 2025-12-02T11:32:42Z

@hydazz given your comment about Talos adjusting themselves to accommodate the existing search paths, how would you propose moving forward with this PR?

@klueska I don't have definitive knowledge, I just inferred that conclusion based on: https://discord.com/channels/673534664354430999/942576972943491113/1434096797562703983

if we could move this to a github discussion under extensions repo, we could collaborate more (I believe /usr/local/glibc/usr/lib was a wrong choice of path from first place [thinking about merged /usr/ though it was kind of done to fix issues with musl libs co-existing), but let's do a discussion on a good path moving forward and trying to use the operator and dra plugins as much as possible without platform specific hacks

(I could not find such referenced discussion)

siderolabs/extensions#836 siderolabs/extensions#476 #605

I don't know if there is talks between nvidia/talos, or whats outside of linked above, but it could easily be fixed here, just with something better than getTalosLibrarySearchPaths (pretty easily?), or on the extension side, but thats probably a larger change.

Perhaps @frezbo would have more insight?

It would be nice if these paths are supported on the nvidia side, we're (SideroLabs) is open to using better paths, but we have a constraint that it cannot be standard /usr/lib since Talos base is musl, so glibc and musl libs needs to exist at different paths it could be as well usr/local/nvidia/lib for example

jgehrcke · 2026-01-29T10:08:56Z

@klueska what do you think: can we still do something here for the next release? I feel like we should. But now it's rather tight again.

Signed-off-by: hydazz <alexanderhyde@icloud.com>

hydazz · 2026-01-30T09:47:54Z

opened PR in the toolkit to remove the overwrite here
NVIDIA/nvidia-container-toolkit#1621

please review and lmk if any changes are needed, keen to jump onto the DRA train 🙂

elezar · 2026-01-30T10:50:55Z

@hydazz @jgehrcke if we're expecting the toolkit to be updated to include this change, I don't think that's something that can be done for the upcoming release.

Would a middle ground be adding a configurable option (envvar / config file option) that allows a user to specify the paths in the container to be searched for libraries explicilty? This can be passed to the CDI library on construction and used in the prestart scripts.

Note that although NVIDIA/nvidia-container-toolkit#1621 gets us some of the way there, we may need to also update the detection logic to also be able to locate nvidia-smi at the non-standard location.

jpsalvesen · 2026-03-02T13:24:08Z

Thank you, @asymingt, for sharing enough to get started with a PoC!

frezbo · 2026-03-10T12:31:48Z

Talos 1.13 now ships /etc/ld.* files, so this might not be a problem anymore, the gpu operator works now

rajatchopra · 2026-03-16T23:04:15Z

@hydazz do we need this addendum to search list based on this comment? Close the PR?

We may still want the cdiRoot as a overriding parameter - we need a real example though.

hydazz · 2026-03-17T21:38:41Z

@hydazz do we need this addendum to search list based on this comment? Close the PR?

We may still want the cdiRoot as a overriding parameter - we need a real example though.

Will play around with 1.13, doubt it'll just work but will confirm minimum changes needed here to get it to work.

Builds a patched version of nvcr.io/nvidia/k8s-dra-driver-gpu that adds /usr/local/glibc/usr/lib and /usr/local/bin to the library/binary search paths, matching upstream kubernetes-sigs/dra-driver-nvidia-gpu#695. Renovate will track NVIDIA/k8s-dra-driver-gpu releases to keep the VERSION in sync. Remove this app once PR #695 is released upstream. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

hydazz · 2026-03-30T07:15:31Z

Talos 1.13.0 observations with raw nvidia-dra/defaults:

k logs -n kube-system nvidia-dra-driver-gpu-kubelet-plugin-s7rpp -c init-container
create symlink: /driver-root -> /driver-root-parent/
2026-03-30T07:10:36Z  /driver-root (/ on host): nvidia-smi: not found, libnvidia-ml.so.1: not found, current contents: [].

Some path overwrites still seem to be needed? either in the scripts here, nvcdi overwrites here, or nvcdi overwrites upstream?
Please recommend which changes are appropriate, so I don't need to go in circles again 😊

Config diff:
hydazz/home-ops@d28c138
hydazz/home-ops@e3a5912

k8s-triage-robot · 2026-04-02T19:53:16Z

Unknown CLA label state. Rechecking for CLA labels.

Send feedback to sig-contributor-experience at kubernetes/community.

/check-cla
/easycla

linux-foundation-easycla · 2026-04-02T19:53:23Z

❌ - login: @hydazz / name: hydazz . The commit (2cdaeff, 302f40e) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

frezbo · 2026-04-03T20:59:25Z

Talos 1.13.0 observations with raw nvidia-dra/defaults:
k logs -n kube-system nvidia-dra-driver-gpu-kubelet-plugin-s7rpp -c init-container
create symlink: /driver-root -> /driver-root-parent/
2026-03-30T07:10:36Z  /driver-root (/ on host): nvidia-smi: not found, libnvidia-ml.so.1: not found, current contents: [].
Some path overwrites still seem to be needed? either in the scripts here, nvcdi overwrites here, or nvcdi overwrites upstream? Please recommend which changes are appropriate, so I don't need to go in circles again 😊

Config diff: hydazz/home-ops@d28c138 hydazz/home-ops@e3a5912

I've a custom installer with nvidia extensions, could you test it, I got dra working

 helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
  helm repo update
  helm upgrade --wait  --install -n gpu-operator gpu-operator nvidia/gpu-operator \
        --set driver.enabled=false \
        --set toolkit.enabled=false \
        --set hostPaths.driverInstallDir=/usr/local/lib \
        --set cdi.nriPluginEnabled=true \
        --version v25.10.1

then disable the deviceplugin

kubectl patch clusterpolicy cluster-policy --type=merge \
        -p '{"spec":{"devicePlugin":{"enabled":false}}}'

helm upgrade --install -n nvidia-dra-driver-gpu --set resources.gpus.enabled=true --set nvidiaDriverRoot=/usr/local nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu --set gpuResourcesEnabledOverride=true

frezbo · 2026-04-03T21:00:02Z

Talos 1.13.0 observations with raw nvidia-dra/defaults:
k logs -n kube-system nvidia-dra-driver-gpu-kubelet-plugin-s7rpp -c init-container
create symlink: /driver-root -> /driver-root-parent/
2026-03-30T07:10:36Z  /driver-root (/ on host): nvidia-smi: not found, libnvidia-ml.so.1: not found, current contents: [].
Some path overwrites still seem to be needed? either in the scripts here, nvcdi overwrites here, or nvcdi overwrites upstream? Please recommend which changes are appropriate, so I don't need to go in circles again 😊
Config diff: hydazz/home-ops@d28c138 hydazz/home-ops@e3a5912
I've a custom installer with nvidia extensions, could you test it, I got dra working
 helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
  helm repo update
  helm upgrade --wait  --install -n gpu-operator gpu-operator nvidia/gpu-operator \
        --set driver.enabled=false \
        --set toolkit.enabled=false \
        --set hostPaths.driverInstallDir=/usr/local/lib \
        --set cdi.nriPluginEnabled=true \
        --version v25.10.1
then disable the deviceplugin
kubectl patch clusterpolicy cluster-policy --type=merge \
        -p '{"spec":{"devicePlugin":{"enabled":false}}}'
helm upgrade --install -n nvidia-dra-driver-gpu --set resources.gpus.enabled=true --set nvidiaDriverRoot=/usr/local nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu --set gpuResourcesEnabledOverride=true

Use this to validate:

---
apiVersion: v1
kind: Namespace
metadata:
  name: dra-gpu-share-test
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  namespace: dra-gpu-share-test
  name: single-gpu
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.nvidia.com
---
apiVersion: v1
kind: Pod
metadata:
  namespace: dra-gpu-share-test
  name: pod
  labels:
    app: pod
spec:
  containers:
  - name: ctr0
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["nvidia-smi -L; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: shared-gpu
  - name: ctr1
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["nvidia-smi -L; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: shared-gpu
  resourceClaims:
  - name: shared-gpu
    resourceClaimTemplateName: single-gpu
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

dims · 2026-04-05T14:16:10Z

@hydazz can you please sign the CLA? or else we are not able to accept the changes.

frezbo · 2026-04-06T01:47:04Z

@hydazz can you please sign the CLA? or else we are not able to accept the changes.

@hydazz I don't think these changes are needed anymore with the work going into Talos 1.13

rothgar · 2026-04-11T04:12:08Z

Would they still be useful for Talos 1.11 and 1.12?

k8s-ci-robot · 2026-04-11T04:12:17Z

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

frezbo · 2026-04-11T05:56:03Z

Would they still be useful for Talos 1.11 and 1.12?

it just makes things complex, we were just using non-standard paths, but now we contained most and not have conflicting stuff

dims · 2026-04-11T11:16:11Z

@hydazz needs CLA and rebase!

frezbo · 2026-04-22T06:00:04Z

Talos 1.13 now support DRA as per official docs and this PR will break stuff on Talos, this maybe closed now

See: https://docs.siderolabs.com/kubernetes-guides/advanced-guides/dynamic-resource-allocation#dynamic-resource-allocation

jpsalvesen · 2026-04-22T07:52:47Z

Can confirm the existing DRA driver runs with talos v1.13.0-rc.0 - the kubelet plugin config pod running nvcr.io/nvidia/k8s-dra-driver-gpu:v25.12.0 proceeds to running state and produces a resourceslice.

klueska added the feature issue/PR that proposes a new feature or functionality label Nov 24, 2025

klueska added this to the unscheduled milestone Nov 24, 2025

klueska reviewed Dec 2, 2025

View reviewed changes

add talos support

302f40e

Signed-off-by: hydazz <alexanderhyde@icloud.com>

hydazz force-pushed the main branch from fdac825 to 302f40e Compare January 30, 2026 09:13

hydazz mentioned this pull request Jan 30, 2026

add Talos glibc library search paths NVIDIA/nvidia-container-toolkit#1621

Closed

remove local talos library search paths

2cdaeff

Signed-off-by: hydazz <alexanderhyde@icloud.com>

hydazz force-pushed the main branch from ce6408f to 2cdaeff Compare January 30, 2026 09:25

arsac mentioned this pull request Mar 18, 2026

feat(nvidia-dra-driver-gpu): add Talos OS support image arsac/containers#1

Merged

3 tasks

rajatchopra mentioned this pull request Mar 30, 2026

Recommended installation method on Talos Linux #605

Open

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Apr 2, 2026

frezbo mentioned this pull request Apr 7, 2026

feat: nvidia extension rework siderolabs/extensions#1034

Merged

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 11, 2026

Conversation

hydazz commented Oct 19, 2025

Uh oh!

copy-pr-bot Bot commented Oct 19, 2025

Uh oh!

asymingt commented Nov 14, 2025

Uh oh!

hydazz commented Nov 15, 2025

Uh oh!

asymingt commented Nov 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

klueska commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hydazz commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

klueska Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

elezar Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

frezbo commented Dec 2, 2025

Uh oh!

jgehrcke commented Jan 29, 2026

Uh oh!

hydazz commented Jan 30, 2026

Uh oh!

elezar commented Jan 30, 2026

Uh oh!

jpsalvesen commented Mar 2, 2026

Uh oh!

frezbo commented Mar 10, 2026

Uh oh!

rajatchopra commented Mar 16, 2026

Uh oh!

hydazz commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hydazz commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-triage-robot commented Apr 2, 2026

Uh oh!

linux-foundation-easycla Bot commented Apr 2, 2026

Uh oh!

frezbo commented Apr 3, 2026

Uh oh!

frezbo commented Apr 3, 2026

Uh oh!

dims commented Apr 5, 2026

Uh oh!

frezbo commented Apr 6, 2026

Uh oh!

rothgar commented Apr 11, 2026

Uh oh!

k8s-ci-robot commented Apr 11, 2026

Uh oh!

frezbo commented Apr 11, 2026

Uh oh!

dims commented Apr 11, 2026

Uh oh!

frezbo commented Apr 22, 2026

Uh oh!

jpsalvesen commented Apr 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

asymingt commented Nov 16, 2025 •

edited

Loading

klueska commented Nov 24, 2025 •

edited

Loading

hydazz commented Nov 25, 2025 •

edited

Loading

hydazz commented Mar 17, 2026 •

edited

Loading

hydazz commented Mar 30, 2026 •

edited

Loading