-
Notifications
You must be signed in to change notification settings - Fork 106
Add SRU Exception for NVIDIA CUDA #517
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
antlassagne
wants to merge
14
commits into
ubuntu:main
Choose a base branch
from
antlassagne:main
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 13 commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
c1655a0
SRU Exception request: NVIDIA CUDA
antlassagne 8645091
* Add a mention to /usr/local
antlassagne b407ebe
Review from Alex
antlassagne cdfb440
Update docs/SRU/reference/exception-nvidia-cuda.rst
antlassagne 767984a
Update docs/SRU/reference/exception-nvidia-cuda.rst
antlassagne 7c174d6
Update docs/SRU/reference/exception-nvidia-cuda.rst
antlassagne f857277
Review from the team
antlassagne 623264b
Merge pull request #1 from antlassagne/nvidia-cuda-sru-exception
antlassagne 1925f5a
Review from Robie
antlassagne 486b98b
Improve phrasing
antlassagne c097446
Fix Built doc, add link to the exception panel
antlassagne e26d53b
Add a paragraph to detail conflicts between minor versions
antlassagne 04390c4
Add the special case for nsight-compute transitional packages require…
antlassagne 4f8994b
Review from Mitchell
antlassagne File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,166 @@ | ||
| .. _reference-exception-Nvidia-Cuda-Updates: | ||
|
|
||
| Nvidia CUDA updates | ||
| =================== | ||
|
|
||
| Introduction | ||
| ------------- | ||
|
|
||
| This document describes the policy, process and criteria for updating NVIDIA | ||
| CUDA libraries in a stable supported distro, including LTS. | ||
|
|
||
| NVIDIA CUDA is broadly used by developers for GPU compute activities, for | ||
| example for AI/ML. Canonical has a redistribution agreement with NVIDIA to | ||
| redistribute the CUDA libraries in the Ubuntu archive. Per the agreement, Canonical | ||
| must deliver the prebuilt binaries from NVIDIA without modifications. | ||
|
|
||
| CUDA consists of about 37 new source packages for every minor version. Minor versions | ||
| are released on average every 3 months. Each minor version usually receives one or | ||
| two patch versions, which will be candidates for SRUs. Since our redistribution | ||
| agreement requires us to match NVIDIA's changes exactly within a given CUDA release, | ||
| and since CUDA consists primarily of precompiled binaries, our CUDA packages must all | ||
| roll forward in tandem on each upstream update. As a result, CUDA SRUs are not | ||
| compliant with the standard SRU policy. | ||
|
|
||
| As of the time of writing, CUDA 13.x packages are under an allowed exception to | ||
| install files under /usr/local. The exception is temporary, and we are working on a | ||
| plan to remove it with CUDA 14.x. Any SRU request that is patching a CUDA version | ||
| relying on the exception is expected to keep installing files under /usr/local. | ||
|
|
||
| .. _release_versions: | ||
|
|
||
| Released versions | ||
| ------------------ | ||
|
|
||
| **Minor versions**: | ||
|
|
||
| CUDA packages are released in the multiverse pocket. | ||
| CUDA's minor versions are meant to be available and installable in parallel, and | ||
| therefore e.g. 13.2 does not replace 13.1 in Ubuntu. Minor versions are always a | ||
| new set of source packages, suffixed with MAJOR.MINOR, e.g. cuda-13-2. | ||
|
|
||
| CUDA's minor version package sets are meant to be broadly SRU'd to | ||
| compatible releases that are supported, starting with 26.04. For example, when | ||
| the currently hypothetical 14.1.0 version is released for 28.04, it will be | ||
| SRU'd to 27.10 and 26.04. | ||
|
|
||
| A few binary package have the same name between two minor versions: | ||
| - cuda | ||
| - cuda-toolkit and cuda-toolkit-<MAJOR> | ||
|
|
||
| Users expect these binary packages to be rolling. They can use | ||
| cuda-<MAJOR>-<MINOR> and cuda-toolkit-<MAJOR>-<MINOR> to be sure to stay on | ||
| the same minor version. NVIDIA recommends the installation of minor-tight | ||
| CUDA versions in their own documentation. | ||
|
|
||
| **Patch versions**: | ||
|
|
||
| Patch versions do replace the corresponding existing version so e.g. 13.2.1 will | ||
| replace 13.2.0. | ||
|
|
||
| CUDA's patch version package sets are also meant to be broadly SRU'd to | ||
| compatible releases that are supported. For example, 26.04 was released with | ||
| 13.1.1 and when 13.1.2 is available in devel it will be SRU'd up until 26.04. | ||
|
|
||
| In the past, a patch version of nsight-compute could change the name of a | ||
| binary package. For example, 13.2.0 had `nsight-compute-2026.1.0` while | ||
| 13.2.1 had `nsight-compute-2026.1.1`. In such cases, the new package, this | ||
| SRU exception allows the patch version to be SRU'd only if there is a | ||
| transitional package from the old name to the new name. | ||
|
|
||
| .. _requesting_sru: | ||
|
|
||
| NVIDIA CUDA Requesting the SRU | ||
| ------------------------------- | ||
|
|
||
| The SRU should be done with a single process bug appropriately named | ||
| `cuda-X-Y`, instead of individual bug | ||
| reports for individual bug fixes. The one bug should have the following: | ||
|
|
||
| - The SRU should be requested per the :ref:`StableReleaseUpdates | ||
| <howto-perform-standard-sru>` documented process | ||
| - The template at the end of this document should be used and all ‘TODO’ items | ||
| filled out | ||
| - This SRU exception applies to only the multiverse archive component. | ||
| - The changelog will contain a reference to the single SRU process bug, not all | ||
| bugs fixed by the SRU. However, if there are very important bugs that are | ||
| deemed worthy of reference they too should be included in the changelog. | ||
| - Major changes should be called out in the SRU template, especially where | ||
| changed behavior is not backward compatible. | ||
| - For each release that is proposed to be updated by the SRU a link to the | ||
| results of the automated tests so that anyone can verify that they have been | ||
| executed successfully. | ||
| - Additionally, the SRU bug should be verbose in documenting any manual testing | ||
| that occurred. | ||
| - Any architecture specific fixes need to be noted and architecture specific | ||
| test results included. | ||
| - Any packaging changes (e.g. a dependency change) need to be stated | ||
|
|
||
| .. _packaging_qa: | ||
|
|
||
|
|
||
| NVIDIA CUDA Packaging QA | ||
| ------------------------- | ||
|
|
||
| The objective of the QA is to test: | ||
|
|
||
| - Package installation from scratch | ||
| - Package upgrades | ||
| - Compliance with NVIDIA's own releases | ||
|
|
||
| - Making sure that we deliver the right binary in the right packages with | ||
| the right license and dependencies | ||
| - Making sure the CUDA installation is working as expected (integration | ||
| tests with cuda samples). | ||
| - Compatibility with NVIDIA's own releases and repos | ||
|
|
||
| - Making sure that NVIDIA's packages supersede ours, and that users adding | ||
| the NVIDIA repo don't end up with a mix of our packages and NVIDIA's packages | ||
|
|
||
| This QA is implemented as an autopkgtest within each source package. The result | ||
| of the tests will be attached to the SRU bug. The package upgrade must be | ||
| attempted manually, from a fresh installation. | ||
|
|
||
| .. _integration_tests: | ||
|
|
||
|
|
||
| Integration tests | ||
| ------------------ | ||
|
|
||
| - `Certification test suite | ||
| <https://github.com/canonical/checkbox/blob/main/providers/gpgpu/units/cuda.pxu>`__ | ||
| must pass on a range of hardware, with the same result as with NVIDIA's | ||
| provided packages. | ||
|
|
||
| .. _sru_template: | ||
|
|
||
|
|
||
| NVIDIA CUDA SRU Template | ||
| ------------------------- | ||
|
|
||
| :: | ||
|
|
||
| [Impact] | ||
| This patch provides both bug fixes and improvements and we would like to | ||
| make sure all of our users have access to these improvements. | ||
|
|
||
| The patched packages are: | ||
|
|
||
| *** <TODO: Provide a list of updated packages, by comparing https://developer.download.nvidia.com/compute/cuda/redist/redistrib_${MAJOR}.${MINOR}.${PATCH}.json for the current and target versions > | ||
|
|
||
| [Test Plan] | ||
| The following development and SRU process was followed: | ||
| https://documentation.ubuntu.com/sru/en/latest/reference/exception-Nvidia-Cuda-Updates/ | ||
|
|
||
| <TODO Document any QA done, automated and manual> | ||
|
|
||
| The QA team that executed the tests will be in charge of attaching the artifacts and | ||
| console output of the appropriate run to the bug. NVIDIACUDA maintainers team members | ||
| will not mark ‘verification-done’ until this has happened. | ||
|
|
||
| [Where problems could occur] | ||
| NVIDIA could deliver a not-so-minor change that could cause regressions. The pre-built nature of CUDA | ||
| prevents us from detecting that. It might fail to install due to a newly added missing package, for example. | ||
| The installation test, the autopkgtests and the integration tests will to detect that. | ||
|
|
||
| <TODO: attach test artifacts for every SRU release, not a link as links expire> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: let's say "must pass on a range of hardware, including at least one workstation and one server GPU"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you, indeed that's better, applied