Skip to content
Open
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 166 additions & 0 deletions docs/SRU/reference/exception-Nvidia-Cuda-Updates.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
.. _reference-exception-Nvidia-Cuda-Updates:

Nvidia CUDA updates
===================

Introduction
-------------

This document describes the policy, process and criteria for updating NVIDIA
CUDA libraries in a stable supported distro, including LTS.

NVIDIA CUDA is broadly used by developers for GPU compute activities, for
example for AI/ML. Canonical has a redistribution agreement with NVIDIA to
redistribute the CUDA libraries in the Ubuntu archive. Per the agreement, Canonical
must deliver the prebuilt binaries from NVIDIA without modifications.

CUDA consists of about 37 new source packages for every minor version. Minor versions
are released on average every 3 months. Each minor version usually receives one or
two patch versions, which will be candidates for SRUs. Since our redistribution
agreement requires us to match NVIDIA's changes exactly within a given CUDA release,
and since CUDA consists primarily of precompiled binaries, our CUDA packages must all
roll forward in tandem on each upstream update. As a result, CUDA SRUs are not
compliant with the standard SRU policy.

As of the time of writing, CUDA 13.x packages are under an allowed exception to
install files under /usr/local. The exception is temporary, and we are working on a
plan to remove it with CUDA 14.x. Any SRU request that is patching a CUDA version
relying on the exception is expected to keep installing files under /usr/local.

.. _release_versions:

Released versions
------------------

**Minor versions**:

CUDA packages are released in the multiverse pocket.
CUDA's minor versions are meant to be available and installable in parallel, and
therefore e.g. 13.2 does not replace 13.1 in Ubuntu. Minor versions are always a
new set of source packages, suffixed with MAJOR.MINOR, e.g. cuda-13-2.

CUDA's minor version package sets are meant to be broadly SRU'd to
compatible releases that are supported, starting with 26.04. For example, when
the currently hypothetical 14.1.0 version is released for 28.04, it will be
SRU'd to 27.10 and 26.04.

A few binary package have the same name between two minor versions:
- cuda
- cuda-toolkit and cuda-toolkit-<MAJOR>

Users expect these binary packages to be rolling. They can use
cuda-<MAJOR>-<MINOR> and cuda-toolkit-<MAJOR>-<MINOR> to be sure to stay on
the same minor version. NVIDIA recommends the installation of minor-tight
CUDA versions in their own documentation.

**Patch versions**:

Patch versions do replace the corresponding existing version so e.g. 13.2.1 will
replace 13.2.0.

CUDA's patch version package sets are also meant to be broadly SRU'd to
compatible releases that are supported. For example, 26.04 was released with
13.1.1 and when 13.1.2 is available in devel it will be SRU'd up until 26.04.

In the past, a patch version of nsight-compute could change the name of a
binary package. For example, 13.2.0 had `nsight-compute-2026.1.0` while
13.2.1 had `nsight-compute-2026.1.1`. In such cases, the new package, this
SRU exception allows the patch version to be SRU'd only if there is a
transitional package from the old name to the new name.

.. _requesting_sru:

NVIDIA CUDA Requesting the SRU
-------------------------------

The SRU should be done with a single process bug appropriately named
`cuda-X-Y`, instead of individual bug
reports for individual bug fixes. The one bug should have the following:

- The SRU should be requested per the :ref:`StableReleaseUpdates
<howto-perform-standard-sru>` documented process
- The template at the end of this document should be used and all ‘TODO’ items
filled out
- This SRU exception applies to only the multiverse archive component.
- The changelog will contain a reference to the single SRU process bug, not all
bugs fixed by the SRU. However, if there are very important bugs that are
deemed worthy of reference they too should be included in the changelog.
- Major changes should be called out in the SRU template, especially where
changed behavior is not backward compatible.
- For each release that is proposed to be updated by the SRU a link to the
results of the automated tests so that anyone can verify that they have been
executed successfully.
- Additionally, the SRU bug should be verbose in documenting any manual testing
that occurred.
- Any architecture specific fixes need to be noted and architecture specific
test results included.
- Any packaging changes (e.g. a dependency change) need to be stated

.. _packaging_qa:


NVIDIA CUDA Packaging QA
-------------------------

The objective of the QA is to test:

- Package installation from scratch
- Package upgrades
- Compliance with NVIDIA's own releases

- Making sure that we deliver the right binary in the right packages with
the right license and dependencies
- Making sure the CUDA installation is working as expected (integration
tests with cuda samples).
- Compatibility with NVIDIA's own releases and repos

- Making sure that NVIDIA's packages supersede ours, and that users adding
the NVIDIA repo don't end up with a mix of our packages and NVIDIA's packages

This QA is implemented as an autopkgtest within each source package. The result
of the tests will be attached to the SRU bug. The package upgrade must be
attempted manually, from a fresh installation.

.. _integration_tests:


Integration tests
------------------

- `Certification test suite
<https://github.com/canonical/checkbox/blob/main/providers/gpgpu/units/cuda.pxu>`__
must pass on a range of hardware, with the same result as with NVIDIA's

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

must pass on a range of hardware

nit: let's say "must pass on a range of hardware, including at least one workstation and one server GPU"

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, indeed that's better, applied

provided packages.

.. _sru_template:


NVIDIA CUDA SRU Template
-------------------------

::

[Impact]
This patch provides both bug fixes and improvements and we would like to
make sure all of our users have access to these improvements.

The patched packages are:

*** <TODO: Provide a list of updated packages, by comparing https://developer.download.nvidia.com/compute/cuda/redist/redistrib_${MAJOR}.${MINOR}.${PATCH}.json for the current and target versions >

[Test Plan]
The following development and SRU process was followed:
https://documentation.ubuntu.com/sru/en/latest/reference/exception-Nvidia-Cuda-Updates/

<TODO Document any QA done, automated and manual>

The QA team that executed the tests will be in charge of attaching the artifacts and
console output of the appropriate run to the bug. NVIDIACUDA maintainers team members
will not mark ‘verification-done’ until this has happened.

[Where problems could occur]
NVIDIA could deliver a not-so-minor change that could cause regressions. The pre-built nature of CUDA
prevents us from detecting that. It might fail to install due to a newly added missing package, for example.
The installation test, the autopkgtests and the integration tests will to detect that.

<TODO: attach test artifacts for every SRU release, not a link as links expire>
1 change: 1 addition & 0 deletions docs/SRU/reference/package-specific.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ the Technical Board.
exception-MariaDB-Galera-Updates
exception-multipath-tools-Updates
exception-NVidia-Updates
exception-Nvidia-Cuda-Updates
exception-Netplan-Updates
exception-OEMMeta-Updates
exception-OpenJDK-Updates
Expand Down