Conversation
uPCIe translates VRAM addresses by exporting a device range as a dma-buf and reading its scatter list, and the per-chunk form of that has been assumed to work on both vendors. It does not: on ROCm the range arguments to hipMemGetHandleForAddressRange are accepted and discarded, and what comes back describes the whole underlying buffer object at the base of the allocation regardless of the offset asked for. CUDA honours the request exactly. Since trusting the first segment address then puts the DMA somewhere other than where the caller pointed, and does so quietly, the difference needs to be measured and written down rather than inferred. This adds a probing tool per flavour and the findings from running both, so the memory work can be designed against what the runtimes do. Probes report rather than assert, which is why they sit apart from tests/. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
Whether a registration can require whole granules turns on what a runtime reports for an allocation that does not end on a granule boundary, and on whether the export agrees with it. It does not: both report the size unrounded, then CUDA exports exactly that while ROCm exports the larger object it actually allocated. A LUT fill that trusts the reported size to bound the export therefore fails on one vendor and overshoots on the other. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
…ce's MMIO VFIO_DEVICE_FEATURE_DMA_BUF exports a slice of a BAR, and test_dmamem_vfio_bar already reports that the descriptor carries no CPU mapping. That matters because cuMemHostRegister() with CU_MEMHOSTREGISTER_IOMEMORY, which is how the GPU-initiated NVMe path reaches a doorbell, takes a host address. A process holding only the descriptor has none to give it. The remaining route was for the runtime to import the descriptor itself, and the answer decides whether MMIO can be delegated to a process that never holds the device fd. It cannot, though not for the reason the question anticipated: CUDA refuses CU_EXTERNAL_MEMORY_HANDLE_TYPE_DMABUF_FD with NOT_SUPPORTED for any exporter, including a dma-buf it exported itself, and HIP has no dma-buf handle type to ask with. The probe carries that control with it, since a runtime that will not import its own descriptor says nothing about vfio. It reads rather than writes and defaults to offset 0, so it touches CAP rather than a doorbell, and prints the host's read of the same register beside the device's. The findings are in tools/README.md next to the probe. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
Passing a vfio device fd over SCM_RIGHTS and mapping BAR0 in the receiver is already known to work, and the GPU-initiated NVMe path is already known to ring doorbells through cuMemHostRegister(CU_MEMHOSTREGISTER_IOMEMORY). What was not established is that the second works on a mapping produced by the first, which is the whole of the case for delegating a controller to another process. It does, as root, and the SM reads what the primary reads. Two processes over a named socket rather than a fork with a setuid in it, since the question is what an unprivileged process can do and a process that dropped privilege is not in that state: it keeps the supplementary groups it started with unless they are cleared, and a uid change clears the dumpable flag. As an unprivileged process the secondary gets the descriptor, maps BAR0 and reads the register, then fails the registration with CUDA_ERROR_NOT_PERMITTED. The standalone mode is the control for that result, opening the device itself with no delegation at all, and it is refused in the same place, so the requirement belongs to the calling process rather than to the descriptor. The read comes from a kernel rather than from cuMemcpyDtoH(), since the copy engine is not what rings a doorbell. It reads CAP rather than writing a doorbell, so running it disturbs nothing. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
Under uio_pci_generic a controller consumes physical addresses and reaches a GPU allocation through its dma-buf scatter list. Under vfio-pci it consumes IOVAs, so the allocation has to be mapped with IOMMU_IOAS_MAP_FILE, and iommufd.h records that as of 6.19 that call takes only dma-bufs exported by vfio-pci. That comment is load-bearing for anything wanting a controller and a GPU under the same IOMMU, it describes a kernel we have moved off, and nothing here asks the kernel we run. A memfd is mapped first as a control, since a MAP_FILE that refuses everything would say nothing about GPU memory in particular. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
The privilege boundary the probe found was measured with CUDA only, while the design it informs speaks about GPU submission without naming a vendor. HIP has hipHostRegisterIoMemory, so the same question can be put to ROCm. The device-side read goes through hipMemcpyDtoH rather than an SM, since this project has no ROCm kernel-compilation path. That is weaker than the CUDA flavour and it is enough for the question being asked, which is whether the registration is accepted at all. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
…the primary Two questions the design turns on were being answered by assumption. Which side of a delegation is charged for pinned pages decides whether an unprivileged secondary needs its RLIMIT_MEMLOCK raised, and the design withholds the iommufd partly on the strength of an answer nobody measured. What a secondary still has once the primary exits decides whether a restart is a supported handover or fatal, and the earlier note on it rested on a premise that was wrong. Both are vfio questions rather than GPU ones, so this probe needs no runtime and builds everywhere. It maps once from each side so a lowered limit on either shows which one it bounds. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
These were written in the xNVMe tree while working out how a controller could be shared between processes, which put them a repository away from the nine probes that ask the same kind of question, and from the vfio and iommufd wrappers they exercise. They belong beside upcie_vfio_delegate_probe, which asks about the accounting and lifetime of the same delegation. Moved verbatim, keeping their original attribution. The share probe gains a _GNU_SOURCE definition because the build here is -std=c11, under which syscall and ftruncate are not declared; it still includes no uPCIe header, since what it asks about is raw kernel behaviour. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The probes under
tools/report what a runtime or the kernel actually does, asopposed to
tests/, which assert what uPCIe requires. They print and exit zero;they do not fail when the answer is inconvenient.
They were carried along in #59 and #63 because that is where
they were needed. They belong on their own: they touch
tools/and onesubdir('tools')line, and depend on nothing either of those branches adds.Draft because the questions they answer are still moving. The kernel refuses
IOMMU_IOAS_MAP_FILEon a GPU-exported dma-buf today, and if that changes thenwhat
upcie_probe_vram_ioas_{cuda,hip}reports changes with it.Worth a careful look
The flavour contract in
probe_dmabuf.h. A flavour defines three functions,includes the header, and calls
probe_run(). The functions are prototyped inthe header rather than described only in prose, so a flavour missing one, or
defining it with another signature, is diagnosed at the prototype instead of at
the point of use or at link time.
Verification
upcie_probe_vfio_cdev,upcie_probe_vfio_shareandupcie_probe_vfio_delegatebuild on Linux with no accelerator toolchain present, and every commit builds on
its own.
clang-formatis clean across the branch.The CUDA and HIP flavours are gated on their toolchains and were not compiled on
the machine this was assembled on, so they are unverified here; the contract in
probe_dmabuf.hwas instead checked with a stub flavour, both complete and witha function withheld, to confirm the prototypes catch it. What the probes report
was measured earlier on an NVIDIA RTX A6000 and an AMD Radeon RX 7800 XT, and
those findings are recorded in
tools/README.md.