HOMI: replace the shared segment with a socket (1/4) - #2
Closed
safl wants to merge 14 commits into
Closed
Conversation
…s usage The usage output is captured from the tool itself rather than transcribed, so it cannot drift from what `--help` actually prints. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
uPCIe grew the primitives needed to share a controller under vfio: a runtime record describing a controller in offsets, a delegation protocol, and a heap that carries its own description. None of them exist at v0.7.0, which is what this tree vendored until now, so the whole tree moves to v0.8.0 rather than mixing versions. Taken verbatim from safl/upcie#65, which tags the layer safl/upcie#63 adds. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
…attach to Sharing a controller under an IOMMU means handing a consumer descriptors, since the device file cannot be bound twice, and descriptors alone are not enough: the consumer still has to know where the controller's queues and buffers live inside the heap it has been given. So the owner publishes a record of that, in data and heap offsets rather than addresses, which mean nothing in another address space. What backs the heap decides what is published. Under an IOMMU the heap is described in IOVAs and its descriptor is the dmamem's, while on the UIO path it is physical addresses and the hugepage's; publishing the wrong one hands a consumer a file it cannot map, or addresses its device will not honour. The record and the heap's description are allocated per export, so the export carries their offsets and releases them on every path out, including the ones that fail after allocating. xnvme_mproc_serve() returns so the caller can close its devices, so a process serving twice would otherwise leave one of each behind per call. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
A process that wants to use a controller somebody else opened has to be handed descriptors, and a unix socket is the only way Linux moves a descriptor between unrelated processes. So the owner holds one and answers what arrives on it, which also settles the questions the shared segment answered badly: binding the address is the election, and a closed connection is how a consumer's death is noticed, whatever it was doing when it died. The listener is non-blocking because accept() is drained in a loop, while what it hands back is not, since reading a message waits for the rest of one. A consumer that connects and then takes a moment to write has not gone away. Serving reads this backend's own state, so a device opened through another backend is refused rather than reinterpreted. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
The heap's allocator stays with homi, so a consumer has nowhere to get a DMA buffer, and every I/O needs one. It asks instead, and homi allocates on its behalf and answers with an offset the consumer's own mapping resolves. What that adds to the serving side is bookkeeping rather than mechanism: a consumer that disconnects has its loans reclaimed along with its queues, and in that order, so the controller loses the ability to reach an address before the address stops meaning anything. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
The counterpart to serving. Where the owner allocated the memory and opened the controller, this receives descriptors for both and builds a view: the same memory mapped here, a translation table for this process's addresses, and BAR0 so that queues granted later can be rung from this side. Nothing is allocated and nothing is owned, which is the point. The physical addresses come from where the owner left them, because reading them here would need a privilege a consumer has no reason to hold, and the socket staying open is what tells the owner this process is still using what it was given. Connecting to nobody is not an error: it means no one is serving that identifier, and the caller decides whether to become the owner instead. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
Opening with an shm_id now tries to attach before it builds anything. Somebody may already own that identifier, and attaching to them is cheaper than allocating a runtime and then discovering they exist. Finding nobody is how a process learns it should become the owner, which is what the shared segment's lock used to say. Buffers follow the same rule as the admin queue: an attached process asks for memory rather than taking it, since the allocator belongs to whoever owns the heap. Teardown closes the socket and returns, because nothing below that is this process's to release, and the close is what tells the owner to reclaim it. The restriction to uio_pci_generic goes with it. It was there because the owner published a hugepage by path for consumers to re-open, which the memfd and container paths could not do. What crosses now is descriptors, and a descriptor is a descriptor whichever way the controller is reached. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
… published An attached process describes the controller rather than opening it, because opening one twice is what the first measurement showed cannot be done. It fills a local structure from the record and points it at its own mapping of BAR0, so that a queue granted later is rung from here rather than through anybody. Queues come the same way. Asking produces identifiers and offsets; this side turns them into addresses, derives the doorbells from its own mapping, and builds the request pool around the scratch the grant named, since a process that cannot allocate from the heap cannot make that scratch for itself. The admin queue is left empty on purpose. There is one of it, it belongs to the owner, and leaving a pointer to something no process here may drive would be worse than leaving nothing. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
With the pieces in place, opening a device while attached takes the short path: describe the controller from the record, ask for a queue to submit on, and skip everything that assumes this process opened anything. Bus Master is left alone for the same reason a shared-segment secondary leaves it alone, since the owner flipped it and config space is not this process's to touch. Closing hands the queue back and frees the description. The controller is not closed, because it is not this process's to close, and the BAR mapping goes with the runtime rather than with the device. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
An attached process has no admin queue, deliberately: there is one of it and it belongs to whoever opened the controller. So the command goes over the socket and the owner submits it. The payload does not go with it. The command names an address this process registered or was granted, so an identify lands in this process's own buffer and only the command and its completion cross. Where a transfer needs a PRP list, the scratch comes from the granted queue's pool, since that is the only scratch a process which cannot allocate has. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
An attached process cannot create a queue: creating one means an admin command on a queue it does not have, and memory from a heap it cannot allocate from. It asks instead, and turns the offsets it is given into addresses and doorbells of its own. Handing the queue back goes the same way, so a consumer that closes cleanly returns what it took, and one that does not is reaped by the owner when the socket closes. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
The segment answered three questions and answered them badly. Who is primary was decided by an flock on a role file; whether a runtime is alive was inferred from a refcount a killed process never decremented, which is why stale-segment detection had to exist; and where a controller's objects live was published as the primary's own pointers, which every consumer then rebased by the offset between two mappings. The socket answers all three: binding the address is the election, a closed connection is a death whatever the process was doing, and what crosses is offsets and descriptors, which mean the same thing everywhere. So the segment goes, and with it the pointer surgery, the process-shared mutex, the magic and version guarding a layout nobody reads any more, and the names of the five filesystem objects two processes used to find each other by. Reporting moves onto the same socket: connecting is most of the answer, since a process that answers is a process holding the controllers. This is a break. A build with this change cannot share a controller with a build without it, and the shm_id that used to name a segment now names a socket. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
A caller that wants to use a shared controller has to know whether anybody is holding one, and a process that starts a primary has to know when it is ready to be attached to. Both were answerable only by opening a device, which is the thing a would-be consumer cannot do while somebody else owns it. So homi asks the runtime instead, over the socket it already serves, and prints what comes back. It exits zero only when a primary is running and every controller it holds has finished coming up, which makes it usable as a readiness check without parsing its output. Asking is not attaching, so whoever asks does not count itself among the consumers. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
An shm_id of 1 sits where a boolean would, so a value that got coerced somewhere along the way would still select the same segment and the mistake would go unnoticed. 42 costs nothing and is unambiguous in a log. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Simon A. F. Lund <os@safl.dk>
Owner
Author
|
Reopening against xnvme/xnvme instead; this fork-local base was only needed for GitHub's stacking. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HOMI's multi-process mode shares a controller by naming a POSIX shared-memory
segment and having consumers map it. That cannot work behind an IOMMU: the vfio
device file is exclusive, so a second process can open it but not bind it, and
no amount of shared memory hands over the binding. The descriptor itself has to
cross, which needs a channel rather than a segment.
So the owner serves consumers over a unix socket, passing the vfio device, the
iommufd and the heap as descriptors, and answers admin and queue requests on the
same channel. The segment goes:
xnvme_be_upcie_mproc.closes 806 lines.Stacked on the uPCIe delegation layer, vendored from safl/upcie#65.
Worth a careful look
That every commit leaves the tree working. The runtime is built up over
eight commits before the segment is removed, so the removal is a deletion rather
than a rewrite.
homi statusis new here rather than ported; it did not existbefore this branch.
The socket is the whole security boundary. A consumer that reaches the
socket receives the device descriptor, so anything it could be stopped from
doing afterwards it can already do. The admin hook accepts everything on
purpose; it is a place for later policy, not a whitelist.
What happens when somebody is already serving. Taking the socket, refusing a
stranger's protocol version, and reaping a consumer that disconnected are the
paths worth reading closely.
Verification
Every commit builds release individually on Linux 7.0.0-28-generic, and the tip
builds release, debug and with
-Dbe_upcie=false.make format-allis clean.Not yet re-run against this cut: the cijoe
test_mprocselection and thetest_homisuite need a controller, and were last exercised against an earlierarrangement of these commits. The GPU-under-vfio paths are in a later PR in this
stack and are not exercised here at all.