Skip to content

nvme/util: support multi-page prplists in PRP #64

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
minwooim merged 1 commit into from
Jul 21, 2026
Merged

nvme/util: support multi-page prplists in PRP #64

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 15 additions & 6 deletions include/vfn/nvme/rq.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,8 +212,11 @@ static inline void nvme_rq_exec(struct nvme_rq *rq, union nvme_cmd *cmd)
*
* Map a buffer of size @len into the command payload.
*
* This helper uses a pre-allocated PRP list page within @rq and same with
* calling ``nvme_map_prp(ctrl, rq->page.vaddr, ...)``.
* This helper uses a pre-allocated PRP list page within @rq and is equivalent
* to calling ``nvme_map_prp(ctrl, rq->page.vaddr, 1, cmd, iova, len)``. Since a
* single page is used, the buffer length is limited to what fits in one PRP
* list page; for larger buffers, allocate a multi-page PRP list and call
* nvme_map_prp() directly.
*
* Return: ``0`` on success, ``-1`` on error and sets errno.
*/
Expand All @@ -233,8 +236,11 @@ int nvme_rq_map_prp(struct nvme_ctrl *ctrl, struct nvme_rq *rq, union nvme_cmd *
* allowed to be unaligned, but the entry MUST end on a page boundary. All
* subsequent entries MUST be page aligned.
*
* This helper uses a pre-allocated PRP list page within @rq and same with
* calling ``nvme_mapv_prp(ctrl, rq->page.vaddr, rq->page.iova, cmd, iova, niov)``;
* This helper uses a pre-allocated PRP list page within @rq and is equivalent
* to calling ``nvme_mapv_prp(ctrl, rq->page.vaddr, 1, cmd, iova, niov)``. Since
* a single page is used, the total buffer length is limited to what fits in one
* PRP list page; for larger buffers, allocate a multi-page PRP list and call
* nvme_mapv_prp() directly.
*
* Return: ``0`` on success, ``-1`` on error and sets errno.
*/
Expand All @@ -254,8 +260,11 @@ int nvme_rq_mapv_prp(struct nvme_ctrl *ctrl, struct nvme_rq *rq, union nvme_cmd
* allowed to be unaligned, but the entry MUST end on a page boundary. All
* subsequent entries MUST be page aligned.
*
* This helper uses a pre-allocated PRP list page within @rq and same with
* calling ``nvme_mapv_iova_prp(ctrl, rq->page.vaddr, rq->page.iova, cmd, iova, niov)``;
* This helper uses a pre-allocated PRP list page within @rq and is equivalent
* to calling ``nvme_mapv_iova_prp(ctrl, rq->page.vaddr, 1, cmd, iova, niov)``.
* Since a single page is used, the total buffer length is limited to what fits
* in one PRP list page; for larger buffers, allocate a multi-page PRP list and
* call nvme_mapv_iova_prp() directly.
*
* Return: ``0`` on success, ``-1`` on error and sets errno.
*/
Expand Down
35 changes: 26 additions & 9 deletions include/vfn/nvme/util.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,24 +115,37 @@ int nvme_admin(struct nvme_ctrl *ctrl, union nvme_cmd *sqe, void *buf, size_t le
* command from a buffer that is contiguous in iova mapped
* memory.
* @ctrl: &struct nvme_ctrl
* @prplist: The first PRP list page address
* @prplists: contiguous array of PRP list pages, each one MPS page in size,
* allocated and iova-mapped by the caller
* @nprplists: number of pages in @prplists
* @cmd: NVMe command prototype (&union nvme_cmd)
* @iova: I/O Virtual Address
* @len: Length of buffer
*
* Map a buffer of size @len into the command payload.
*
* When the buffer requires more PRP entries than fit in a single page, the PRP
* list is chained across @prplists: the last entry of a non-final page holds
* the iova of the next page. The number of data PRPs that may be stored in
* @prplists is therefore ``(nprplists - 1) * (max_prps - 1) + max_prps``, where
* ``max_prps`` is the number of PRP entries per page; if @len requires more,
* ``-1`` is returned.
*
* @prplists must be a single contiguous mapping that is contiguous in both
* virtual and iova space, so that page @p sits at
* ``prplists + p * max_prps`` and its iova is ``base_iova + p * pagesize``.
*
* Return: ``0`` on success, ``-1`` on error and sets errno.
*/
int nvme_map_prp(struct nvme_ctrl *ctrl, leint64_t *prplist, union nvme_cmd *cmd,
iova_t iova, size_t len);
int nvme_map_prp(struct nvme_ctrl *ctrl, leint64_t *prplists, int nprplists,
union nvme_cmd *cmd, iova_t iova, size_t len);

/**
* nvme_mapv_prp - Set up the Physical Region Pages in the data pointer of
* the command from an iovec.
* @ctrl: &struct nvme_ctrl
* @prplist: The first PRP list page address
* @prplist_iova: The first PRP list iova address
* @prplists: contiguous array of PRP list pages, see nvme_map_prp()
* @nprplists: number of pages in @prplists
* @cmd: NVMe command prototype (&union nvme_cmd)
* @iov: array of iovecs
* @niov: number of iovec in @iovec
Expand All @@ -141,17 +154,19 @@ int nvme_map_prp(struct nvme_ctrl *ctrl, leint64_t *prplist, union nvme_cmd *cmd
* allowed to be unaligned, but the entry MUST end on a page boundary. All
* subsequent entries MUST be page aligned.
*
* See nvme_map_prp() for the chaining semantics and capacity of @prplists.
*
* Return: ``0`` on success, ``-1`` on error and sets errno.
*/
int nvme_mapv_prp(struct nvme_ctrl *ctrl, leint64_t *prplist, iova_t prplist_iova,
int nvme_mapv_prp(struct nvme_ctrl *ctrl, leint64_t *prplists, int nprplists,
union nvme_cmd *cmd, struct iovec *iov, int niov);

/**
* nvme_mapv_iova_prp - Set up the Physical Region Pages in the data pointer of
* the command from an iova_vec.
* @ctrl: &struct nvme_ctrl
* @prplist: The first PRP list page address
* @prplist_iova: The first PRP list iova address
* @prplists: contiguous array of PRP list pages, see nvme_map_prp()
* @nprplists: number of pages in @prplists
* @cmd: NVMe command prototype (&union nvme_cmd)
* @iov: array of iova_vecs
* @niov: number of iova_vec in @iovec
Expand All @@ -160,9 +175,11 @@ int nvme_mapv_prp(struct nvme_ctrl *ctrl, leint64_t *prplist, iova_t prplist_iov
* allowed to be unaligned, but the entry MUST end on a page boundary. All
* subsequent entries MUST be page aligned.
*
* See nvme_map_prp() for the chaining semantics and capacity of @prplists.
*
* Return: ``0`` on success, ``-1`` on error and sets errno.
*/
int nvme_mapv_iova_prp(struct nvme_ctrl *ctrl, leint64_t *prplist, iova_t prplist_iova,
int nvme_mapv_iova_prp(struct nvme_ctrl *ctrl, leint64_t *prplists, int nprplists,
union nvme_cmd *cmd, struct iova_vec *iov, int niov);

/**
Expand Down
6 changes: 3 additions & 3 deletions src/nvme/rq.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,19 +41,19 @@
int nvme_rq_map_prp(struct nvme_ctrl *ctrl, struct nvme_rq *rq, union nvme_cmd *cmd,
iova_t iova, size_t len)
{
return nvme_map_prp(ctrl, rq->page.vaddr, cmd, iova, len);
return nvme_map_prp(ctrl, rq->page.vaddr, 1, cmd, iova, len);
}

int nvme_rq_mapv_prp(struct nvme_ctrl *ctrl, struct nvme_rq *rq, union nvme_cmd *cmd,
struct iovec *iov, int niov)
{
return nvme_mapv_prp(ctrl, rq->page.vaddr, rq->page.iova, cmd, iov, niov);
return nvme_mapv_prp(ctrl, rq->page.vaddr, 1, cmd, iov, niov);
}

int nvme_rq_mapv_iova_prp(struct nvme_ctrl *ctrl, struct nvme_rq *rq, union nvme_cmd *cmd,
struct iova_vec *iov, int niov)
{
return nvme_mapv_iova_prp(ctrl, rq->page.vaddr, rq->page.iova, cmd, iov, niov);
return nvme_mapv_iova_prp(ctrl, rq->page.vaddr, 1, cmd, iov, niov);
}

int nvme_rq_mapv_sgl(struct nvme_ctrl *ctrl, struct nvme_rq *rq, union nvme_cmd *cmd,
Expand Down
73 changes: 72 additions & 1 deletion src/nvme/rq_test.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@

#define __max_prps 513

/* for mps=0: one prplist page holds this many entries */
#define __max_prps_per_page (1 << (__mps_to_pageshift(0) - 3))

bool iommu_translate_vaddr(struct iommu_ctx *ctx UNUSED, void *vaddr, iova_t *iova)
{
*iova = (uint64_t)vaddr;
Expand Down Expand Up @@ -64,14 +67,22 @@ int main(void)
struct iovec iov[8];
struct iova_vec iovav[8];

plan_tests(179);
/* multi-page prplist: two contiguous mapped pages used as one array */
leint64_t *mprplists;
void *mppages;

plan_tests(179 + 12);

assert(pgmap((void **)&rq.page.vaddr, __VFN_PAGESIZE) > 0);

rq.page.iova = (uint64_t)rq.page.vaddr;
prplist = rq.page.vaddr;
sglds = rq.page.vaddr;

/* two contiguous pages for multi-page prplist chaining */
assert(pgmapn(&mppages, 2, __VFN_PAGESIZE) > 0);
mprplists = mppages;

/* test 512b aligned */
memset((void *)prplist, 0x0, __VFN_PAGESIZE);
ok1(nvme_rq_map_prp(&ctrl, &rq, &cmd, (iova_t)0x1000000, 0x200) == 0);
Expand Down Expand Up @@ -329,6 +340,7 @@ int main(void)
ok1(le64_to_cpu(cmd.dptr.prp2) == 0x1001000);



/*
* PRPs with iova tests
*/
Expand Down Expand Up @@ -474,6 +486,65 @@ int main(void)
ok1(nvme_rq_mapv_iova_prp(&ctrl, &rq, &cmd, iovav, 2) == -1);


/*
* Multi-page prplist chaining (direct nvme_map_prp/nvme_mapv_prp with two
* pages). For mps=0, max_prps is 512 and two pages hold up to
* (2 - 1) * (512 - 1) + 512 = 1023 data PRPs.
*/

/*
* (__max_prps_per_page + 1) PRPs (prp1 + __max_prps_per_page list entries)
* overflow a single page. With two pages the list is chained: the last list
* entry is written to the second page, and the first page's last slot holds
* the chain link.
*/
memset(mppages, 0x0, 2 * __VFN_PAGESIZE);
ok1(nvme_map_prp(&ctrl, mprplists, 2, &cmd, (iova_t)0x1000000,
(__max_prps_per_page + 1) * 0x1000) == 0);
ok1(le64_to_cpu(cmd.dptr.prp1) == 0x1000000);
ok1(le64_to_cpu(cmd.dptr.prp2) == (uint64_t)mprplists);
/* last slot of the first page chains to the second page */
ok1(le64_to_cpu(mprplists[__max_prps_per_page - 1]) ==
(uint64_t)mprplists + __VFN_PAGESIZE);
/* the chained entry on the second page is the last list PRP */
ok1(le64_to_cpu(mprplists[__max_prps_per_page]) ==
0x1000000 + (uint64_t)__max_prps_per_page * 0x1000);

/*
* Two pages hold at most __max_prps_per_page + (__max_prps_per_page - 1)
* list PRPs (i.e. that many + 1 total). Requiring one more total PRP must
* fail.
*/
memset(mppages, 0x0, 2 * __VFN_PAGESIZE);
ok1(nvme_map_prp(&ctrl, mprplists, 2, &cmd, (iova_t)0x1000000,
(2 * __max_prps_per_page) * 0x1000 + 0x1000) == -1);

/*
* Multi-page chaining across iovec segments: the first iovec fills the
* first page and spills one entry onto the second page, the second iovec
* appends a further entry on the second page.
*/
memset(mppages, 0x0, 2 * __VFN_PAGESIZE);
iov[0] = (struct iovec) {
.iov_base = (void *)0x1000000,
.iov_len = (__max_prps_per_page + 1) * 0x1000,
};
iov[1] = (struct iovec) {
.iov_base = (void *)(0x1000000 + (__max_prps_per_page + 1) * 0x1000),
.iov_len = 0x1000,
};
ok1(nvme_mapv_prp(&ctrl, mprplists, 2, &cmd, iov, 2) == 0);
ok1(le64_to_cpu(cmd.dptr.prp1) == 0x1000000);
ok1(le64_to_cpu(cmd.dptr.prp2) == (uint64_t)mprplists);
/* first page chains to the second */
ok1(le64_to_cpu(mprplists[__max_prps_per_page - 1]) ==
(uint64_t)mprplists + __VFN_PAGESIZE);
/* second page: first entry is the spill from iov[0], next is iov[1] */
ok1(le64_to_cpu(mprplists[__max_prps_per_page]) ==
0x1000000 + (uint64_t)__max_prps_per_page * 0x1000);
ok1(le64_to_cpu(mprplists[__max_prps_per_page + 1]) ==
0x1000000 + (uint64_t)(__max_prps_per_page + 1) * 0x1000);

/*
* SGLs
*/
Expand Down
Loading
Loading