Skip to content

feat(thread): add interrupt mode and FdGroup API wrappers#105

Merged
mergify[bot] merged 5 commits intoopenebs:developfrom
jr42:pr/interrupt-mode
Apr 21, 2026
Merged

feat(thread): add interrupt mode and FdGroup API wrappers#105
mergify[bot] merged 5 commits intoopenebs:developfrom
jr42:pr/interrupt-mode

Conversation

@jr42
Copy link
Copy Markdown
Contributor

@jr42 jr42 commented Apr 12, 2026

Summary

Safe Rust wrappers for SPDK's interrupt mode facility and fd_group
hierarchical event multiplexing API (available since SPDK 24.09),
enabling custom reactors to sleep in fd_group_wait() instead of
busy-polling.

New Thread methods:

  • interrupt_mode_enable() / interrupt_mode_is_enabled(): global init and query
  • set_interrupt_mode(enable): per-thread poll/interrupt switching
  • get_interrupt_fd() / get_interrupt_fd_group(): access thread event fds

New FdGroup struct with RAII lifecycle:

  • create() / Drop: lifecycle management
  • wait(timeout): block until events (fd_group_wait)
  • nest(child) / unnest(child): hierarchical event multiplexing
  • add() / add_with_fd_type(): fd registration with optional auto-drain

New constant:

  • FD_TYPE_EVENTFD: marks eventfds for auto-draining in fd_group_wait()

These APIs enable the mayastor reactor to implement interrupt mode,
reducing CPU from ~1000m per core to <300m when idle.

Depends-On: openebs/spdk#70 (iSCSI + bdev interrupt fixes)
Ref: openebs/mayastor#1745

Test plan

  • Builds cleanly with updated libspdk
  • Used by mayastor reactor interrupt mode (73/86 pytest tests pass)
  • Production validated on 3-node cluster (16 volumes, 10+ days)
  • No regressions in poll mode

@jr42 jr42 requested a review from a team as a code owner April 12, 2026 19:13
This was referenced Apr 12, 2026
Merged
Closed
@jr42 jr42 force-pushed the pr/interrupt-mode branch from f97d1d7 to 45647ff Compare April 13, 2026 19:27
tiagolobocastro
tiagolobocastro approved these changes Apr 14, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@tiagolobocastro tiagolobocastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll have to merge the dependent PR before this one, I'll try to review it a little later today

Comment thread src/thread.rs Outdated
Comment thread src/thread.rs
Comment thread src/thread.rs Outdated
@jr42
feat(thread): add interrupt mode and FdGroup API wrappers
419c2ef 
Add safe Rust wrappers for SPDK's interrupt mode facility (available
since SPDK 24.09) and the fd_group hierarchical event multiplexing API.

New Thread methods:
- interrupt_mode_enable/is_enabled: global init and query
- set_interrupt_mode: per-thread poll/interrupt switching
- get_interrupt_fd/get_interrupt_fd_group: access thread event fds

New FdGroup struct with RAII lifecycle:
- create/wait/nest/unnest: hierarchical event multiplexing
- add/add_with_fd_type: fd registration with optional auto-drain

New FD_TYPE_EVENTFD constant for auto-draining eventfds in
fd_group_wait.

These APIs enable custom reactors to sleep in fd_group_wait() instead
of busy-polling, reducing CPU usage from ~100% to near-zero when idle.

Ref: openebs/mayastor#1745
Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
@jr42 jr42 force-pushed the pr/interrupt-mode branch 4 times, most recently from da98c7a to 5c30902 Compare April 18, 2026 10:12
jr42 added a commit to jr42/mayastor that referenced this pull request Apr 18, 2026
@jr42
chore: bump spdk-rs for interrupt mode APIs
9f0aa98 
Update spdk-rs submodule to include FdGroup wrapper and Thread
interrupt mode API wrappers required by the reactor interrupt mode
implementation. spdk-rs also bumps its libspdk nix pin to pull in
the iSCSI poll group interrupt registration and bdev
wait_for_examine periodic poller fixes on the SPDK side
(openebs/spdk#70), required for interrupt mode with custom
reactors.

Depends-On: openebs/spdk-rs#105
Depends-On: openebs/spdk#70
Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
@jr42 jr42 force-pushed the pr/interrupt-mode branch from 5c30902 to 56f5dea Compare April 18, 2026 11:47
jr42 added a commit to jr42/mayastor that referenced this pull request Apr 18, 2026
@jr42
chore: bump spdk-rs for interrupt mode APIs
736de11 
Update spdk-rs submodule to include FdGroup wrapper and Thread
interrupt mode API wrappers required by the reactor interrupt mode
implementation. spdk-rs also bumps its libspdk nix pin to pull in
the iSCSI poll group interrupt registration and bdev
wait_for_examine periodic poller fixes on the SPDK side
(openebs/spdk#70), required for interrupt mode with custom
reactors.

Depends-On: openebs/spdk-rs#105
Depends-On: openebs/spdk#70
Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
jr42 added a commit to jr42/spdk-rs that referenced this pull request Apr 18, 2026
@jr42
refactor(thread): return nix::Errno from FdGroup methods
807f41e 
Addresses openebs#105 review from @tiagolobocastro
("Should we return `nix::Error` here? and throughout"). Converts
`FdGroup::create`, `add`, `add_with_fd_type`, `nest`, `unnest`
and `Thread::interrupt_mode_enable` from `Result<_, i32>` to
`Result<_, Errno>`, matching the idiom used elsewhere in this
crate (`src/bdev.rs`, `src/ffihelper.rs`).

Left `FdGroup::wait()` on raw `i32` — it returns a non-negative
event count on success, so the Result-shape doesn't fit.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
jr42 added a commit to jr42/mayastor that referenced this pull request Apr 18, 2026
@jr42
refactor(reactor): migrate FdGroup results to Errno
bbad3c9 
Addresses PR openebs#1966 review (openebs/spdk-rs#105 review thread from
@tiagolobocastro). spdk-rs' FdGroup methods and
`Thread::interrupt_mode_enable` now return `Result<_, nix::Errno>`
instead of `Result<_, i32>`, matching the idiom used elsewhere in
that crate. Update every call site in reactor.rs to drop the `rc`
positional formatting and use the `Errno`'s Display impl directly
in error messages, and tighten the interrupt_mode_enable startup
path to `.expect(...)` rather than `assert_eq!(rc, 0, ...)`.

Bumps spdk-rs submodule to the commit that introduces the new
signatures.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
jr42 added a commit to jr42/spdk-rs that referenced this pull request Apr 18, 2026
@jr42
refactor(thread): drop unused FdGroup owned field
c93e468 
Addresses openebs#105 review from @tiagolobocastro. Only
`FdGroup::create()` constructs the struct and it always sets
`owned: true`; Drop now unconditionally destroys.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
jr42 added a commit to jr42/mayastor that referenced this pull request Apr 18, 2026
@jr42
refactor(reactor): migrate FdGroup results to Errno
c6c0508 
Addresses PR openebs#1966 review (openebs/spdk-rs#105 review thread from
@tiagolobocastro). spdk-rs' FdGroup methods and
`Thread::interrupt_mode_enable` now return `Result<_, nix::Errno>`
instead of `Result<_, i32>`, matching the idiom used elsewhere in
that crate. Update every call site in reactor.rs to drop the `rc`
positional formatting and use the `Errno`'s Display impl directly
in error messages, and tighten the interrupt_mode_enable startup
path to `.expect(...)` rather than `assert_eq!(rc, 0, ...)`.

Bumps spdk-rs submodule to the commit that introduces the new
signatures.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
jr42 added a commit to jr42/spdk-rs that referenced this pull request Apr 18, 2026
@jr42
refactor(thread): expose SPDK_FD_TYPE via bindgen
b3d54b4 
Addresses openebs#105 review from @tiagolobocastro. Adds
`spdk_fd_type` to the bindgen allowlist so `SPDK_FD_TYPE_EVENTFD`
lands in `libspdk.rs` as a module-level const — same pattern as
`SPDK_BDEV_IO_TYPE_*` / `SPDK_BDEV_RESET_STAT_*`. Drops the mirror
constant from the public thread API.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
jr42 added a commit to jr42/mayastor that referenced this pull request Apr 18, 2026
@jr42
refactor(reactor): use libspdk::SPDK_FD_TYPE_EVENTFD
b64c44c 
Follow-up to openebs/spdk-rs#105 exposing the SPDK fd_type enum
directly via bindgen. Drops the `spdk_rs::FD_TYPE_EVENTFD` mirror
reference in favour of `spdk_rs::libspdk::SPDK_FD_TYPE_EVENTFD`.

Bumps spdk-rs submodule.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
jr42 added a commit to jr42/spdk-rs that referenced this pull request Apr 20, 2026
@jr42
refactor(thread): return nix::Errno from FdGroup methods
c20c446 
Addresses openebs#105 review from @tiagolobocastro
("Should we return `nix::Error` here? and throughout"). Converts
`FdGroup::create`, `add`, `add_with_fd_type`, `nest`, `unnest`
and `Thread::interrupt_mode_enable` from `Result<_, i32>` to
`Result<_, Errno>`, matching the idiom used elsewhere in this
crate (`src/bdev.rs`, `src/ffihelper.rs`).

Left `FdGroup::wait()` on raw `i32` — it returns a non-negative
event count on success, so the Result-shape doesn't fit.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
jr42 added a commit to jr42/spdk-rs that referenced this pull request Apr 20, 2026
@jr42
refactor(thread): drop unused FdGroup owned field
91d2cbb 
Addresses openebs#105 review from @tiagolobocastro. Only
`FdGroup::create()` constructs the struct and it always sets
`owned: true`; Drop now unconditionally destroys.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
jr42 added a commit to jr42/spdk-rs that referenced this pull request Apr 20, 2026
@jr42
refactor(thread): expose SPDK_FD_TYPE via bindgen
0b98a64 
Addresses openebs#105 review from @tiagolobocastro. Adds
`spdk_fd_type` to the bindgen allowlist so `SPDK_FD_TYPE_EVENTFD`
lands in `libspdk.rs` as a module-level const — same pattern as
`SPDK_BDEV_IO_TYPE_*` / `SPDK_BDEV_RESET_STAT_*`. Drops the mirror
constant from the public thread API.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
@jr42 jr42 force-pushed the pr/interrupt-mode branch from b3d54b4 to 0b98a64 Compare April 20, 2026 22:19
jr42 added a commit to jr42/mayastor that referenced this pull request Apr 20, 2026
@jr42
refactor(reactor): use libspdk::SPDK_FD_TYPE_EVENTFD
edbd617 
Follow-up to openebs/spdk-rs#105 exposing the SPDK fd_type enum
directly via bindgen. Drops the `spdk_rs::FD_TYPE_EVENTFD` mirror
reference in favour of `spdk_rs::libspdk::SPDK_FD_TYPE_EVENTFD`.

Bumps spdk-rs submodule.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
tiagolobocastro
tiagolobocastro reviewed Apr 21, 2026
View reviewed changes
Comment thread nix/pkgs/libspdk/default.nix Outdated
jr42 added 4 commits April 21, 2026 06:56
@jr42
chore(nix): bump libspdk rev for interrupt mode fixes
df051b2 
Update SPDK revision to include iSCSI poll group interrupt
registration and bdev wait_for_examine periodic poller fix, both
required for interrupt mode with custom reactors.

Pin targets jr42/spdk@pr/interrupt-mode-fixes (approved but not yet
merged upstream). When openebs/spdk#70 lands on
openebs/v25.05.x-mayastor, flip owner back to "openebs" and update
rev/sha256 to the merged commit.

Depends-On: openebs/spdk#70
Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
@jr42
refactor(thread): return nix::Errno from FdGroup methods
9d2587b 
Addresses openebs#105 review from @tiagolobocastro
("Should we return `nix::Error` here? and throughout"). Converts
`FdGroup::create`, `add`, `add_with_fd_type`, `nest`, `unnest`
and `Thread::interrupt_mode_enable` from `Result<_, i32>` to
`Result<_, Errno>`, matching the idiom used elsewhere in this
crate (`src/bdev.rs`, `src/ffihelper.rs`).

Left `FdGroup::wait()` on raw `i32` — it returns a non-negative
event count on success, so the Result-shape doesn't fit.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
@jr42
refactor(thread): drop unused FdGroup owned field
74532c8 
Addresses openebs#105 review from @tiagolobocastro. Only
`FdGroup::create()` constructs the struct and it always sets
`owned: true`; Drop now unconditionally destroys.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
@jr42
refactor(thread): expose SPDK_FD_TYPE via bindgen
d74334f 
Addresses openebs#105 review from @tiagolobocastro. Adds
`spdk_fd_type` to the bindgen allowlist so `SPDK_FD_TYPE_EVENTFD`
lands in `libspdk.rs` as a module-level const — same pattern as
`SPDK_BDEV_IO_TYPE_*` / `SPDK_BDEV_RESET_STAT_*`. Drops the mirror
constant from the public thread API.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
@jr42 jr42 force-pushed the pr/interrupt-mode branch from 0b98a64 to d74334f Compare April 21, 2026 04:56
jr42 added a commit to jr42/mayastor that referenced this pull request Apr 21, 2026
@jr42
refactor(reactor): use libspdk::SPDK_FD_TYPE_EVENTFD
3c36276 
Follow-up to openebs/spdk-rs#105 exposing the SPDK fd_type enum
directly via bindgen. Drops the `spdk_rs::FD_TYPE_EVENTFD` mirror
reference in favour of `spdk_rs::libspdk::SPDK_FD_TYPE_EVENTFD`.

Bumps spdk-rs submodule.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
@tiagolobocastro
Copy link
Copy Markdown
Member

tiagolobocastro commented Apr 21, 2026

@mergify queue

@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented Apr 21, 2026

Deprecation notice: This pull request comes from a fork and was queued with update_method=rebase and update_bot_account impersonation. This capability will be removed on July 1, 2026. After this date, the merge queue will no longer be able to rebase fork pull requests with this configuration. To avoid disruption, switch to update_method=merge in your queue rule.

@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented Apr 21, 2026 •
edited
Loading

Merge Queue Status

  • ✅ Entered queue — 2026-04-21 09:29 UTC · Rule: default
  • ✅ Checks skipped · PR is already up-to-date
  • ✅ Merged — 2026-04-21 09:29 UTC · at d74334f313213ffd7bedbb0fe47c725d953eb99a

This pull request spent 12 seconds in the queue, including 3 seconds running CI.

Required conditions to merge
  • #approved-reviews-by >= 1 [🛡 GitHub branch protection]
  • #changes-requested-reviews-by = 0 [🛡 GitHub branch protection]
  • #review-threads-unresolved = 0 [🛡 GitHub branch protection]
  • branch-protection-review-decision = APPROVED [🛡 GitHub branch protection]
  • check-success = code-lint
  • check-success = commitlint
  • any of [🛡 GitHub branch protection]:
    • check-success = DCO
    • check-neutral = DCO
    • check-skipped = DCO
  • any of [🛡 GitHub branch protection]:
    • check-success = rust-lint (ubuntu-latest)
    • check-neutral = rust-lint (ubuntu-latest)
    • check-skipped = rust-lint (ubuntu-latest)
  • any of [🛡 GitHub branch protection]:
    • check-success = rust-lint (github-arm64-2c-8gb)
    • check-neutral = rust-lint (github-arm64-2c-8gb)
    • check-skipped = rust-lint (github-arm64-2c-8gb)
  • any of [🛡 GitHub branch protection]:
    • check-success = commitlint
    • check-neutral = commitlint
    • check-skipped = commitlint
  • any of [🛡 GitHub branch protection]:
    • check-success = nix-lint
    • check-neutral = nix-lint
    • check-skipped = nix-lint
  • any of [🛡 GitHub branch protection]:
    • check-success = rust-dev (github-arm64-2c-8gb)
    • check-neutral = rust-dev (github-arm64-2c-8gb)
    • check-skipped = rust-dev (github-arm64-2c-8gb)
  • any of [🛡 GitHub branch protection]:
    • check-success = rust-dev (ubuntu-latest)
    • check-neutral = rust-dev (ubuntu-latest)
    • check-skipped = rust-dev (ubuntu-latest)
  • any of [🛡 GitHub branch protection]:
    • check-neutral = Mergify Merge Protections
    • check-skipped = Mergify Merge Protections
    • check-success = Mergify Merge Protections

@mergify mergify Bot added the queued label Apr 21, 2026
@mergify mergify Bot merged commit f1c6701 into openebs:develop Apr 21, 2026
12 checks passed
@mergify mergify Bot removed the queued label Apr 21, 2026
jr42 added a commit to jr42/mayastor that referenced this pull request Apr 21, 2026
@jr42
refactor(reactor): use libspdk::SPDK_FD_TYPE_EVENTFD
350c83a 
Follow-up to openebs/spdk-rs#105 exposing the SPDK fd_type enum
directly via bindgen. Drops the `spdk_rs::FD_TYPE_EVENTFD` mirror
reference in favour of `spdk_rs::libspdk::SPDK_FD_TYPE_EVENTFD`.

Bumps spdk-rs submodule.

Signed-off-by: Jeremias Reith <jr42@users.noreply.github.com>
bors-openebs-mayastor Bot pushed a commit to openebs/mayastor that referenced this pull request Apr 24, 2026
@jr42
Merge #1966
04a3811 
1966: feat(reactor): add SPDK interrupt mode support r=tiagolobocastro a=jr42

## Summary

Opt-in SPDK interrupt mode for io-engine: reactors sleep in
`fd_group_wait()` instead of busy-polling, reducing CPU from ~1000m
per core to <300m when idle.

- Enable with `ENABLE_INTERRUPT_MODE=true` (default: off, backward compatible)
- Follows Longhorn v2 hybrid pattern (LEP 2025-07-21): epoll for NVMe-oF
  TCP targets, timerfd polling for NVMe initiators
- Includes pytest compose wiring for interrupt mode testing

### Implementation

Reactor changes (`reactor.rs`, +271 lines):
- New `ReactorState::Interrupt` with `fd_group_wait`-based event loop
- Reactor-level `FdGroup` nests all thread fd_groups for hierarchical mux
- Wakeup eventfd (`FD_TYPE_EVENTFD`, auto-drained) for Rust future delivery
- Cross-core wake on thread schedule to prevent multi-core init deadlock
- Late fd_group nesting in `add_incoming()` for dynamic thread assignment
- Clean shutdown path restoring poll mode

### Why interrupt mode instead of SPDK's dynamic scheduler (#1745)

Mayastor implements its own reactor loop (`reactor.rs`), bypassing
SPDK's stock reactor entirely. SPDK's dynamic scheduler monitors thread
busyness *within SPDK's reactor* -- since mayastor's reactor replaces
it, the scheduler has nothing to observe or control.

Instead, we implement interrupt mode directly in the custom reactor,
following the same pattern validated by Longhorn v2 (LEP 2025-07-21).
This is simpler, more predictable, and doesn't require restructuring the
reactor to use SPDK's scheduler infrastructure. The dynamic scheduler
remains a future option if the reactor is ever migrated closer to SPDK's
stock implementation.

Depends-On: openebs/spdk-rs#105
Depends-On: openebs/spdk#70
Closes: #1745

## Test plan

- [x] 73/86 pytest tests pass in single-core interrupt mode
      (13 failures are env-related, identical in poll mode)
- [x] Multi-core validated (2-core smoke test)
- [x] Production: 3-node cluster, 16 volumes, CPU ~3000m to ~463m (85% reduction)
- [x] Rolling restart validated: volumes auto-recover, nexuses redistribute
- [ ] CI pipeline passes


Co-authored-by: Jeremias Reith <jr42@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tiagolobocastro tiagolobocastro tiagolobocastro approved these changes

@Abhinandan-Purkait Abhinandan-Purkait Abhinandan-Purkait approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jr42 @tiagolobocastro @Abhinandan-Purkait