Skip to content

[13.4-stable] pillar/containerd: drop bogus scheduler from debug container exec - #6248

Merged
rene merged 1 commit into
lf-edge:13.4-stablefrom
christoph-zededa:13.4-collectinfo_edgesync_fix_scheduler_policy
Aug 3, 2026
Merged

[13.4-stable] pillar/containerd: drop bogus scheduler from debug container exec#6248
rene merged 1 commit into
lf-edge:13.4-stablefrom
christoph-zededa:13.4-collectinfo_edgesync_fix_scheduler_policy

Conversation

@christoph-zededa

@christoph-zededa christoph-zededa commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Description

Backport of #6231 to 13.4-stable.

The process spec used for execs into the debug container sets
Scheduler.Deadline, but leaves Scheduler.Policy empty. runc rejects that in
ToSchedAttr(), so the container init fails before it can execute the requested
program:

OCI runtime exec failed: exec failed: unable to start container process: invalid scheduler policy:: unknown

runc ignored process.scheduler on the exec path until v1.3.0-rc1
(opencontainers/runc#4585), so the bogus spec stayed unnoticed until the rootfs
moved from runc 1.1.12 to 1.3.3. This branch ships runc 1.3.3, so it is
affected.

Rather than completing the scheduler spec, this drops it. Scheduler.Deadline
is a SCHED_DEADLINE bandwidth parameter in nanoseconds, not a limit on how
long a process may run, and it was assigned a unix timestamp in seconds, so it
never did what it looks like it does — the kernel does not kill tasks for
missing a deadline either. The timeout that matters is enforced by the exec
wrapper itself, which kills the process once its timer fires. See #6231 for the
full rationale.

Adapted cherry-pick. Unlike the other three branches of this backport set,
13.4-stable has neither pkg/pillar/containerd/run.go nor the collectinfo
agent — the same bogus spec sits inline in runInDebugContainer() in
pkg/pillar/agentlog/http-debug.go, so the identical three-line hunk was
dropped there instead. The -x reference to
b7d446377a5a69712d9c1f57427961b9bdc042aa (the squash-merge of #6231 on
master) is preserved in the commit message, together with a note recording the
adaptation. Consequences for this branch:

  • only the bpftrace endpoint of pillar's http-debug interface is affected here
  • collect info is unaffected: this branch has no collectinfo agent, so there is
    no local-operator-console collect-info path going through an OCI exec

How to test and validate this PR

Not covered by an automated test — the path needs a real containerd/runc, and
there is currently no unit or Eden coverage for execs into the debug container.

Run a bpftrace script through pillar's http-debug interface:

# on the device
eve http-debug
# on the dev host - the debug server only listens on localhost, so tunnel it
ssh -p 2222 -L 6543:localhost:6543 root@127.1
bpftrace-compiler run-via-http 127.1:6543 examples/opensnoop.bt

Without this PR the response is
Error happened: process start failed: … invalid scheduler policy:: unknown;
with it the program runs and its JSON output is returned.

Note that bpftrace-compiler run-via-ssh does not exercise this path and
works either way: sshd runs inside the debug container itself, so the program
is started as a plain child of the ssh session, without an OCI exec at all.
Only run-via-http and run-via-edgeview POST to /debug/bpftrace and
therefore go through runInDebugContainer().

Changelog notes

Fixed pillar's bpftrace debug endpoint, which failed to start with an
invalid scheduler policy error from the container runtime.

PR Backports

All four current LTS branches ship runc 1.3.3, so all of them are affected
(verified in #6231 by extracting /usr/bin/runc from the linuxkit/runc image
each branch pins). This PR is one of a set of four:

Checklist

  • I've provided a proper description
  • I've added the proper documentation — not needed, no user-facing behaviour
    or config surface changes
  • I've tested my PR on amd64 device — not tested on this branch; the
    equivalent change was verified on amd64 in pillar/containerd: drop bogus scheduler from debug container exec #6231. Since the hunk had to be
    re-applied to a different file here, this one deserves an actual run of the
    bpftrace reproduction above before it leaves draft
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

For backport PRs (remove it if it's not a backport):

  • I've added a reference link to the original PR
  • PR's title follows the template

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

The process spec used for execs into the debug container sets
Scheduler.Deadline, but leaves Scheduler.Policy empty. runc rejects
that in ToSchedAttr(), so the container init fails before it can
execute the requested program:

  OCI runtime exec failed: exec failed: unable to start container
  process: invalid scheduler policy:: unknown

Both users of RunInDebugContainer() are affected: the collect-info run
triggered through a local operator console (collectinfo agent) and the
bpftrace endpoint of pillar's http-debug interface.

The spec has looked like this since the bpftrace interface was added,
but runc ignored process.scheduler on the exec path until v1.3.0-rc1
(runc#4585), so it stayed unnoticed until the rootfs moved from runc
1.1.12 to 1.3.3 in aa18688 ("bump runc to v3.3.0, containerd to
v2.2.0; addresses critical CVEs").

Rather than completing the scheduler spec, drop it. Scheduler.Deadline
is a SCHED_DEADLINE bandwidth parameter in nanoseconds, not a limit on
how long a process may run, and it was assigned a unix timestamp in
seconds, so it never did what it looks like it does - the kernel does
not kill tasks for missing a deadline either. The timeout is enforced
by RunInDebugContainer() itself, which kills the process once its timer
fires. Without process.scheduler runc leaves the exec'ed process with
the scheduling attributes it inherits from the debug container, which
is what CtrExec() relies on as well.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Christoph Ostarek <christoph@zededa.com>
(cherry picked from commit b7d4463)

[ 13.4-stable: this branch has neither containerd/run.go nor the
  collectinfo agent - the same bogus spec sits inline in
  runInDebugContainer() in agentlog/http-debug.go, so the identical
  hunk was dropped there instead. Only the bpftrace endpoint of
  pillar's http-debug interface is affected on this branch. ]
@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 32.66%. Comparing base (8077a3d) to head (9294c51).
⚠️ Report is 224 commits behind head on 13.4-stable.

Additional details and impacted files
@@               Coverage Diff               @@
##           13.4-stable    #6248      +/-   ##
===============================================
+ Coverage        24.78%   32.66%   +7.88%     
===============================================
  Files                8        9       +1     
  Lines             1138     1090      -48     
===============================================
+ Hits               282      356      +74     
+ Misses             788      654     -134     
- Partials            68       80      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@christoph-zededa
christoph-zededa marked this pull request as ready for review August 3, 2026 10:45
@rene
rene merged commit b7fa157 into lf-edge:13.4-stable Aug 3, 2026
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants