Detect a seccomp-blocked set_thread_area so wibo runs under AWS Lambda

[JackPriceBurns]

Picks up the thread from #113 — running wibo on AWS Lambda, the GC/2.0 mwcceppc dies with SIGSYS during thread setup. It's setThreadArea64: it makes its set_thread_area call via int 0x80, and Lambda's seccomp filter blocks the compat syscall path. The signal lands before the existing modify_ldt fallback can run, since that only triggers when set_thread_area returns an error.

I started from the SIGSYS-handler idea in the #113 review (catch the trap, rewrite the return to -ENOSYS, let the normal fallback take over). It's neat, but it doesn't actually work on Lambda: the filter there uses a KILL action, not SECCOMP_RET_TRAP, so the SIGSYS is uncatchable and a handler never runs. I confirmed it with a small probe — install an SA_SIGINFO SIGSYS handler, fire int 0x80/set_thread_area, and the process is killed outright (the handler's si_code/si_arch checks never get a chance).

So rather than catch the signal, this detects whether set_thread_area is usable by trying it once in a forked child. If the child gets killed by the filter (or the syscall fails), the parent treats it as unavailable and uses modify_ldt. Forking isolates the trap: a KILL action only takes down the child's process group, so the parent survives either way, and it works for both KILL and TRAP filters.

Tested on Lambda (x86_64): mwcceppc -version and real compiles work and produce correct PPC objects, no env vars or config. The setup log shows the fall-through:

setup_linux: set_thread_area blocked by seccomp, using LDT
setup_linux: Using FS selector 0x17

On a normal machine it's one extra fork at startup (cached, so at most once) and otherwise unchanged. If you'd rather not fork on the common path, I can gate it behind a /proc/self/status Seccomp check so it only probes when a filter is actually installed — happy to add that.

Scope note: this only covers the x86_64 compat path (the int 0x80 in setThreadArea64); the 32-bit build's native set_thread_area is untouched.

[encounter]

I suggested a different approach here: #113 (comment)
But the original author never updated their PR with my suggestions. Feel free to try it

[encounter]

Thanks, I like the fork approach. Could you measure how much impact it has on startup (if any?)

[simonlindholm]

If fork overhead turns out to be measurable, then I wonder how vfork compares

[JackPriceBurns]

On the perf question — I benchmarked it on a normal x86_64 box (no seccomp, so the probe's fork is pure wasted overhead here, which is the worst case for this question). Release builds, hyperfine, pinned to one core.

Three variants:

• before — current main, no probe
• after — this PR (unconditional fork probe)
• gated — the probe behind a /proc/self/status Seccomp check, so it only forks when a filter is actually installed

workload	before	after	gated
mwcceppc -version (near-pure startup)	4.67 ms	4.73 ms (Δ +60 µs)	4.71 ms (Δ +36 µs)
real compile (4 funcs, -O4,p)	12.37 ms	12.41 ms (Δ +34 µs)	12.39 ms (Δ +16 µs)

800 / 400 runs per variant; 1σ ≈ 0.4 ms and 0.5–1.1 ms respectively.

So the probe adds ~60 µs, once, at startup — about 1.3% of a do-nothing -version and ~0.3% of a real compile, and comfortably inside the run-to-run noise. It forks once per process (the result is cached) and only the calling thread is duplicated COW, which is why it barely registers.

Box: Intel Core Ultra 7 265, Ubuntu 24.04, kernel 6.8.

[encounter]

Thanks for verifying!