Skip to content

A native-sim eval hangs silently on a headless box instead of naming MUJOCO_GL #617

Description

@vertix

On a headless Linux box, a native-sim eval hangs instead of running, and says nothing about why:

positronic eval run --eval=.sim.positronic.stack_cubes --eval.trial_count=3 \
  --policy=.remote --policy.url=localhost:8010 --output_dir=~/out

It sat for 7 minutes at ~2% CPU with no episode written and no error. The same command with MUJOCO_GL=egl
in front of it finished three episodes in 49 s. The box has no display (DISPLAY empty); MujocoSim constructs
its mj.Renderer with whatever backend mujoco picks by default, which without a display is GLFW.

The backend is pinned everywhere else it is needed, and only the native sim run through the CLI is left out:

  • simulator/libero/launcher.py sets env.setdefault('MUJOCO_GL', 'egl') for the env server it spawns, with a
    comment on exactly this failure — GLFW "wants a display no headless host has";
  • docker/docker-compose.yml sets MUJOCO_GL: egl on the positronic-inference service;
  • the test suites set it themselves (test_inference_integration.py, env_server/tests/test_remote_env.py),
    and the libero-e2e workflow sets osmesa.

So the operator running a native sim eval outside Docker is the one case with nothing pinning it, and the
failure mode is a silent hang rather than a message naming the display.

Worth deciding which of these it should be:

  • have the native sim path pin a headless default the way the LIBERO launcher does (setdefault, so an
    operator's own backend still wins), or
  • leave the backend to the operator but fail fast with a message that names MUJOCO_GL when there is no
    display, instead of blocking in GL setup.

The hang is the part worth fixing either way — the fix is cheap and the symptom costs an operator a long time
to attribute, since nothing in the output points at rendering.

I did not capture a stack from the blocked process, so "blocks in GL context creation" is inferred from the
symptom disappearing under MUJOCO_GL=egl rather than observed directly.

Found while running a live-policy eval for #591; unrelated to that change.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions