On a headless Linux box, a native-sim eval hangs instead of running, and says nothing about why:
positronic eval run --eval=.sim.positronic.stack_cubes --eval.trial_count=3 \
--policy=.remote --policy.url=localhost:8010 --output_dir=~/out
It sat for 7 minutes at ~2% CPU with no episode written and no error. The same command with MUJOCO_GL=egl
in front of it finished three episodes in 49 s. The box has no display (DISPLAY empty); MujocoSim constructs
its mj.Renderer with whatever backend mujoco picks by default, which without a display is GLFW.
The backend is pinned everywhere else it is needed, and only the native sim run through the CLI is left out:
simulator/libero/launcher.py sets env.setdefault('MUJOCO_GL', 'egl') for the env server it spawns, with a
comment on exactly this failure — GLFW "wants a display no headless host has";
docker/docker-compose.yml sets MUJOCO_GL: egl on the positronic-inference service;
- the test suites set it themselves (
test_inference_integration.py, env_server/tests/test_remote_env.py),
and the libero-e2e workflow sets osmesa.
So the operator running a native sim eval outside Docker is the one case with nothing pinning it, and the
failure mode is a silent hang rather than a message naming the display.
Worth deciding which of these it should be:
- have the native sim path pin a headless default the way the LIBERO launcher does (
setdefault, so an
operator's own backend still wins), or
- leave the backend to the operator but fail fast with a message that names
MUJOCO_GL when there is no
display, instead of blocking in GL setup.
The hang is the part worth fixing either way — the fix is cheap and the symptom costs an operator a long time
to attribute, since nothing in the output points at rendering.
I did not capture a stack from the blocked process, so "blocks in GL context creation" is inferred from the
symptom disappearing under MUJOCO_GL=egl rather than observed directly.
Found while running a live-policy eval for #591; unrelated to that change.
On a headless Linux box, a native-sim eval hangs instead of running, and says nothing about why:
It sat for 7 minutes at ~2% CPU with no episode written and no error. The same command with
MUJOCO_GL=eglin front of it finished three episodes in 49 s. The box has no display (
DISPLAYempty);MujocoSimconstructsits
mj.Rendererwith whatever backend mujoco picks by default, which without a display is GLFW.The backend is pinned everywhere else it is needed, and only the native sim run through the CLI is left out:
simulator/libero/launcher.pysetsenv.setdefault('MUJOCO_GL', 'egl')for the env server it spawns, with acomment on exactly this failure — GLFW "wants a display no headless host has";
docker/docker-compose.ymlsetsMUJOCO_GL: eglon thepositronic-inferenceservice;test_inference_integration.py,env_server/tests/test_remote_env.py),and the libero-e2e workflow sets
osmesa.So the operator running a native sim eval outside Docker is the one case with nothing pinning it, and the
failure mode is a silent hang rather than a message naming the display.
Worth deciding which of these it should be:
setdefault, so anoperator's own backend still wins), or
MUJOCO_GLwhen there is nodisplay, instead of blocking in GL setup.
The hang is the part worth fixing either way — the fix is cheap and the symptom costs an operator a long time
to attribute, since nothing in the output points at rendering.
I did not capture a stack from the blocked process, so "blocks in GL context creation" is inferred from the
symptom disappearing under
MUJOCO_GL=eglrather than observed directly.Found while running a live-policy eval for #591; unrelated to that change.