Skip to content

docs: add headless rendering troubleshooting guide to AGENTS.md #5389

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pv-nvidia wants to merge 1 commit into
base: develop
Choose a base branch
from
+63 −0
Open

docs: add headless rendering troubleshooting guide to AGENTS.md #5389

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions AGENTS.md
View file
Open in desktop
Copy link
Copy Markdown
Collaborator

@AntoineRichard AntoineRichard Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like some of these could be moved to troubleshooting of the isaaclab doc. And the agent.md could reference that part of the doc. This would avoid having an Agent.md that's too large. Other teams try to keep this file lean. If you feel otherwise let me know.

Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,69 @@ Follow conventional commit message practices.

- IMPORTANT: Pin actions by SHA hash. Use `action@<sha> # vX.Y.Z` format for supply-chain security. Check existing workflows in `.github/workflows/` for the allowlisted hashes. New actions or versions require repo admin approval to be added to the allowlist.

## Headless Rendering (DGX Cloud / Containers / CI)

When running Isaac Lab with cameras or rendering in headless environments (DGX Cloud, Docker containers, CI), several issues can arise.

### `DISPLAY` environment variable

**Problem**: If `DISPLAY` is set (e.g., `DISPLAY=:99` in `/etc/environment`), Kit tries to use GLX which causes `GLXBadFBConfig` crashes even when no display is available.

**Fix**: Unset `DISPLAY` before running:
```bash
DISPLAY= ./isaaclab.sh -p -m pytest ...
# or
unset DISPLAY
```

### EGL vs Vulkan

**Problem**: `eglInitialize` fails if `nvidia_drm` kernel module is not loaded (common in containers without GPU display passthrough).

**Diagnosis**:
```bash
# Check if nvidia_drm is loaded
lsmod | grep nvidia_drm
# Check for render nodes
ls -la /dev/dri/render*
# Check modeset parameter
cat /sys/module/nvidia_drm/parameters/modeset
```

**Note**: Even without EGL, **Vulkan typically works** in headless mode. PhysX, Fabric, Warp, and CUDA all function normally. RTX rendering works via Vulkan without EGL.

### Missing `.kit` file dependencies for headless camera rendering

**Problem**: Camera annotators (RGB, depth, etc.) return empty data in headless mode because required extensions aren't loaded.

**Root cause**: The headless rendering experience file (`apps/isaaclab.python.headless.rendering.kit`) may be missing dependencies needed for the HydraTexture and viewport pipeline.

**Symptoms**:
- `rep.orchestrator.step()` hangs or loops infinitely
- Annotator `get_data()` returns None or zero-filled arrays
- Thousands of `createViewport` calls in Kit log

**Key extensions** needed for headless camera rendering:
- `omni.kit.hydra_texture` — Must be loaded before `omni.replicator.core` imports (availability is cached at import time)
- A viewport infrastructure extension — needed for render product creation

**Note**: `omni.kit.viewport.window` works but pulls in `omni.ui` which is heavyweight. Prefer a lighter alternative if available (see [PR discussions](https://github.com/isaac-sim/IsaacLab/pull/5375) for current status).
Comment on lines +194 to +209
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 .kit section stops short of a concrete fix

The section identifies root cause and symptoms well, but the "Key extensions" list tells readers what is needed without explaining how to add it. Since apps/isaaclab.python.headless.rendering.kit already exists in the repo and is missing omni.kit.hydra_texture in its [dependencies] block, readers will likely end up editing the kit file incorrectly (e.g., adding it after omni.replicator.core when it must come before). A one-line example of the correct dependency ordering in the kit file would prevent that mistake.


### Conda conflicts

**Problem**: `isaaclab.sh` checks `CONDA_PREFIX` before `_isaac_sim/python.sh` — if conda is activated, it picks the conda Python which lacks Isaac Sim packages.
Comment on lines +211 to +213
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Conda section omits VIRTUAL_ENV precedence

The current isaaclab.sh checks VIRTUAL_ENV before CONDA_PREFIX (line 15–16 of the script). Describing it as "CONDA_PREFIX is checked before _isaac_sim/python.sh" is technically correct, but a developer who has both a venv and conda active will be confused when the venv wins instead of conda. Mentioning the full priority chain (VIRTUAL_ENV → CONDA_PREFIX → env_isaaclab → _isaac_sim) would make the section self-contained and accurate without needing readers to inspect the script.


**Fix**: Deactivate conda first:
```bash
conda deactivate
./isaaclab.sh --install
```
Or override: `CONDA_PREFIX= ./isaaclab.sh --install`

### Shared `_isaac_sim` symlink pitfall

When multiple Isaac Lab working copies share the same Isaac Sim build via `_isaac_sim` symlink, `pip install -e` in one overwrites the other's packages in the shared site-packages directory. Always re-run `isaaclab.sh --install` when switching between repos.

## Testing Guidelines

- **Always verify regression tests fail without the fix.** When writing a regression test for a bug fix, temporarily revert the fix and run the test to confirm it fails. Then reapply the fix and verify the test passes. This ensures the test actually covers the bug.
Expand Down