groot-ray-serve: parameterize live rollout into an N_ROLLOUTS fan-out

[IanDJordan]

Summary

Lands two things for the groot-ray-serve template in one diff:

1. a content/quality overhaul of the walkthrough notebook (and a security note in policy_server.py), and
2. a new N_ROLLOUTS fan-out so the headline story — one shared policy fleet, many independent simulators — is actually runnable from a single knob.

New this round — N_ROLLOUTS fan-out

• run_rollout takes a worker_id; GIF/results filenames are now per-worker (was hardcoded to 99, so fanned-out rollouts would clobber each other).
• Step 5 launches N_ROLLOUTS SimActors (each holds one GPU; the Anyscale autoscaler provisions nodes to fit, capped by max-workers). Step 6 collects the list, displays the first success, and falls back to the pre-recorded clip only if none succeed. Cleanup kills every actor.
• N_ROLLOUTS = 1 reproduces the prior single-rollout testing behavior exactly.

Notebook overhaul (correctness / robustness)

• GPU leak fix: Cleanup now kills the SimActor — serve.shutdown() doesn't reclaim it, so it previously pinned a whole GPU + CPU after the run.
• Removed the shell-script layer in the python → bash → python call chain: run_rollout previously built a shell=True bash command string (a Ray actor shelling out to bash, which cd'd and ran the timeout binary before launching python sim_worker.py). It now runs subprocess.run([...], cwd=…, timeout=…) directly — python → python, dropping the bash layer and its shell quoting/injection footgun. The subprocess itself is retained on purpose — Isaac Sim/Omniverse Kit can't boot inside a Ray worker.
• Stale-GIF guard: Step 5/6 requires exit_code == 0 and a fresh file before trusting the live GIF (prevents re-displaying a previous run's clip as if it succeeded); throttled the progress print.
• fixed the "Going further" snippets to the real actor API, made wording consistently "actor" (not "task"), and changed "sub-second latency" → realistic ~200 ms-1.5 s.
• Import/serve fixes: removed a dead sys.path.insert; documented that serve.start(host=0.0.0.0) is load-bearing (binds the proxy so off-node sim workers can reach the policy), not redundant with serve.run.
• policy_server.py: documented the pickle-over-HTTP trust assumption on both predict_http endpoints (comment only, no behavior change).

Validation

• pre-commit run --all-files passes (incl. check-readme byte-exact match and check-build-yaml); no template dependency changes (so check-depsets is unaffected).
• All notebook code cells compile.
• Per-template test (/test-template groot-ray-serve) + dev->staging->prod re-publish to follow after approval.

🤖 Generated with Claude Code