Add Newton deformable object API (modular, contrib-based)#5383
Add Newton deformable object API (modular, contrib-based)#5383xiangdonglai wants to merge 19 commits intoisaac-sim:developfrom
Conversation
…pawners, schemas)
…article sync Create isaaclab_experimental/deformable/ module containing: - DeformableObject and DeformableObjectData (Newton backend) - Warp kernels for particle gather/scatter operations - CoupledSolver for rigid-body + VBD cloth interaction - VBDSolverCfg, CoupledSolverCfg, NewtonModelCfg configs - Solver factory functions registered with NewtonManager - Particle sync for USD/Fabric viewport rendering - Cloner hooks for deformable body replication - Model cfg hook for post-finalize parameter application Bug fixes included: - init_pos/init_rot zeroed after Xform bake to prevent double-application - vis_mesh_prim fallback for empty vis_candidates (surface cloth) - No weakref in data class (direct references for particle_q/qd) - model_cfg always applied (not gated behind contact attributes)
Add test_deformable_object.py and test_rigid_deformable_coupling.py under isaaclab_experimental/test/deformable/ with imports adapted for the experimental module layout. Introduce register_hooks() in the deformable __init__ so hooks survive NewtonManager.clear() across test fixtures. Include pre-commit formatting fixes.
Add two deformable object manipulation tasks to isaaclab_tasks_experimental: - Isaac-Pick-Cloth-Direct-v0: Franka robot + cloth (shirt) with coupled MJWarp + VBD solver - Isaac-Pick-VBD-Cube-Direct-v0: Franka robot + deformable cube with coupled MJWarp + VBD solver Both tasks use DeformableNewtonCfg (NewtonCfg subclass with model_cfg field) and import isaaclab_experimental.deformable to trigger hook registration. Also add isaaclab_tasks_experimental import to zero_agent.py for task discovery.
Update the deformable tutorial to support both PhysX and Newton backends via --backend argument. Use core isaaclab.assets imports for backend-agnostic DeformableObject, and import VBDSolverCfg from isaaclab_experimental for the Newton backend path.
Add _get_deformable_ignore_paths() to skip deformable sim/visual mesh prims in builder.add_usd() calls, preventing Newton from creating redundant static mesh colliders for deformable bodies. Also add rebuild_bvh() call in _simulate_physics_only() for solvers that require BVH rebuilds (e.g. VBD cloth).
Gracefully handle particle-only scenes (no rigid bodies) by logging a warning instead of raising RuntimeError when body_label/body_key is empty. Add Fabric particle sync setup for deformable bodies: resize visual mesh topology to match sim mesh, and create per-instance particle offset attributes so the sync kernel can write points.
Defer heavy imports in isaaclab_experimental.deformable behind __getattr__ so pxr is not loaded before Kit starts. Ensure register_hooks() always re-registers after NewtonManager.clear() by resetting the _hooks_registered guard on each explicit call. Fix SurfaceDeformableBodyMaterialCfg class-identity mismatch in from_files.py that caused isinstance() to always fail, making PhysX create volume hierarchies for surface cloth. In deformable_object.py, prefer the original authored Mesh over the PhysX sim_mesh proxy (which has no points at construction time) when a surface deformable material is configured. In newton_replicate.py, pass deformable prim paths as ignore_paths to add_usd so Newton's USD importer skips PhysX proxy meshes.
Replace bare `import isaaclab_experimental.deformable` with explicit `register_hooks()` calls since deferred imports no longer trigger hook registration on import.
The Newton deformable implementation (solvers, kernels, particle sync, cloner hooks) and its tests now live under isaaclab_contrib. All import references in tutorials, task envs, and the isaaclab_newton re-export stub are updated accordingly.
xiangdonglai
requested review from
Mayankm96,
jtigue-bdai,
kellyguo11,
masoudmoghani,
ooctipus and
pascal-roth
as code owners
![isaaclab-review-bot[bot]](https://avatars.githubusercontent.com/u/157846462?s=60&v=4){kind=small}
There was a problem hiding this comment.
🤖 Isaac Lab Review Bot
Summary
This PR adds a modular Newton deformable object API (cloth and volumetric soft bodies) with a plugin architecture: core abstractions in isaaclab, minimal hooks in isaaclab_newton, and all Newton-specific implementation in isaaclab_contrib. The architecture is sound, but there are several correctness issues in state management, potential data races, and missing error handling that need attention before merging.
Architecture Impact
Cross-module impact is significant:
isaaclab.assets.deformable_objectintroduces new base classes and factory pattern for backend dispatchisaaclab.sim.schemasaddsdefine_deformable_body_properties()withpytetwildtetrahedralizationisaaclab.sim.spawners.meshesaddsTetMeshclass and tet mesh spawnersisaaclab.sim.spawners.materialsadds deformable body material configs and spawnerisaaclab_newton.physics.NewtonManagermodified with hooks, registries, and solver factoriesisaaclab_contrib.deformablecontains all Newton-specific deformable logic
The factory pattern via FactoryBase enables backend dispatch but requires careful initialization ordering.
Implementation Verdict
Significant concerns — Several correctness issues need addressing before ship.
Test Coverage
The PR includes 20 tests across two files:
test_deformable_object.py: 17 tests covering initialization, state read/write, kinematic targets, transformstest_rigid_deformable_coupling.py: 3 tests for one-way/two-way coupling and contact deflection
Missing coverage:
- No tests for the new
define_deformable_body_properties()schema function - No tests for
TetMeshclass construction (cuboid, from_file) - No tests for
spawn_deformable_body_material() - No tests for the
MeshFromFileCfgspawner - No regression test for the
add_usd_reference()defaultPrim fallback fix
CI Status
No CI checks available yet.
Findings
🔴 Critical: source/isaaclab_contrib/isaaclab_contrib/deformable/deformable_object.py:413-414 — Missing self.sim.forward() sync after state reset can cause stale state
In pick_cloth_env.py:398-430, after writing robot state to sim, self.sim.forward() is called, but the deformable state sync to state_1 (lines 403-430) happens after this. The cloth write_nodal_state_to_sim_index at line 428 writes to both state_0 and state_1, but the robot state sync to state_1 at lines 407-423 reads from state_0 which may not reflect the forward() result correctly. The ordering should be: write robot state → forward() → sync robot to state_1 → write cloth state.
# pick_cloth_env.py:398-428 - forward() is called before state_1 sync
self.sim.forward() # line 398
# Then state_1 is synced from state_0 (lines 407-423)
# But state_0 was just modified by forward()🔴 Critical: source/isaaclab_contrib/isaaclab_contrib/deformable/deformable_object_data.py:88-99 — Race condition in lazy property evaluation
The nodal_pos_w property writes to self._nodal_pos_w.data via a Warp kernel, but the timestamp check and update is not atomic. If update() is called from another thread while a kernel is in-flight, the timestamp could be updated before the kernel completes:
# Line 92-99
if self._nodal_pos_w.timestamp < self._sim_timestamp:
wp.launch(...) # Async kernel launch
self._nodal_pos_w.timestamp = self._sim_timestamp # Updated before kernel completesThis needs wp.synchronize() before the timestamp update or use of a completion flag.
🔴 Critical: source/isaaclab/isaaclab/sim/schemas/schemas.py:1123-1128 — pytetwild tetrahedralization can produce invalid meshes
The define_deformable_body_properties() calls pytetwild.tetrahedralize() with hardcoded parameters that may fail for certain mesh geometries. There's no error handling if tetrahedralization fails or produces degenerate tets:
tet_mesh_points, tet_mesh_indices = tetrahedralize(
vertices,
faces.reshape(-1, 3),
edge_length_fac=0.2, # May be too coarse for small meshes
# No try/except for tetrahedralization failures
)🟡 Warning: source/isaaclab_contrib/isaaclab_contrib/deformable/__init__.py:61-67 — Hook registration in _do_deferred_imports() modifies global state unsafely
The deferred import mechanism modifies NewtonManager class attributes (_particle_sync_fn, _per_world_builder_hooks, etc.) without synchronization. If multiple threads import simultaneously, hooks could be registered multiple times or incompletely:
# Lines 61-67 in _register_hooks_impl
NewtonManager._particle_sync_fn = sync_particles_to_usd
if per_world_deformable_hook not in NewtonManager._per_world_builder_hooks:
NewtonManager._per_world_builder_hooks.append(per_world_deformable_hook)🟡 Warning: source/isaaclab_contrib/isaaclab_contrib/deformable/coupled_solver.py:296-303 — _MAX_REACTION_CONTACTS=2048 may overflow
The reaction kernel is launched with a fixed dimension of 2048, but if soft_contact_count[0] exceeds this, contacts will be silently ignored. The kernel early-exits for tid >= contact_count[0], but there's no warning or dynamic resizing:
wp.launch(
_kernel_body_particle_reaction,
dim=_MAX_REACTION_CONTACTS, # Fixed at 2048
...
)🟡 Warning: source/isaaclab/isaaclab/sim/spawners/meshes/meshes.py:543-567 — spawn_mesh_from_file uses naive fan triangulation
The triangulation logic at lines 560-567 assumes convex quads for fan triangulation, which produces incorrect geometry for non-convex polygons:
# Manual fan triangulation - incorrect for non-convex faces
for i in range(1, count - 1):
tris.append([face_indices[idx], face_indices[idx + i], face_indices[idx + i + 1]])🔵 Improvement: source/isaaclab_contrib/isaaclab_contrib/deformable/deformable_object.py:285-295 — Redundant _deformable_type check in _register_deformable
The _is_surface_material check at line 285 and the subsequent tet_prims/mesh_prims logic is complex and has overlapping conditions. Consider consolidating the deformable type determination:
# Lines 285-295 have complex nested conditions that could be simplified
_is_surface_material = (
self.cfg.spawn is not None
and getattr(self.cfg.spawn, "physics_material", None) is not None
and "Surface" in type(self.cfg.spawn.physics_material).__name__
)🔵 Improvement: source/isaaclab_tasks_experimental/isaaclab_tasks_experimental/direct/pick_cloth/pick_cloth_env_cfg.py:14-17 — Hardcoded path to Newton assets may fail in distribution
The shirt USD path is constructed from the Newton package location, which may not be portable:
_newton_spec = importlib.util.find_spec("newton")
_SHIRT_USD = os.path.join(
os.path.dirname(_newton_spec.origin),
"examples", "assets", "unisex_shirt.usd",
)This will fail if Newton is installed as a zip or if the examples directory is not distributed.
Greptile SummaryThis PR adds Newton deformable object support (cloth and volumetric soft bodies) using a layered plugin architecture: core abstractions in
Confidence Score: 4/5Safe to merge for rigid-only scenes; two P1 bugs affect two-way friction correctness and multi-deformable particle sync that should be fixed before production use of the new deformable feature. Two P1 findings: incorrect particle_q_prev aliasing silently breaks Coulomb friction in two-way coupling mode, and num_points hardcoded from the first registry entry breaks Fabric sync for mixed-particle-count scenes. Both are confined to isaaclab_contrib and do not affect existing rigid-body functionality. The broader architecture is solid. source/isaaclab_contrib/isaaclab_contrib/deformable/coupled_solver.py (friction bug) and source/isaaclab_contrib/isaaclab_contrib/deformable/particle_sync.py (multi-deformable num_points bug) Important Files Changed
Sequence DiagramsequenceDiagram
participant Task as Task/Env
participant DO as DeformableObject
participant NM as NewtonManager
participant CS as CoupledSolver
participant FS as Fabric/USD
Task->>DO: __init__(cfg)
DO->>NM: _deformable_registry.append(entry)
Task->>NM: finalize_model()
NM->>NM: post_finalize_model_fn() [apply_model_cfg]
NM->>CS: create solver via _solver_factories
NM->>FS: setup Fabric particle attrs (newton:particleOffset)
loop Each simulation step
Task->>NM: step()
NM->>CS: rebuild_bvh(state_0)
NM->>CS: solver.step(state_in, state_out, ...)
Note over CS: one_way: rigid → collide → VBD<br/>two_way: collide → reactions → rigid → VBD
NM->>NM: _mark_state_dirty()
end
Task->>NM: pre_render()
NM->>NM: sync_transforms_to_usd() [rigid bodies]
NM->>FS: sync_particles_to_usd() [deformables via Warp kernel]
Task->>DO: write_nodal_kinematic_target_to_sim_index(targets)
DO->>DO: update nodal_kinematic_target buffer
Task->>NM: step() [write_data_to_sim enforces targets]
DO->>NM: enforce_kinematic_targets kernel → state_0, state_1
Reviews (1): Last reviewed commit: "Move deformable module from isaaclab_exp..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
|
||
| # 5. Clear spurious particle forces from rigid step | ||
| state_in.particle_f.zero_() | ||
|
|
||
| # 6. VBD step -- uses same contacts detected in step 2 | ||
| self.vbd.step(state_in, state_out, control, self.contacts, dt) | ||
|
|
||
| def _rigid_step(self, state_in: State, state_out: State, control: Control, dt: float) -> None: | ||
| """Advance rigid bodies with the configured sub-solver.""" | ||
| model = self._model | ||
|
|
||
| saved_particle_count = model.particle_count | ||
| model.particle_count = 0 | ||
|
|
||
| self.rigid_solver.step(state_in, state_out, control, None, dt) | ||
|
|
||
| model.particle_count = saved_particle_count | ||
|
|
||
| def _apply_reactions(self, state: State, dt: float) -> None: | ||
| """Launch the reaction kernel to inject normal + friction forces into body_f.""" | ||
| model = self._model | ||
| contacts = self.contacts | ||
|
|
||
| wp.launch( | ||
| _kernel_body_particle_reaction, | ||
| dim=_MAX_REACTION_CONTACTS, | ||
| inputs=[ |
There was a problem hiding this comment.
particle_q_prev aliased to particle_q, zeroing friction slip
_apply_reactions passes state.particle_q for both the particle_q and particle_q_prev kernel arguments. Inside _kernel_body_particle_reaction, friction is computed as:
dx = particle_q[p_idx] - particle_q_prev[p_idx] # always 0 when same array
relative_translation = dx - bv * dt # reduces to -bv * dt
Because both arguments point to the same array, dx is always zero, so the relative slip between particle and body surface is never measured — friction becomes purely a function of the body velocity, not the actual particle-body relative motion. The docstring says this kernel "mirrors the complete contact model from evaluate_body_particle_contact()", which uses a dedicated previous-position buffer. A separate particle_q_prev snapshot must be maintained (e.g. clone state.particle_q before the rigid step and pass it here) for the friction term to be physically correct.
Moves the two deformable task environments from isaaclab_tasks_experimental to isaaclab_tasks so they are discovered by zero_agent.py without needing an extra import of the experimental package.
… code Restore dt=0.01 in SimulationCfg, update env prim comment to explain Newton's env_\d+ naming requirement, restore original reset() ordering, and remove stale commented-out loop.
Remove TetMesh class, spawn_tet_mesh_cuboid, spawn_tet_mesh_from_file, spawn_mesh_from_file, _spawn_tet_mesh_geom_from_tet_mesh and their config counterparts. None are used by any task, test, or tutorial. Also restore Newton docstring in modify_deformable_body_properties.
The isaaclab_newton/assets/deformable_object/__init__.py stub re-exported DeformableObject from isaaclab_contrib, creating a reverse dependency. This is unnecessary since register_hooks() already registers the Newton backend with the DeformableObject factory directly.
Extract the 50-line inline block that sets up Fabric attributes for deformable particle sync from NewtonManager.start_simulation() into setup_fabric_particle_sync() in isaaclab_contrib/deformable/particle_sync. Registered via a new _post_start_simulation_fn hook, keeping isaaclab_newton free of deformable-specific logic.
Cherry-pick Mike's fixes from #4: - Move visual mesh topology overwrite to define_deformable_body_properties in schemas.py so it happens at USD authoring time, not simulation start. - Add per-instance newton:particleCount Fabric attribute so the sync kernel supports heterogeneous deformable bodies instead of assuming a single particles_per_body for all instances. - Simplify setup_fabric_particle_sync() to only create Fabric attributes.
…e logging - Replace CPU-bound Python for-loop in write_nodal_kinematic_target_to_sim_index with vectorized torch advanced indexing (eliminates per-env GPU-CPU sync). - Log sync_particles_to_usd failures at warning level on first occurrence, then throttle to debug for subsequent failures. - Cache usdrt module import to avoid re-importing every render frame. - Add clarifying comment on particle_q_prev aliasing in two-way coupling friction (known limitation, acceptable for one-way primary use case). - Document _MAX_REACTION_CONTACTS fixed upper bound in coupled_solver.
xiangdonglai
force-pushed
the
donglaix/deformable_experimental
branch
from
717693e to
28f4b25
Compare
1 participant
Summary
This PR adds Newton deformable object support (cloth and volumetric soft bodies with coupled rigid-deformable simulation) using a modular, layered architecture. It is a restructured version of #5287 that splits the monolithic implementation into clean layers with the bulk of the Newton-specific logic in
isaaclab_contrib.Fixes #5285
Motivation
The original PR (#5287) placed all deformable Newton logic inside
isaaclab_newton. This restructured version follows a plugin architecture: core abstractions live inisaaclab, minimal hooks inisaaclab_newton, and all Newton deformable implementation inisaaclab_contrib. This keepsisaaclab_newtonfree of deformable-specific code and makes the feature opt-in.Architecture
```
isaaclab (core abstractions)
↑
isaaclab_newton (minimal hooks, no deformable logic)
↑
isaaclab_contrib/deformable (all Newton deformable implementation)
```
No circular dependencies. Each layer only imports from layers below it.
What stays in `isaaclab` core
Backend-agnostic infrastructure needed by both PhysX and Newton deformable:
What stays in `isaaclab_newton` (minimal hooks only)
Small, additive changes to `NewtonManager` and `newton_replicate.py` that let `isaaclab_contrib` plug in without modifying the manager later. No deformable logic, no behavioral change for rigid-only scenes.
What lives in `isaaclab_contrib/deformable` (new)
All Newton-specific deformable implementation, registered as plugins via hooks:
Tasks (`isaaclab_tasks`)
Tests
Type of change
Checklist