isaac/CLAUDE.md

Context for Claude

Working notes for this repo: how the cell is put together, where the rest of the assets live, and the traps that are not visible from the code. Read this before changing anything in robozon_sorter/sim/ — several constants here are load-bearing and look arbitrary.

---

1. What this is

A conveyor sorting cell in Isaac Sim 6.0 (Kit 110). Goods ride a belt, three stereo pairs measure each one under a camera portal, and a pusher diverts the round ones onto a branch belt and into a bin.

infeed belt ──▶ camera portal ──▶ laser gate ──▶ pusher ──▶ Belt_01 ──▶ bin
   x=+2.3          x=-0.75          x=-3.74      x=-3.90              y=+1.3…2.0
                (CRE-ROI v2b)                  (class D only)

Goods travel in −X at 1 m/s. Belt surface is at z = 1.781 everywhere on the main run and the branch.

Classes: B = sortable, C = oversize (any dim >450×320×320 mm or <10 mm), D = round (K = r_in/R_out > 0.8). Only D is diverted.

---

2. Where things are

This repo

scene/sorter.usd is the real authored cell, not a procedural rebuild. It was exported from 90_degree.usd with two changes: conveyor references re-pointed to ../assets/conveyors/, and the 48-object library dropped in favour of six bundled test meshes with baked geometry.

Do not "simplify" the scene by regenerating it from code. That was tried and rejected — the result looked nothing like the real cell.

scene/plow_cell.usd is the same authored build carried over whole, but as the bare mechanics: conveyors, the Y-split pusher and the plow, with no camera portal, no laser gate and no item library. Cameras, speed scenarios and laser sensors get added on top of it. Rebuild it with scripts/build_plow_cell.py, which re-points the S3 conveyor references at assets/conveyors/, points the plow at assets/plow/, and strips the baked targetPosition.timeSamples track off the pusher drive — time samples outrank the attribute default, so while they are present nothing can control that drive.

The plow meshes matter: assets/meshes/plow_*.usd are 8-point placeholder boxes that sorter.usd still uses, while assets/plow/plow_*.usd is the real geometry (72k / 23k points). scripts/smoke_plow_cell.py asserts the point count precisely because a stub composes without error and just looks wrong.

Remote machine (where Isaac Sim runs)

host      dasha@46.39.224.77
isaac     /home/whatevenif/isaacsim          (python.sh lives here)
project   /home/dasha/robozon-sorter          (deployed copy, ~308 MB, runnable)
assets    /home/dasha/isaac_assets            (the original working tree)

Connect with an SSH ControlMaster (plain repeated ssh exhausts local ephemeral ports on long sessions) and tunnel the Kit python server:

ssh -M -S ~/.ssh/cm/dasha -o ControlPersist=900 -fN dasha@46.39.224.77
ssh -fN -L 8226:127.0.0.1:8226 dasha@46.39.224.77       # isaacsim.code_editor.python_server

Then send code with the isaac-sim-remote skill's isaacsim_send.py. Named contexts keep state between calls, which is how the long experiments were run.

More meshes, if six are not enough

What	Where	Notes
31 classified objects	/home/dasha/isaac_assets/categories.json	13 C, 11 D, 7 B; has zone, obb_extents_m, k_round, label_ru
the objects themselves	/home/dasha/isaac_assets/robozon_conveyor_scaled.usd under /World/CVObjects	48 prims, 31 of them classified
raw source geometry	/home/dasha/.objaverse/hf-objaverse-v1/glbs/	19 .glb, 123 MB — what CVObjects reference
already-exported	/home/dasha/isaac_assets/export_meshes/	flattened USDs, the six in this repo came from here
conveyor art	/home/dasha/isaac_assets/Props/Conveyors/	283 MB, mirrors the Omniverse S3 tree

To add a test mesh, flatten it out of the source scene so it carries no .glb reference:

ns = Usd.Stage.CreateInMemory()
root = UsdGeom.Xform.Define(ns, "/Item")
root.GetPrim().GetReferences().AddReference(
    "/home/dasha/isaac_assets/robozon_conveyor_scaled.usd", f"/World/CVObjects/{name}")
ns.SetDefaultPrim(root.GetPrim())
ns.Flatten().Export(f"assets/meshes/{name}.usd")

Then add an entry to assets/meshes/manifest.json with zone and gt_dims_mm. Objects in that scene sit at 0.49× real size (1/3-scale meshes × 1.4706), which is why config.DIM_SCALE = 1/2.041 converts metres to real millimetres.

Scene backups

/home/dasha/isaac_assets/backups/ — robozon_prescale_2252.usd is the clean pre-scale original, useful if the working scene ever gets damaged.

---

3. Mechanics

sim/scene.py opens the USD and re-applies the runtime settings USD does not carry. sim/mechanics.py is the per-item behaviour. sim/spawner.py hooks a PhysX step callback so the cell runs itself from the Play button.

Belts

Driven by PhysxSurfaceVelocityAPI on kinematic slabs, not by the authored ConveyorBeltGraph nodes — those carry no velocity and only fight the explicit setting, so they are switched off on load.

surfaceVelocity is expressed in the body's LOCAL frame. Belt_01 (/World/ConveyorTrack_03/Belt_01) is rotated: its local X points along world −Y. Carrying goods toward the bin therefore needs (−speed, 0, 0). Setting the intuitive (0, +speed, 0) drags them sideways and they sit there looking stuck. Resolve the local axes with XformCache.GetLocalToWorldTransform(prim).TransformDir(...) before setting it.

Pusher

A kinematic blade moved directly from script. It is not joint-driven: the authored PusherSlide prismatic drive is unusable because USD drive-target writes reach PhysX about a second late, so the blade never finishes its stroke while the item is still in reach. The joint is disabled and DiverterAnimGraph is switched off (it rewrites diverter targets every tick).

Blade collisions are filtered against every belt so it sweeps through the side rails instead of grinding on them.

Speed and physics rate are a matched pair: 2.5 m/s at 120 Hz. Do not change one alone.

• The blade covers 0.70 m of belt; the beam trips with the item at x = −3.61, so the whole cycle must fit in 0.64 m of travel = 0.64 s at 1 m/s.
• Stroke is 0.72 m → extension alone needs >1.12 m/s, extend+retract >2.25 m/s.
• Measured: 1.0 / 1.5 / 2.0 / 2.5 m/s all deliver to the bin. 3.0 m/s throws the item (ends up at y≈122, z≈−2070) — the kinematic blade injects too much impulse.
• Raising the solver rate to 480 Hz to "smooth" the impulse also breaks the landing. It was tried; the item stopped reaching the bin. 120 Hz is the validated setting.

Plow sorting station (plow_cell.usd)

Two-way sort at the plow, verified by measurement. Rebuild the geometry with scripts/narrow_plow.py then scripts/place_plow_lanes.py.

Arm width is 600 mm, not the authored 730. The wider arm overhung both rails and clipped goods it should have passed. narrow_plow.py does not guess which local axis carries the length - it tries each and measures; the answer is local Y, scale 0.8219. Hinge, drive and limits untouched.

Lane geometry, all verified against the belt (main belt x -7.00..-6.00, y +-0.45, top z 1.7805):

	span	contact
lane B, perpendicular, travel -Y	x -7.03..-6.58, y -2.45..-0.45	height 0.0 mm, Y 0.0 mm
lane C, 45 deg, travel (-X,+Y)	x -8.12..-6.39, y +0.45..+2.18	height 0.0 mm, Y 0.1 mm

Three placement mistakes were made and corrected, all of which looked fine in the tree and wrong in the viewport:

1. Height. These tracks carry an authored -0.1 z offset, putting their belts at z 1.681 - 100 mm below the run, so they read as separate furniture. Use LANE_DROP = 0.
2. X position. Lane B was first placed at x -7.48..-7.03, entirely past where the main belt stops (-7.00). A plow sweeps goods sideways while they are still on the belt, so a lane must run alongside it inside the arm's span (x -7.12..-6.52), never beyond the end. Same for C: at x -6.90 its near corner sat behind the plow and +Y deflections had nothing to land on; -6.55 puts the corner at the arm tip.
3. A 45 deg lane does not meet a straight edge at its centreline. Its near corner runs ahead by 159 mm (measured), so the lane needs that much offset or it cuts into the belt.

An angled lane needs a transition DECK, not a corner patch. Offsetting by 159 mm makes its near corner touch, but only that one corner - the other stands off by width/sqrt2 = 318 mm. Patching just that triangle is not enough either: the plow can put a item anywhere across the discharge width, so anything pushed wide still drops through the second gap. place_plow_lanes.add_transition() therefore decks the whole corner - the convex span of the belt edge across the junction (x -7.00..-6.00) and both end-face corners, giving y 0.45..0.768 - coplanar with both belt surfaces. A square lane like B meets flush along its whole face, so the builder returns nothing rather than emitting a zero-area collider.

Every lane also needs a corner deck, flush or not. The belt is wider than the lane - the run reaches x=-6.00 while lane B stops at -6.58 - so the right angle between them is open air, and anything the plow pushes sideways in that leftover span drops through. add_corner_deck() fills it with a triangular fillet (B: legs 575 mm, corners (-6.575,-0.45), (-6.00,-0.45), (-6.575,-1.025); C: legs 391 mm), turning the right angle into a chute. Deck plus fillet together leave no open surface across the discharge.

Lanes start at y = +-0.45, which clears the arm's swept envelope: a 600 mm arm at +-35 deg reaches 0.60*sin(35) = 0.344 m either side.

The conveyor shell collides. SM_ConveyorBelt_*_02 has collision=True, and that includes the blue side rails - they physically block goods from leaving the belt, which is exactly what "nothing reaches the bins" looks like. plow_sort.open_junction() clears the collider on the three shells at the junction, the way a real plow station has its rails cut away. Each Belt keeps its own collider, so nothing falls through.

Plow delivery is geometrically impossible as currently placed - measured. The arm is 600 mm and pivots at the belt centre (-7.05, 0), so its lateral reach is 0.60*sin(35) = 0.344 m. Both lanes start at y = +-0.45. That leaves a 106 mm dead band: the blade can push an item to 0.344 and no further, the lane begins at 0.45, and the item runs off the belt end and stops at x ~= -7.0 with v = 0, or falls. Every trace in runs/gt_run5.json shows exactly that - barrel drifts from y=-0.09 to y=+0.148 and halts.

The arm sits at z 1.810..1.890 while the lane surfaces are at 1.7805, i.e. 29 mm above them, so it would sweep over a lane edge rather than into it. The earlier reasoning that lanes had to start beyond the swept envelope was therefore unnecessary, and it is what opened the dead band. Closing it means one of: lanes in to ~y=0.33, a longer arm, or a larger swing (the joint limit is +-35 authored).

The angle does not track the command. Measured in a demo run: an item commanded 0.0 saw the arm at +30.5, one commanded +30.0 saw -34.6. Both then went to the same lane. So on top of the reach gap the control sign and the settling are wrong. Two things are known about this path and both matter:

• Plow.target(deg) writes the whole angle at once; calling it and step_toward() leaves the ramp nothing to do and the blade snaps. Use one or the other, and plow_sort uses the ramp.

• Plow.home() does not settle the arm - it sets a target, and the compliant drive needs time. Consecutive trials therefore start from wherever the previous one left the blade (measured: -26.4 and -14.4 at the start of runs meant to begin at 0).

• A positive command deflects to -Y (lane B), opposite to the natural reading. Measured, not assumed, exactly as the module docstring warns.

Until reach and tracking are fixed the plow does not deliver: scripts/run_demo.py reports the classification honestly and a delivery rate near zero.

Retract interlocks

The blade returns only when (a) the pushed item has cleared to y > 0.5 and (b) no other item is inside the blade's footprint. Retracting blindly sweeps the blade back through the next item and knocks it over — that was a real observed failure, not a hypothetical.

Plow (plow_cell.usd only)

The second diverter, at x = −7.05. Mechanically the opposite of the pusher: a dynamic arm on a revolute joint with an angular force drive (axis Z, limits ±35°, stiffness 120000, damping 1500, 12 kg with gravity disabled). Being force-driven it is compliant — it yields on contact instead of teleporting through cargo — so sim/plow.py commands a drive target rather than writing a transform.

Two consequences: a target is a request, not a position (read Plow.angle, which measures the arm's simulated pose, never assume it arrived), and the rate is not free. The authored graph swings 30° in 7 ms (72 rad/s), which is a display animation, not a sortable motion — at that rate the blade lands as an impulse, the same failure the pusher shows above 2.5 m/s. Plow.step_toward ramps the target at config.PLOW_RATE (180 °/s) instead; config.PLOW_RATE_AUTHORED keeps the original figure for reference.

The scene keeps its authored DiverterAnimGraph, so pressing Play alone demonstrates the cell. plow_cell.prepare(..., script_control=True) switches that graph off — it has to go, or it rewrites the drive target every tick and overwrites anything Python commands.

There is also a stray second ConveyorTrack_01 at stage root, outside /World, left over from how the cell was assembled; it composes as a duplicate belt in the same place and is deactivated on load rather than deleted, so the file stays as authored.

Laser gate

A genuine raycast_closest across the belt, not a coordinate test. The beam starts at y = −0.24, which is deliberately clear of the blade's retracted footprint (the blade spans y −0.33…−0.27). Start it any further out and the ray simply reads the blade and the gate never sees cargo.

---

4. Vision (CRE-ROI v2b)

cv/pipeline.py. Per item, once, while it sits under the portal: FastSAM segment-everything → keep the blob covering the projected inspection point in every view → ROI crop to a fixed 320 px side → one batched CRE-Stereo pass over all three crops → fuse, dropping views whose 3D centroid disagrees with the median by >10 cm → belt-plane OBB + r_in/R_out.

The crop must use the identical column window in both eyes, left-padded by the maximum disparity, or the right-hand counterpart falls outside the crop. Disparity is invariant to an equal column shift, so depth stays correct.

The stereo rig must be rectified. Both eyes of a pair share one orientation, with the right eye offset along its X. Aiming each eye separately at the target verges the pair and breaks depth = fx·b/disp — reconstruction came back at 2–5 m instead of 0.63 m. If config/calib.json ever loses its "rectified": true flag, the pipeline refuses to load rather than silently producing garbage.

Measured: dimensions land within ~34 mm of ground truth on the largest edge; the batched CRE pass costs ~170 ms per item.

---

5. Traps that cost real time

Each of these presented as a silent failure, not an error.

• Copied items arrive kinematic and hidden. Kinematic bodies ignore gravity and belt friction; hidden ones are invisible to the cameras while still simulating. Both must be cleared, plus sleepThreshold = 0 or a settled item is never woken by the belt.
• BBoxCache / XformCache return the AUTHORED transform during simulation. A moving item looks frozen. Use RigidPrim.get_world_poses().
• timeline.stop() resets items to their authored poses, so captures of a finished run must be taken while still playing.
• rep.orchestrator.step_async() stops the timeline. Any live loop that classifies must call timeline.play() again afterwards or the whole line freezes mid-run.
• The viewport's active camera may not be Persp. Setting Persp's pose then does nothing; call vp.set_active_camera("/OmniverseKit_Persp") first.
• Camera bodies carry cosmetic AimRay cones that sit right over the inspection point and dominate the frame. Hidden on load.
• Stale composition: if a stage was opened while a referenced asset was missing, USD caches the failure. Dropping the file in later does not fix it — the prim stays valid with localErrors: none but empty typeName and bbox. Diagnose with GetPrimStack(), fix by re-opening the stage.
• Stage.TraverseAll() crashes Isaac on this scene. Use targeted GetPrimAtPath, or run read-only checks headlessly via /home/whatevenif/isaacsim/python.sh (has pxr, no Kit).
• Cube colliders: use size = 2.0 so the scale op equals the half-extent. Any other arrangement makes PhysX use the wrong bounds and goods drop through the belt.

---

6. Known limitations

• Classification is weak. Metrology is sound but the roundness metric does not separate classes at this scale: genuinely round items read K ≈ 0.75 against a 0.80 threshold while a plain box reads 0.76. config.ROUND_K needs recalibration before the vision output should drive the pusher for real. Use --no-vision to exercise mechanics on ground truth.
• Consecutive class-D items are missed at the default 700 mm pitch. Headway is 0.70 s and one pusher cycle is 0.62 s; the gate is blind while a cycle runs, so a D item immediately behind another D crosses the beam unseen. Verified: D,C,D,B sorts both D correctly, D,D back-to-back loses the second. Fixing it needs a larger pitch, a slower belt or a second diverter — not a faster blade, which is already at its stability limit.
• The plow (DiverterEnd) is untouched in sorter.usd. There its geometry was restored and its arm returned to rest, but its control logic is left exactly as authored. It is scene/plow_cell.usd that makes the plow controllable — see below.