# Context for Claude Working notes for this repo: how the cell is put together, where the rest of the assets live, and the traps that are not visible from the code. Read this before changing anything in `robozon_sorter/sim/` — several constants here are load-bearing and look arbitrary. --- ## 1. What this is A conveyor sorting cell in Isaac Sim 6.0 (Kit 110). Goods ride a belt, three stereo pairs measure each one under a camera portal, and a pusher diverts the round ones onto a branch belt and into a bin. ``` infeed belt ──▶ camera portal ──▶ laser gate ──▶ pusher ──▶ Belt_01 ──▶ bin x=+2.3 x=-0.75 x=-3.74 x=-3.90 y=+1.3…2.0 (CRE-ROI v2b) (class D only) ``` Goods travel in **−X** at 1 m/s. Belt surface is at **z = 1.781** everywhere on the main run and the branch. **Classes:** B = sortable, C = oversize (any dim >450×320×320 mm or <10 mm), D = round (`K = r_in/R_out > 0.8`). Only D is diverted. --- ## 2. Where things are ### This repo `scene/sorter.usd` is the real authored cell, not a procedural rebuild. It was exported from `90_degree.usd` with two changes: conveyor references re-pointed to `../assets/conveyors/`, and the 48-object library dropped in favour of six bundled test meshes with baked geometry. Do not "simplify" the scene by regenerating it from code. That was tried and rejected — the result looked nothing like the real cell. `scene/plow_cell.usd` is the same authored build carried over whole, but as the **bare mechanics**: conveyors, the Y-split pusher and the plow, with no camera portal, no laser gate and no item library. Cameras, speed scenarios and laser sensors get added on top of it. Rebuild it with `scripts/build_plow_cell.py`, which re-points the S3 conveyor references at `assets/conveyors/`, points the plow at `assets/plow/`, and strips the baked `targetPosition.timeSamples` track off the pusher drive — time samples outrank the attribute default, so while they are present nothing can *control* that drive. The plow meshes matter: `assets/meshes/plow_*.usd` are 8-point placeholder boxes that `sorter.usd` still uses, while `assets/plow/plow_*.usd` is the real geometry (72k / 23k points). `scripts/smoke_plow_cell.py` asserts the point count precisely because a stub composes without error and just looks wrong. ### Remote machine (where Isaac Sim runs) ``` host dasha@46.39.224.77 isaac /home/whatevenif/isaacsim (python.sh lives here) project /home/dasha/robozon-sorter (deployed copy, ~308 MB, runnable) assets /home/dasha/isaac_assets (the original working tree) ``` Connect with an SSH ControlMaster (plain repeated ssh exhausts local ephemeral ports on long sessions) and tunnel the Kit python server: ```bash ssh -M -S ~/.ssh/cm/dasha -o ControlPersist=900 -fN dasha@46.39.224.77 ssh -fN -L 8226:127.0.0.1:8226 dasha@46.39.224.77 # isaacsim.code_editor.python_server ``` Then send code with the `isaac-sim-remote` skill's `isaacsim_send.py`. Named contexts keep state between calls, which is how the long experiments were run. ### More meshes, if six are not enough | What | Where | Notes | |---|---|---| | 31 classified objects | `/home/dasha/isaac_assets/categories.json` | 13 C, 11 D, 7 B; has `zone`, `obb_extents_m`, `k_round`, `label_ru` | | the objects themselves | `/home/dasha/isaac_assets/robozon_conveyor_scaled.usd` under `/World/CVObjects` | 48 prims, 31 of them classified | | raw source geometry | `/home/dasha/.objaverse/hf-objaverse-v1/glbs/` | 19 `.glb`, 123 MB — what CVObjects reference | | already-exported | `/home/dasha/isaac_assets/export_meshes/` | flattened USDs, the six in this repo came from here | | conveyor art | `/home/dasha/isaac_assets/Props/Conveyors/` | 283 MB, mirrors the Omniverse S3 tree | To add a test mesh, flatten it out of the source scene so it carries no `.glb` reference: ```python ns = Usd.Stage.CreateInMemory() root = UsdGeom.Xform.Define(ns, "/Item") root.GetPrim().GetReferences().AddReference( "/home/dasha/isaac_assets/robozon_conveyor_scaled.usd", f"/World/CVObjects/{name}") ns.SetDefaultPrim(root.GetPrim()) ns.Flatten().Export(f"assets/meshes/{name}.usd") ``` Then add an entry to `assets/meshes/manifest.json` with `zone` and `gt_dims_mm`. Objects in that scene sit at **0.49× real size** (1/3-scale meshes × 1.4706), which is why `config.DIM_SCALE = 1/2.041` converts metres to real millimetres. ### Scene backups `/home/dasha/isaac_assets/backups/` — `robozon_prescale_2252.usd` is the clean pre-scale original, useful if the working scene ever gets damaged. --- ## 3. Mechanics `sim/scene.py` opens the USD and re-applies the runtime settings USD does not carry. `sim/mechanics.py` is the per-item behaviour. `sim/spawner.py` hooks a PhysX step callback so the cell runs itself from the Play button. ### Belts Driven by `PhysxSurfaceVelocityAPI` on kinematic slabs, not by the authored `ConveyorBeltGraph` nodes — those carry no velocity and only fight the explicit setting, so they are switched off on load. **`surfaceVelocity` is expressed in the body's LOCAL frame.** `Belt_01` (`/World/ConveyorTrack_03/Belt_01`) is rotated: its local X points along world −Y. Carrying goods toward the bin therefore needs `(−speed, 0, 0)`. Setting the intuitive `(0, +speed, 0)` drags them sideways and they sit there looking stuck. Resolve the local axes with `XformCache.GetLocalToWorldTransform(prim).TransformDir(...)` before setting it. ### Pusher A kinematic blade moved directly from script. It is **not** joint-driven: the authored `PusherSlide` prismatic drive is unusable because USD drive-target writes reach PhysX about a second late, so the blade never finishes its stroke while the item is still in reach. The joint is disabled and `DiverterAnimGraph` is switched off (it rewrites diverter targets every tick). Blade collisions are filtered against every belt so it sweeps through the side rails instead of grinding on them. **Speed and physics rate are a matched pair: 2.5 m/s at 120 Hz.** Do not change one alone. * The blade covers 0.70 m of belt; the beam trips with the item at x = −3.61, so the whole cycle must fit in 0.64 m of travel = 0.64 s at 1 m/s. * Stroke is 0.72 m → extension alone needs >1.12 m/s, extend+retract >2.25 m/s. * Measured: 1.0 / 1.5 / 2.0 / 2.5 m/s all deliver to the bin. **3.0 m/s throws the item** (ends up at y≈122, z≈−2070) — the kinematic blade injects too much impulse. * Raising the solver rate to 480 Hz to "smooth" the impulse also breaks the landing. It was tried; the item stopped reaching the bin. 120 Hz is the validated setting. ### Plow sorting station (`plow_cell.usd`) Two-way sort at the plow, verified by measurement. Rebuild the geometry with `scripts/narrow_plow.py` then `scripts/place_plow_lanes.py`. **Arm width is 600 mm**, not the authored 730. The wider arm overhung both rails and clipped goods it should have passed. `narrow_plow.py` does not guess which local axis carries the length - it tries each and measures; the answer is **local Y, scale 0.8219**. Hinge, drive and limits untouched. **Lane geometry, all verified against the belt (main belt x -7.00..-6.00, y +-0.45, top z 1.7805):** | | span | contact | |---|---|---| | lane B, perpendicular, travel -Y | x -7.03..-6.58, y -2.45..-0.45 | height 0.0 mm, Y 0.0 mm | | lane C, 45 deg, travel (-X,+Y) | x -8.12..-6.39, y +0.45..+2.18 | height 0.0 mm, Y 0.1 mm | Three placement mistakes were made and corrected, all of which looked fine in the tree and wrong in the viewport: 1. **Height.** These tracks carry an authored `-0.1` z offset, putting their belts at z 1.681 - 100 mm below the run, so they read as separate furniture. Use `LANE_DROP = 0`. 2. **X position.** Lane B was first placed at x -7.48..-7.03, entirely *past* where the main belt stops (-7.00). A plow sweeps goods sideways while they are still on the belt, so a lane must run **alongside** it inside the arm's span (x -7.12..-6.52), never beyond the end. Same for C: at x -6.90 its near corner sat behind the plow and +Y deflections had nothing to land on; -6.55 puts the corner at the arm tip. 3. **A 45 deg lane does not meet a straight edge at its centreline.** Its near corner runs ahead by 159 mm (measured), so the lane needs that much offset or it cuts into the belt. **An angled lane needs a transition DECK, not a corner patch.** Offsetting by 159 mm makes its near corner touch, but only that one corner - the other stands off by width/sqrt2 = 318 mm. Patching just that triangle is not enough either: the plow can put a item anywhere across the discharge width, so anything pushed wide still drops through the second gap. `place_plow_lanes.add_transition()` therefore decks the whole corner - the convex span of the belt edge across the junction (x -7.00..-6.00) and both end-face corners, giving y 0.45..0.768 - coplanar with both belt surfaces. A square lane like B meets flush along its whole face, so the builder returns nothing rather than emitting a zero-area collider. **Every lane also needs a corner deck**, flush or not. The belt is wider than the lane - the run reaches x=-6.00 while lane B stops at -6.58 - so the right angle between them is open air, and anything the plow pushes sideways in that leftover span drops through. `add_corner_deck()` fills it with a triangular fillet (B: legs 575 mm, corners (-6.575,-0.45), (-6.00,-0.45), (-6.575,-1.025); C: legs 391 mm), turning the right angle into a chute. Deck plus fillet together leave no open surface across the discharge. Lanes start at y = +-0.45, which clears the arm's swept envelope: a 600 mm arm at +-35 deg reaches 0.60*sin(35) = **0.344 m** either side. **The conveyor shell collides.** `SM_ConveyorBelt_*_02` has `collision=True`, and that includes the blue side rails - they physically block goods from leaving the belt, which is exactly what "nothing reaches the bins" looks like. `plow_sort.open_junction()` clears the collider on the three shells at the junction, the way a real plow station has its rails cut away. Each `Belt` keeps its own collider, so nothing falls through. **Plow delivery is geometrically impossible as currently placed - measured.** The arm is 600 mm and pivots at the belt centre (-7.05, 0), so its lateral reach is 0.60*sin(35) = **0.344 m**. Both lanes start at **y = +-0.45**. That leaves a **106 mm dead band**: the blade can push an item to 0.344 and no further, the lane begins at 0.45, and the item runs off the belt end and stops at x ~= -7.0 with v = 0, or falls. Every trace in `runs/gt_run5.json` shows exactly that - barrel drifts from y=-0.09 to y=+0.148 and halts. The arm sits at z 1.810..1.890 while the lane surfaces are at 1.7805, i.e. **29 mm above them**, so it would sweep *over* a lane edge rather than into it. The earlier reasoning that lanes had to start beyond the swept envelope was therefore unnecessary, and it is what opened the dead band. Closing it means one of: lanes in to ~y=0.33, a longer arm, or a larger swing (the joint limit is +-35 authored). **The angle does not track the command.** Measured in a demo run: an item commanded 0.0 saw the arm at **+30.5**, one commanded +30.0 saw **-34.6**. Both then went to the same lane. So on top of the reach gap the control sign and the settling are wrong. Two things are known about this path and both matter: * `Plow.target(deg)` writes the whole angle at once; calling it *and* `step_toward()` leaves the ramp nothing to do and the blade snaps. Use one or the other, and `plow_sort` uses the ramp. * `Plow.home()` does not settle the arm - it sets a target, and the compliant drive needs time. Consecutive trials therefore start from wherever the previous one left the blade (measured: -26.4 and -14.4 at the start of runs meant to begin at 0). * A **positive command deflects to -Y (lane B)**, opposite to the natural reading. Measured, not assumed, exactly as the module docstring warns. Until reach and tracking are fixed the plow does not deliver: `scripts/run_demo.py` reports the classification honestly and a delivery rate near zero. ### Retract interlocks The blade returns only when (a) the pushed item has cleared to y > 0.5 and (b) no other item is inside the blade's footprint. Retracting blindly sweeps the blade back through the next item and knocks it over — that was a real observed failure, not a hypothetical. ### Plow (`plow_cell.usd` only) The second diverter, at x = −7.05. Mechanically the opposite of the pusher: a **dynamic arm on a revolute joint with an angular force drive** (axis Z, limits ±35°, stiffness 120000, damping 1500, 12 kg with gravity disabled). Being force-driven it is compliant — it yields on contact instead of teleporting through cargo — so `sim/plow.py` commands a drive *target* rather than writing a transform. Two consequences: a target is a request, not a position (read `Plow.angle`, which measures the arm's simulated pose, never assume it arrived), and the rate is not free. The authored graph swings 30° in **7 ms** (72 rad/s), which is a display animation, not a sortable motion — at that rate the blade lands as an impulse, the same failure the pusher shows above 2.5 m/s. `Plow.step_toward` ramps the target at `config.PLOW_RATE` (180 °/s) instead; `config.PLOW_RATE_AUTHORED` keeps the original figure for reference. The scene keeps its authored `DiverterAnimGraph`, so pressing Play alone demonstrates the cell. `plow_cell.prepare(..., script_control=True)` switches that graph off — it has to go, or it rewrites the drive target every tick and overwrites anything Python commands. There is also a stray second `ConveyorTrack_01` at stage root, outside `/World`, left over from how the cell was assembled; it composes as a duplicate belt in the same place and is deactivated on load rather than deleted, so the file stays as authored. ### Laser gate A genuine `raycast_closest` across the belt, not a coordinate test. The beam starts at y = −0.24, which is deliberately **clear of the blade's retracted footprint** (the blade spans y −0.33…−0.27). Start it any further out and the ray simply reads the blade and the gate never sees cargo. --- ## 4. Vision (CRE-ROI v2b) `cv/pipeline.py`. Per item, once, while it sits under the portal: FastSAM segment-everything → keep the blob covering the projected inspection point in every view → ROI crop to a fixed 320 px side → **one batched CRE-Stereo pass over all three crops** → fuse, dropping views whose 3D centroid disagrees with the median by >10 cm → belt-plane OBB + `r_in/R_out`. The crop must use the **identical column window in both eyes**, left-padded by the maximum disparity, or the right-hand counterpart falls outside the crop. Disparity is invariant to an equal column shift, so depth stays correct. **The stereo rig must be rectified.** Both eyes of a pair share one orientation, with the right eye offset along its X. Aiming each eye separately at the target verges the pair and breaks `depth = fx·b/disp` — reconstruction came back at 2–5 m instead of 0.63 m. If `config/calib.json` ever loses its `"rectified": true` flag, the pipeline refuses to load rather than silently producing garbage. Measured: dimensions land within ~34 mm of ground truth on the largest edge; the batched CRE pass costs ~170 ms per item. --- ## 5. Traps that cost real time Each of these presented as a silent failure, not an error. - **Copied items arrive kinematic and hidden.** Kinematic bodies ignore gravity and belt friction; hidden ones are invisible to the cameras while still simulating. Both must be cleared, plus `sleepThreshold = 0` or a settled item is never woken by the belt. - **`BBoxCache` / `XformCache` return the AUTHORED transform during simulation.** A moving item looks frozen. Use `RigidPrim.get_world_poses()`. - **`timeline.stop()` resets items to their authored poses**, so captures of a finished run must be taken while still playing. - **`rep.orchestrator.step_async()` stops the timeline.** Any live loop that classifies must call `timeline.play()` again afterwards or the whole line freezes mid-run. - **The viewport's active camera may not be Persp.** Setting Persp's pose then does nothing; call `vp.set_active_camera("/OmniverseKit_Persp")` first. - **Camera bodies carry cosmetic `AimRay` cones** that sit right over the inspection point and dominate the frame. Hidden on load. - **Stale composition:** if a stage was opened while a referenced asset was missing, USD caches the failure. Dropping the file in later does not fix it — the prim stays valid with `localErrors: none` but empty typeName and bbox. Diagnose with `GetPrimStack()`, fix by re-opening the stage. - **`Stage.TraverseAll()` crashes Isaac on this scene.** Use targeted `GetPrimAtPath`, or run read-only checks headlessly via `/home/whatevenif/isaacsim/python.sh` (has pxr, no Kit). - **Cube colliders:** use `size = 2.0` so the scale op equals the half-extent. Any other arrangement makes PhysX use the wrong bounds and goods drop through the belt. --- ## 6. Known limitations - **Classification is weak.** Metrology is sound but the roundness metric does not separate classes at this scale: genuinely round items read K ≈ 0.75 against a 0.80 threshold while a plain box reads 0.76. `config.ROUND_K` needs recalibration before the vision output should drive the pusher for real. Use `--no-vision` to exercise mechanics on ground truth. - **Consecutive class-D items are missed at the default 700 mm pitch.** Headway is 0.70 s and one pusher cycle is 0.62 s; the gate is blind while a cycle runs, so a D item immediately behind another D crosses the beam unseen. Verified: `D,C,D,B` sorts both D correctly, `D,D` back-to-back loses the second. Fixing it needs a larger pitch, a slower belt or a second diverter — not a faster blade, which is already at its stability limit. - **The plow (`DiverterEnd`) is untouched in `sorter.usd`.** There its geometry was restored and its arm returned to rest, but its control logic is left exactly as authored. It is `scene/plow_cell.usd` that makes the plow controllable — see below.