Сортировочная ячейка Isaac Sim: CV-пайплайн и меши товаров
Замкнутый контур "поток -> CV -> механика": товары идут по конвейеру с шагом 700 мм, класс определяется стереопайплайном во время движения, пушер и плуг реагируют физически. Состав: * control_test/ - ячейка и CV. run_sorting_cv.py + cv_worker.py (два процесса, потому что torch внутри Isaac роняет сцену), cell.py (физика лент, плуга, пушера), measure_plane.py (замер габаритов), README.md и .memory.md с замерами, проблемами и ловушками * robozon_sorter/ - модули симуляции, scripts/ - утилиты, scene/ - сцены * assets/ - меши товаров, плуг, объекты Objaverse Бейзлайн CV: DEFOM-Stereo vitl, вход 480, iters 24, кроп зоны осмотра, без сегментации. На потоке 700 мм - классы 8/9, габариты MAE 32.8 мм, 469 мс на товар при такте 700 мс. Веса моделей (4.5 ГБ) и пропсы конвейера NVIDIA (274 МБ) не включены - источники и команды скачивания в MODELS.md. Выход прогонов (captures/, runtime/) не включён: воспроизводится. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@@ -0,0 +1,25 @@
|
|||||||
|
# Веса моделей в репозиторий не кладутся - ссылки на источники в MODELS.md
|
||||||
|
*.pth
|
||||||
|
*.pt
|
||||||
|
*.onnx
|
||||||
|
*.ckpt
|
||||||
|
*.safetensors
|
||||||
|
|
||||||
|
# Пропсы конвейера NVIDIA (274 МБ): не наши и скачиваются штатным скриптом
|
||||||
|
assets/conveyors/
|
||||||
|
# восстановить: python3 scripts/fetch_assets.py
|
||||||
|
|
||||||
|
# Выход прогонов - воспроизводится, в истории не нужен
|
||||||
|
control_test/captures/
|
||||||
|
control_test/runtime/
|
||||||
|
control_test/diag/
|
||||||
|
runs/
|
||||||
|
|
||||||
|
# Резервные копии, которые делались перед правками
|
||||||
|
*.bak
|
||||||
|
*.orig
|
||||||
|
*.pre*
|
||||||
|
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
.DS_Store
|
||||||
@@ -0,0 +1,328 @@
|
|||||||
|
# Context for Claude
|
||||||
|
|
||||||
|
Working notes for this repo: how the cell is put together, where the rest of the assets
|
||||||
|
live, and the traps that are not visible from the code. Read this before changing anything
|
||||||
|
in `robozon_sorter/sim/` — several constants here are load-bearing and look arbitrary.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. What this is
|
||||||
|
|
||||||
|
A conveyor sorting cell in Isaac Sim 6.0 (Kit 110). Goods ride a belt, three stereo pairs
|
||||||
|
measure each one under a camera portal, and a pusher diverts the round ones onto a branch
|
||||||
|
belt and into a bin.
|
||||||
|
|
||||||
|
```
|
||||||
|
infeed belt ──▶ camera portal ──▶ laser gate ──▶ pusher ──▶ Belt_01 ──▶ bin
|
||||||
|
x=+2.3 x=-0.75 x=-3.74 x=-3.90 y=+1.3…2.0
|
||||||
|
(CRE-ROI v2b) (class D only)
|
||||||
|
```
|
||||||
|
|
||||||
|
Goods travel in **−X** at 1 m/s. Belt surface is at **z = 1.781** everywhere on the main
|
||||||
|
run and the branch.
|
||||||
|
|
||||||
|
**Classes:** B = sortable, C = oversize (any dim >450×320×320 mm or <10 mm), D = round
|
||||||
|
(`K = r_in/R_out > 0.8`). Only D is diverted.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Where things are
|
||||||
|
|
||||||
|
### This repo
|
||||||
|
`scene/sorter.usd` is the real authored cell, not a procedural rebuild. It was exported
|
||||||
|
from `90_degree.usd` with two changes: conveyor references re-pointed to
|
||||||
|
`../assets/conveyors/`, and the 48-object library dropped in favour of six bundled test
|
||||||
|
meshes with baked geometry.
|
||||||
|
|
||||||
|
Do not "simplify" the scene by regenerating it from code. That was tried and rejected —
|
||||||
|
the result looked nothing like the real cell.
|
||||||
|
|
||||||
|
`scene/plow_cell.usd` is the same authored build carried over whole, but as the **bare
|
||||||
|
mechanics**: conveyors, the Y-split pusher and the plow, with no camera portal, no laser
|
||||||
|
gate and no item library. Cameras, speed scenarios and laser sensors get added on top of
|
||||||
|
it. Rebuild it with `scripts/build_plow_cell.py`, which re-points the S3 conveyor
|
||||||
|
references at `assets/conveyors/`, points the plow at `assets/plow/`, and strips the baked
|
||||||
|
`targetPosition.timeSamples` track off the pusher drive — time samples outrank the
|
||||||
|
attribute default, so while they are present nothing can *control* that drive.
|
||||||
|
|
||||||
|
The plow meshes matter: `assets/meshes/plow_*.usd` are 8-point placeholder boxes that
|
||||||
|
`sorter.usd` still uses, while `assets/plow/plow_*.usd` is the real geometry (72k / 23k
|
||||||
|
points). `scripts/smoke_plow_cell.py` asserts the point count precisely because a stub
|
||||||
|
composes without error and just looks wrong.
|
||||||
|
|
||||||
|
### Remote machine (where Isaac Sim runs)
|
||||||
|
|
||||||
|
```
|
||||||
|
host dasha@46.39.224.77
|
||||||
|
isaac /home/whatevenif/isaacsim (python.sh lives here)
|
||||||
|
project /home/dasha/robozon-sorter (deployed copy, ~308 MB, runnable)
|
||||||
|
assets /home/dasha/isaac_assets (the original working tree)
|
||||||
|
```
|
||||||
|
|
||||||
|
Connect with an SSH ControlMaster (plain repeated ssh exhausts local ephemeral ports on
|
||||||
|
long sessions) and tunnel the Kit python server:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -M -S ~/.ssh/cm/dasha -o ControlPersist=900 -fN dasha@46.39.224.77
|
||||||
|
ssh -fN -L 8226:127.0.0.1:8226 dasha@46.39.224.77 # isaacsim.code_editor.python_server
|
||||||
|
```
|
||||||
|
|
||||||
|
Then send code with the `isaac-sim-remote` skill's `isaacsim_send.py`. Named contexts keep
|
||||||
|
state between calls, which is how the long experiments were run.
|
||||||
|
|
||||||
|
### More meshes, if six are not enough
|
||||||
|
|
||||||
|
| What | Where | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| 31 classified objects | `/home/dasha/isaac_assets/categories.json` | 13 C, 11 D, 7 B; has `zone`, `obb_extents_m`, `k_round`, `label_ru` |
|
||||||
|
| the objects themselves | `/home/dasha/isaac_assets/robozon_conveyor_scaled.usd` under `/World/CVObjects` | 48 prims, 31 of them classified |
|
||||||
|
| raw source geometry | `/home/dasha/.objaverse/hf-objaverse-v1/glbs/` | 19 `.glb`, 123 MB — what CVObjects reference |
|
||||||
|
| already-exported | `/home/dasha/isaac_assets/export_meshes/` | flattened USDs, the six in this repo came from here |
|
||||||
|
| conveyor art | `/home/dasha/isaac_assets/Props/Conveyors/` | 283 MB, mirrors the Omniverse S3 tree |
|
||||||
|
|
||||||
|
To add a test mesh, flatten it out of the source scene so it carries no `.glb` reference:
|
||||||
|
|
||||||
|
```python
|
||||||
|
ns = Usd.Stage.CreateInMemory()
|
||||||
|
root = UsdGeom.Xform.Define(ns, "/Item")
|
||||||
|
root.GetPrim().GetReferences().AddReference(
|
||||||
|
"/home/dasha/isaac_assets/robozon_conveyor_scaled.usd", f"/World/CVObjects/{name}")
|
||||||
|
ns.SetDefaultPrim(root.GetPrim())
|
||||||
|
ns.Flatten().Export(f"assets/meshes/{name}.usd")
|
||||||
|
```
|
||||||
|
|
||||||
|
Then add an entry to `assets/meshes/manifest.json` with `zone` and `gt_dims_mm`. Objects in
|
||||||
|
that scene sit at **0.49× real size** (1/3-scale meshes × 1.4706), which is why
|
||||||
|
`config.DIM_SCALE = 1/2.041` converts metres to real millimetres.
|
||||||
|
|
||||||
|
### Scene backups
|
||||||
|
`/home/dasha/isaac_assets/backups/` — `robozon_prescale_2252.usd` is the clean pre-scale
|
||||||
|
original, useful if the working scene ever gets damaged.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Mechanics
|
||||||
|
|
||||||
|
`sim/scene.py` opens the USD and re-applies the runtime settings USD does not carry.
|
||||||
|
`sim/mechanics.py` is the per-item behaviour. `sim/spawner.py` hooks a PhysX step callback
|
||||||
|
so the cell runs itself from the Play button.
|
||||||
|
|
||||||
|
### Belts
|
||||||
|
Driven by `PhysxSurfaceVelocityAPI` on kinematic slabs, not by the authored
|
||||||
|
`ConveyorBeltGraph` nodes — those carry no velocity and only fight the explicit setting, so
|
||||||
|
they are switched off on load.
|
||||||
|
|
||||||
|
**`surfaceVelocity` is expressed in the body's LOCAL frame.** `Belt_01`
|
||||||
|
(`/World/ConveyorTrack_03/Belt_01`) is rotated: its local X points along world −Y. Carrying
|
||||||
|
goods toward the bin therefore needs `(−speed, 0, 0)`. Setting the intuitive `(0, +speed, 0)`
|
||||||
|
drags them sideways and they sit there looking stuck. Resolve the local axes with
|
||||||
|
`XformCache.GetLocalToWorldTransform(prim).TransformDir(...)` before setting it.
|
||||||
|
|
||||||
|
### Pusher
|
||||||
|
A kinematic blade moved directly from script. It is **not** joint-driven: the authored
|
||||||
|
`PusherSlide` prismatic drive is unusable because USD drive-target writes reach PhysX about
|
||||||
|
a second late, so the blade never finishes its stroke while the item is still in reach.
|
||||||
|
The joint is disabled and `DiverterAnimGraph` is switched off (it rewrites diverter targets
|
||||||
|
every tick).
|
||||||
|
|
||||||
|
Blade collisions are filtered against every belt so it sweeps through the side rails
|
||||||
|
instead of grinding on them.
|
||||||
|
|
||||||
|
**Speed and physics rate are a matched pair: 2.5 m/s at 120 Hz.** Do not change one alone.
|
||||||
|
|
||||||
|
* The blade covers 0.70 m of belt; the beam trips with the item at x = −3.61, so the whole
|
||||||
|
cycle must fit in 0.64 m of travel = 0.64 s at 1 m/s.
|
||||||
|
* Stroke is 0.72 m → extension alone needs >1.12 m/s, extend+retract >2.25 m/s.
|
||||||
|
* Measured: 1.0 / 1.5 / 2.0 / 2.5 m/s all deliver to the bin. **3.0 m/s throws the item**
|
||||||
|
(ends up at y≈122, z≈−2070) — the kinematic blade injects too much impulse.
|
||||||
|
* Raising the solver rate to 480 Hz to "smooth" the impulse also breaks the landing. It was
|
||||||
|
tried; the item stopped reaching the bin. 120 Hz is the validated setting.
|
||||||
|
|
||||||
|
|
||||||
|
### Plow sorting station (`plow_cell.usd`)
|
||||||
|
|
||||||
|
Two-way sort at the plow, verified by measurement. Rebuild the geometry with
|
||||||
|
`scripts/narrow_plow.py` then `scripts/place_plow_lanes.py`.
|
||||||
|
|
||||||
|
**Arm width is 600 mm**, not the authored 730. The wider arm overhung both rails and
|
||||||
|
clipped goods it should have passed. `narrow_plow.py` does not guess which local axis
|
||||||
|
carries the length - it tries each and measures; the answer is **local Y, scale 0.8219**.
|
||||||
|
Hinge, drive and limits untouched.
|
||||||
|
|
||||||
|
**Lane geometry, all verified against the belt (main belt x -7.00..-6.00, y +-0.45, top
|
||||||
|
z 1.7805):**
|
||||||
|
|
||||||
|
| | span | contact |
|
||||||
|
|---|---|---|
|
||||||
|
| lane B, perpendicular, travel -Y | x -7.03..-6.58, y -2.45..-0.45 | height 0.0 mm, Y 0.0 mm |
|
||||||
|
| lane C, 45 deg, travel (-X,+Y) | x -8.12..-6.39, y +0.45..+2.18 | height 0.0 mm, Y 0.1 mm |
|
||||||
|
|
||||||
|
Three placement mistakes were made and corrected, all of which looked fine in the tree and
|
||||||
|
wrong in the viewport:
|
||||||
|
|
||||||
|
1. **Height.** These tracks carry an authored `-0.1` z offset, putting their belts at
|
||||||
|
z 1.681 - 100 mm below the run, so they read as separate furniture. Use `LANE_DROP = 0`.
|
||||||
|
2. **X position.** Lane B was first placed at x -7.48..-7.03, entirely *past* where the
|
||||||
|
main belt stops (-7.00). A plow sweeps goods sideways while they are still on the belt,
|
||||||
|
so a lane must run **alongside** it inside the arm's span (x -7.12..-6.52), never beyond
|
||||||
|
the end. Same for C: at x -6.90 its near corner sat behind the plow and +Y deflections
|
||||||
|
had nothing to land on; -6.55 puts the corner at the arm tip.
|
||||||
|
3. **A 45 deg lane does not meet a straight edge at its centreline.** Its near corner runs
|
||||||
|
ahead by 159 mm (measured), so the lane needs that much offset or it cuts into the belt.
|
||||||
|
|
||||||
|
**An angled lane needs a transition DECK, not a corner patch.** Offsetting by 159 mm makes
|
||||||
|
its near corner touch, but only that one corner - the other stands off by
|
||||||
|
width/sqrt2 = 318 mm. Patching just that triangle is not enough either: the plow can put a
|
||||||
|
item anywhere across the discharge width, so anything pushed wide still drops through the
|
||||||
|
second gap. `place_plow_lanes.add_transition()` therefore decks the whole corner - the
|
||||||
|
convex span of the belt edge across the junction (x -7.00..-6.00) and both end-face
|
||||||
|
corners, giving y 0.45..0.768 - coplanar with both belt surfaces. A square lane like B
|
||||||
|
meets flush along its whole face, so the builder returns nothing rather than emitting a
|
||||||
|
zero-area collider.
|
||||||
|
|
||||||
|
**Every lane also needs a corner deck**, flush or not. The belt is wider than the lane -
|
||||||
|
the run reaches x=-6.00 while lane B stops at -6.58 - so the right angle between them is
|
||||||
|
open air, and anything the plow pushes sideways in that leftover span drops through.
|
||||||
|
`add_corner_deck()` fills it with a triangular fillet (B: legs 575 mm, corners
|
||||||
|
(-6.575,-0.45), (-6.00,-0.45), (-6.575,-1.025); C: legs 391 mm), turning the right angle
|
||||||
|
into a chute. Deck plus fillet together leave no open surface across the discharge.
|
||||||
|
|
||||||
|
Lanes start at y = +-0.45, which clears the arm's swept envelope: a 600 mm arm at +-35 deg
|
||||||
|
reaches 0.60*sin(35) = **0.344 m** either side.
|
||||||
|
|
||||||
|
**The conveyor shell collides.** `SM_ConveyorBelt_*_02` has `collision=True`, and that
|
||||||
|
includes the blue side rails - they physically block goods from leaving the belt, which is
|
||||||
|
exactly what "nothing reaches the bins" looks like. `plow_sort.open_junction()` clears the
|
||||||
|
collider on the three shells at the junction, the way a real plow station has its rails cut
|
||||||
|
away. Each `Belt` keeps its own collider, so nothing falls through.
|
||||||
|
|
||||||
|
**Plow delivery is geometrically impossible as currently placed - measured.** The arm is
|
||||||
|
600 mm and pivots at the belt centre (-7.05, 0), so its lateral reach is
|
||||||
|
0.60*sin(35) = **0.344 m**. Both lanes start at **y = +-0.45**. That leaves a **106 mm dead
|
||||||
|
band**: the blade can push an item to 0.344 and no further, the lane begins at 0.45, and the
|
||||||
|
item runs off the belt end and stops at x ~= -7.0 with v = 0, or falls. Every trace in
|
||||||
|
`runs/gt_run5.json` shows exactly that - barrel drifts from y=-0.09 to y=+0.148 and halts.
|
||||||
|
|
||||||
|
The arm sits at z 1.810..1.890 while the lane surfaces are at 1.7805, i.e. **29 mm above
|
||||||
|
them**, so it would sweep *over* a lane edge rather than into it. The earlier reasoning that
|
||||||
|
lanes had to start beyond the swept envelope was therefore unnecessary, and it is what
|
||||||
|
opened the dead band. Closing it means one of: lanes in to ~y=0.33, a longer arm, or a
|
||||||
|
larger swing (the joint limit is +-35 authored).
|
||||||
|
|
||||||
|
**The angle does not track the command.** Measured in a demo run: an item commanded 0.0 saw
|
||||||
|
the arm at **+30.5**, one commanded +30.0 saw **-34.6**. Both then went to the same lane. So
|
||||||
|
on top of the reach gap the control sign and the settling are wrong. Two things are known
|
||||||
|
about this path and both matter:
|
||||||
|
|
||||||
|
* `Plow.target(deg)` writes the whole angle at once; calling it *and* `step_toward()` leaves
|
||||||
|
the ramp nothing to do and the blade snaps. Use one or the other, and `plow_sort` uses the
|
||||||
|
ramp.
|
||||||
|
* `Plow.home()` does not settle the arm - it sets a target, and the compliant drive needs
|
||||||
|
time. Consecutive trials therefore start from wherever the previous one left the blade
|
||||||
|
(measured: -26.4 and -14.4 at the start of runs meant to begin at 0).
|
||||||
|
|
||||||
|
* A **positive command deflects to -Y (lane B)**, opposite to the natural reading. Measured,
|
||||||
|
not assumed, exactly as the module docstring warns.
|
||||||
|
|
||||||
|
Until reach and tracking are fixed the plow does not deliver: `scripts/run_demo.py` reports
|
||||||
|
the classification honestly and a delivery rate near zero.
|
||||||
|
|
||||||
|
### Retract interlocks
|
||||||
|
The blade returns only when (a) the pushed item has cleared to y > 0.5 and (b) no other item
|
||||||
|
is inside the blade's footprint. Retracting blindly sweeps the blade back through the next
|
||||||
|
item and knocks it over — that was a real observed failure, not a hypothetical.
|
||||||
|
|
||||||
|
### Plow (`plow_cell.usd` only)
|
||||||
|
The second diverter, at x = −7.05. Mechanically the opposite of the pusher: a **dynamic
|
||||||
|
arm on a revolute joint with an angular force drive** (axis Z, limits ±35°, stiffness
|
||||||
|
120000, damping 1500, 12 kg with gravity disabled). Being force-driven it is compliant —
|
||||||
|
it yields on contact instead of teleporting through cargo — so `sim/plow.py` commands a
|
||||||
|
drive *target* rather than writing a transform.
|
||||||
|
|
||||||
|
Two consequences: a target is a request, not a position (read `Plow.angle`, which measures
|
||||||
|
the arm's simulated pose, never assume it arrived), and the rate is not free. The authored
|
||||||
|
graph swings 30° in **7 ms** (72 rad/s), which is a display animation, not a sortable
|
||||||
|
motion — at that rate the blade lands as an impulse, the same failure the pusher shows
|
||||||
|
above 2.5 m/s. `Plow.step_toward` ramps the target at `config.PLOW_RATE` (180 °/s) instead;
|
||||||
|
`config.PLOW_RATE_AUTHORED` keeps the original figure for reference.
|
||||||
|
|
||||||
|
The scene keeps its authored `DiverterAnimGraph`, so pressing Play alone demonstrates the
|
||||||
|
cell. `plow_cell.prepare(..., script_control=True)` switches that graph off — it has to go,
|
||||||
|
or it rewrites the drive target every tick and overwrites anything Python commands.
|
||||||
|
|
||||||
|
There is also a stray second `ConveyorTrack_01` at stage root, outside `/World`, left over
|
||||||
|
from how the cell was assembled; it composes as a duplicate belt in the same place and is
|
||||||
|
deactivated on load rather than deleted, so the file stays as authored.
|
||||||
|
|
||||||
|
### Laser gate
|
||||||
|
A genuine `raycast_closest` across the belt, not a coordinate test. The beam starts at
|
||||||
|
y = −0.24, which is deliberately **clear of the blade's retracted footprint** (the blade
|
||||||
|
spans y −0.33…−0.27). Start it any further out and the ray simply reads the blade and the
|
||||||
|
gate never sees cargo.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Vision (CRE-ROI v2b)
|
||||||
|
|
||||||
|
`cv/pipeline.py`. Per item, once, while it sits under the portal: FastSAM segment-everything
|
||||||
|
→ keep the blob covering the projected inspection point in every view → ROI crop to a fixed
|
||||||
|
320 px side → **one batched CRE-Stereo pass over all three crops** → fuse, dropping views
|
||||||
|
whose 3D centroid disagrees with the median by >10 cm → belt-plane OBB + `r_in/R_out`.
|
||||||
|
|
||||||
|
The crop must use the **identical column window in both eyes**, left-padded by the maximum
|
||||||
|
disparity, or the right-hand counterpart falls outside the crop. Disparity is invariant to
|
||||||
|
an equal column shift, so depth stays correct.
|
||||||
|
|
||||||
|
**The stereo rig must be rectified.** Both eyes of a pair share one orientation, with the
|
||||||
|
right eye offset along its X. Aiming each eye separately at the target verges the pair and
|
||||||
|
breaks `depth = fx·b/disp` — reconstruction came back at 2–5 m instead of 0.63 m. If
|
||||||
|
`config/calib.json` ever loses its `"rectified": true` flag, the pipeline refuses to load
|
||||||
|
rather than silently producing garbage.
|
||||||
|
|
||||||
|
Measured: dimensions land within ~34 mm of ground truth on the largest edge; the batched CRE
|
||||||
|
pass costs ~170 ms per item.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Traps that cost real time
|
||||||
|
|
||||||
|
Each of these presented as a silent failure, not an error.
|
||||||
|
|
||||||
|
- **Copied items arrive kinematic and hidden.** Kinematic bodies ignore gravity and belt
|
||||||
|
friction; hidden ones are invisible to the cameras while still simulating. Both must be
|
||||||
|
cleared, plus `sleepThreshold = 0` or a settled item is never woken by the belt.
|
||||||
|
- **`BBoxCache` / `XformCache` return the AUTHORED transform during simulation.** A moving
|
||||||
|
item looks frozen. Use `RigidPrim.get_world_poses()`.
|
||||||
|
- **`timeline.stop()` resets items to their authored poses**, so captures of a finished run
|
||||||
|
must be taken while still playing.
|
||||||
|
- **`rep.orchestrator.step_async()` stops the timeline.** Any live loop that classifies must
|
||||||
|
call `timeline.play()` again afterwards or the whole line freezes mid-run.
|
||||||
|
- **The viewport's active camera may not be Persp.** Setting Persp's pose then does nothing;
|
||||||
|
call `vp.set_active_camera("/OmniverseKit_Persp")` first.
|
||||||
|
- **Camera bodies carry cosmetic `AimRay` cones** that sit right over the inspection point
|
||||||
|
and dominate the frame. Hidden on load.
|
||||||
|
- **Stale composition:** if a stage was opened while a referenced asset was missing, USD
|
||||||
|
caches the failure. Dropping the file in later does not fix it — the prim stays valid with
|
||||||
|
`localErrors: none` but empty typeName and bbox. Diagnose with `GetPrimStack()`, fix by
|
||||||
|
re-opening the stage.
|
||||||
|
- **`Stage.TraverseAll()` crashes Isaac on this scene.** Use targeted `GetPrimAtPath`, or run
|
||||||
|
read-only checks headlessly via `/home/whatevenif/isaacsim/python.sh` (has pxr, no Kit).
|
||||||
|
- **Cube colliders:** use `size = 2.0` so the scale op equals the half-extent. Any other
|
||||||
|
arrangement makes PhysX use the wrong bounds and goods drop through the belt.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Known limitations
|
||||||
|
|
||||||
|
- **Classification is weak.** Metrology is sound but the roundness metric does not separate
|
||||||
|
classes at this scale: genuinely round items read K ≈ 0.75 against a 0.80 threshold while a
|
||||||
|
plain box reads 0.76. `config.ROUND_K` needs recalibration before the vision output should
|
||||||
|
drive the pusher for real. Use `--no-vision` to exercise mechanics on ground truth.
|
||||||
|
- **Consecutive class-D items are missed at the default 700 mm pitch.** Headway is 0.70 s and
|
||||||
|
one pusher cycle is 0.62 s; the gate is blind while a cycle runs, so a D item immediately
|
||||||
|
behind another D crosses the beam unseen. Verified: `D,C,D,B` sorts both D correctly, `D,D`
|
||||||
|
back-to-back loses the second. Fixing it needs a larger pitch, a slower belt or a second
|
||||||
|
diverter — not a faster blade, which is already at its stability limit.
|
||||||
|
- **The plow (`DiverterEnd`) is untouched in `sorter.usd`.** There its geometry was
|
||||||
|
restored and its arm returned to rest, but its control logic is left exactly as authored.
|
||||||
|
It is `scene/plow_cell.usd` that makes the plow controllable — see below.
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
# Веса моделей
|
||||||
|
|
||||||
|
В репозиторий не попадают: около 4.5 ГБ, и все скачиваются из первоисточников. Ниже —
|
||||||
|
откуда именно, чтобы окружение поднималось повторяемо.
|
||||||
|
|
||||||
|
## DEFOM-Stereo — рабочий бейзлайн пайплайна
|
||||||
|
|
||||||
|
Репозиторий: <https://github.com/Insta360-Research-Team/DEFOM-Stereo>
|
||||||
|
Веса: <https://drive.google.com/drive/folders/1cZLcIjHlmUo986gkR6FbofG1cj5BT36x>
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/Insta360-Research-Team/DEFOM-Stereo.git
|
||||||
|
cd DEFOM-Stereo && mkdir -p checkpoints && cd checkpoints
|
||||||
|
gdown 1XuAM4vqzura_6NKN70hMW5lFD4TafnDL # defomstereo_vitl_sceneflow.pth 1.53 ГБ <- используется
|
||||||
|
gdown 1qyXKO-Nxq3ndl2H0deQpo6BSvwlGKYEg # defomstereo_vits_sceneflow.pth 173 МБ (лёгкий вариант)
|
||||||
|
wget https://huggingface.co/depth-anything/Depth-Anything-V2-Large/resolve/main/depth_anything_v2_vitl.pth
|
||||||
|
```
|
||||||
|
|
||||||
|
Энкодер DepthAnythingV2 обязателен: без него модель не поднимается.
|
||||||
|
|
||||||
|
Скачивание папкой целиком (`gdown --folder`) не работает — перечисляет файлы и ничего не
|
||||||
|
качает. Только по одному идентификатору.
|
||||||
|
|
||||||
|
В репозитории есть и штатные скрипты: `scripts/download_models.sh`, `scripts/download_dav2.sh`.
|
||||||
|
|
||||||
|
## CREStereo — второй движок, точнее по габаритам
|
||||||
|
|
||||||
|
<https://github.com/megvii-research/CREStereo> — веса `crestereo_eth3d.pth` (95 МБ) по
|
||||||
|
ссылкам из его README.
|
||||||
|
|
||||||
|
## Сегментация
|
||||||
|
|
||||||
|
Обе модели в текущем бейзлайне **не используются** — товар отделяется от полотна
|
||||||
|
превышением над плоскостью. Оставлены для сравнительных прогонов.
|
||||||
|
|
||||||
|
* FastSAM-s.pt (23 МБ) — <https://github.com/CASIA-IVA-Lab/FastSAM>
|
||||||
|
* yolo26n-seg.pt — <https://huggingface.co/openvision/yolo26-s-seg> (у нас вариант `n`, не `s`)
|
||||||
|
|
||||||
|
## Fast FoundationStereo — проверялся, в бейзлайн не вошёл
|
||||||
|
|
||||||
|
<https://github.com/NVlabs/fast-foundationstereo>
|
||||||
|
Веса: <https://drive.google.com/drive/folders/1HuTt7UIp7gQsMiDvJwVuWmKpvFzIIMap>
|
||||||
|
|
||||||
|
Брался ONNX-экспорт `23_36_37_iters_4_res_320x736` (69 МБ, id `1p9vgRh_8R1FXA79l8VdnN28dDQ3NEEhf`)
|
||||||
|
и его yaml (`11oKt_6_jSdTqwBw1qotxmwqKxzYkvKdI`).
|
||||||
|
|
||||||
|
Замер получился нечестным по времени: `onnxruntime-gpu` 1.28 требует CUDA 13, на сервере
|
||||||
|
12.8, колёс под CUDA 12 для python 3.12 нет, `onnx2torch` падает на динамическом `Clip`,
|
||||||
|
TensorRT не установлен — модель шла на CPU. Точность при этом от устройства не зависит и
|
||||||
|
вышла вчетверо хуже CRE (152.8 против 32.4 мм).
|
||||||
|
|
||||||
|
## Пропсы конвейера NVIDIA
|
||||||
|
|
||||||
|
`assets/conveyors/` (274 МБ) тоже не в репозитории — восстанавливаются штатным
|
||||||
|
`python3 scripts/fetch_assets.py`. Источник: `omniverse-content-production.s3-us-west-2.amazonaws.com`,
|
||||||
|
раздел `Assets/Isaac/6.0/Isaac/Props/Conveyors/`.
|
||||||
|
|
||||||
|
## Куда смотрят пути в коде
|
||||||
|
|
||||||
|
Сейчас они абсолютные и указывают на сервер:
|
||||||
|
|
||||||
|
* `control_test/measure_flow.py`, `control_test/cv_worker.py` — константа `CV = "/home/dasha/isaac_assets/cv"`
|
||||||
|
* `control_test/measure_plane.py` — путь к `defom-stereo/checkpoints/`
|
||||||
|
|
||||||
|
При переносе на другую машину поправить в этих трёх файлах.
|
||||||
@@ -0,0 +1,252 @@
|
|||||||
|
{
|
||||||
|
"backpack": {
|
||||||
|
"zone": "B",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
455,
|
||||||
|
370,
|
||||||
|
301
|
||||||
|
],
|
||||||
|
"k_round": 0.82,
|
||||||
|
"label_ru": "Не подходит для сортировки по габаритам"
|
||||||
|
},
|
||||||
|
"bag": {
|
||||||
|
"zone": "D",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
202,
|
||||||
|
175,
|
||||||
|
170
|
||||||
|
],
|
||||||
|
"k_round": 0.896,
|
||||||
|
"label_ru": "Не подходит для сортировки без доупаковки"
|
||||||
|
},
|
||||||
|
"banana": {
|
||||||
|
"zone": "D",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
183,
|
||||||
|
71,
|
||||||
|
33
|
||||||
|
],
|
||||||
|
"k_round": 0.94,
|
||||||
|
"label_ru": "Не подходит для сортировки без доупаковки"
|
||||||
|
},
|
||||||
|
"bolts_cluster": {
|
||||||
|
"zone": "B",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
194,
|
||||||
|
136,
|
||||||
|
53
|
||||||
|
],
|
||||||
|
"k_round": 0.718,
|
||||||
|
"label_ru": "Подходит для сортировки"
|
||||||
|
},
|
||||||
|
"bottle": {
|
||||||
|
"zone": "D",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
305,
|
||||||
|
91,
|
||||||
|
91
|
||||||
|
],
|
||||||
|
"k_round": 0.995,
|
||||||
|
"label_ru": "Не подходит для сортировки без доупаковки"
|
||||||
|
},
|
||||||
|
"box_300x200x200": {
|
||||||
|
"zone": "B",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
301,
|
||||||
|
200,
|
||||||
|
200
|
||||||
|
],
|
||||||
|
"k_round": 0.72,
|
||||||
|
"label_ru": "Подходит для сортировки"
|
||||||
|
},
|
||||||
|
"box_400x400x300": {
|
||||||
|
"zone": "B",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
401,
|
||||||
|
400,
|
||||||
|
300
|
||||||
|
],
|
||||||
|
"k_round": 0.716,
|
||||||
|
"label_ru": "Не подходит для сортировки по габаритам"
|
||||||
|
},
|
||||||
|
"bucket": {
|
||||||
|
"zone": "D",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
287,
|
||||||
|
287,
|
||||||
|
272
|
||||||
|
],
|
||||||
|
"k_round": 0.995,
|
||||||
|
"label_ru": "Не подходит для сортировки без доупаковки"
|
||||||
|
},
|
||||||
|
"chip_bag": {
|
||||||
|
"zone": "D",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
250,
|
||||||
|
162,
|
||||||
|
69
|
||||||
|
],
|
||||||
|
"k_round": 0.811,
|
||||||
|
"label_ru": "Не подходит для сортировки без доупаковки"
|
||||||
|
},
|
||||||
|
"cone": {
|
||||||
|
"zone": "D",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
500,
|
||||||
|
350,
|
||||||
|
350
|
||||||
|
],
|
||||||
|
"k_round": 0.991,
|
||||||
|
"label_ru": "Не подходит для сортировки по габаритам"
|
||||||
|
},
|
||||||
|
"cylinder": {
|
||||||
|
"zone": "D",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
435,
|
||||||
|
50,
|
||||||
|
43
|
||||||
|
],
|
||||||
|
"k_round": 0.867,
|
||||||
|
"label_ru": "Не подходит для сортировки без доупаковки"
|
||||||
|
},
|
||||||
|
"detergent": {
|
||||||
|
"zone": "B",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
278,
|
||||||
|
260,
|
||||||
|
180
|
||||||
|
],
|
||||||
|
"k_round": 0.742,
|
||||||
|
"label_ru": "Подходит для сортировки"
|
||||||
|
},
|
||||||
|
"headphones": {
|
||||||
|
"zone": "D",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
198,
|
||||||
|
195,
|
||||||
|
93
|
||||||
|
],
|
||||||
|
"k_round": 0.807,
|
||||||
|
"label_ru": "Не подходит для сортировки без доупаковки"
|
||||||
|
},
|
||||||
|
"helmet": {
|
||||||
|
"zone": "D",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
354,
|
||||||
|
297,
|
||||||
|
280
|
||||||
|
],
|
||||||
|
"k_round": 0.895,
|
||||||
|
"label_ru": "Не подходит для сортировки без доупаковки"
|
||||||
|
},
|
||||||
|
"lunchbox": {
|
||||||
|
"zone": "B",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
201,
|
||||||
|
152,
|
||||||
|
62
|
||||||
|
],
|
||||||
|
"k_round": 0.646,
|
||||||
|
"label_ru": "Подходит для сортировки"
|
||||||
|
},
|
||||||
|
"mug": {
|
||||||
|
"zone": "D",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
113,
|
||||||
|
99,
|
||||||
|
83
|
||||||
|
],
|
||||||
|
"k_round": 0.985,
|
||||||
|
"label_ru": "Не подходит для сортировки без доупаковки"
|
||||||
|
},
|
||||||
|
"parcel_box": {
|
||||||
|
"zone": "B",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
344,
|
||||||
|
155,
|
||||||
|
144
|
||||||
|
],
|
||||||
|
"k_round": 0.699,
|
||||||
|
"label_ru": "Подходит для сортировки"
|
||||||
|
},
|
||||||
|
"pen": {
|
||||||
|
"zone": "D",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
148,
|
||||||
|
13,
|
||||||
|
9
|
||||||
|
],
|
||||||
|
"k_round": 0.842,
|
||||||
|
"label_ru": "Не подходит для сортировки по габаритам"
|
||||||
|
},
|
||||||
|
"perfume": {
|
||||||
|
"zone": "D",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
120,
|
||||||
|
53,
|
||||||
|
53
|
||||||
|
],
|
||||||
|
"k_round": 0.924,
|
||||||
|
"label_ru": "Не подходит для сортировки без доупаковки"
|
||||||
|
},
|
||||||
|
"pillow": {
|
||||||
|
"zone": "C",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
455,
|
||||||
|
431,
|
||||||
|
213
|
||||||
|
],
|
||||||
|
"k_round": 0.905,
|
||||||
|
"label_ru": "Не подходит для сортировки по габаритам"
|
||||||
|
},
|
||||||
|
"plate": {
|
||||||
|
"zone": "D",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
209,
|
||||||
|
209,
|
||||||
|
27
|
||||||
|
],
|
||||||
|
"k_round": 0.998,
|
||||||
|
"label_ru": "Не подходит для сортировки без доупаковки"
|
||||||
|
},
|
||||||
|
"pouf": {
|
||||||
|
"zone": "C",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
489,
|
||||||
|
489,
|
||||||
|
264
|
||||||
|
],
|
||||||
|
"k_round": 0.994,
|
||||||
|
"label_ru": "Не подходит для сортировки по габаритам"
|
||||||
|
},
|
||||||
|
"sneaker": {
|
||||||
|
"zone": "B",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
270,
|
||||||
|
208,
|
||||||
|
125
|
||||||
|
],
|
||||||
|
"k_round": 0.706,
|
||||||
|
"label_ru": "Подходит для сортировки"
|
||||||
|
},
|
||||||
|
"tool_case": {
|
||||||
|
"zone": "B",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
300,
|
||||||
|
144,
|
||||||
|
60
|
||||||
|
],
|
||||||
|
"k_round": 0.454,
|
||||||
|
"label_ru": "Подходит для сортировки"
|
||||||
|
},
|
||||||
|
"watch": {
|
||||||
|
"zone": "C",
|
||||||
|
"gt_dims_mm": [
|
||||||
|
230,
|
||||||
|
230,
|
||||||
|
5
|
||||||
|
],
|
||||||
|
"k_round": 0.995,
|
||||||
|
"label_ru": "Не подходит для сортировки по габаритам"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
After Width: | Height: | Size: 82 KiB |
|
After Width: | Height: | Size: 571 KiB |
|
After Width: | Height: | Size: 436 KiB |
|
After Width: | Height: | Size: 1.9 KiB |
|
After Width: | Height: | Size: 7.3 KiB |
|
After Width: | Height: | Size: 223 KiB |
|
After Width: | Height: | Size: 364 KiB |
|
After Width: | Height: | Size: 222 B |
|
After Width: | Height: | Size: 12 KiB |
|
After Width: | Height: | Size: 222 B |
|
After Width: | Height: | Size: 3.0 KiB |
|
After Width: | Height: | Size: 306 KiB |
|
After Width: | Height: | Size: 567 KiB |
|
After Width: | Height: | Size: 920 KiB |
|
After Width: | Height: | Size: 2.2 MiB |
|
After Width: | Height: | Size: 394 KiB |
|
After Width: | Height: | Size: 11 KiB |
|
After Width: | Height: | Size: 665 KiB |
|
After Width: | Height: | Size: 518 KiB |
|
After Width: | Height: | Size: 387 KiB |
|
After Width: | Height: | Size: 494 KiB |
|
After Width: | Height: | Size: 574 KiB |
|
After Width: | Height: | Size: 1.2 MiB |
|
After Width: | Height: | Size: 921 KiB |
|
After Width: | Height: | Size: 234 KiB |
|
After Width: | Height: | Size: 374 KiB |
|
After Width: | Height: | Size: 987 KiB |
|
After Width: | Height: | Size: 960 KiB |
|
After Width: | Height: | Size: 252 KiB |
|
After Width: | Height: | Size: 80 KiB |
|
After Width: | Height: | Size: 80 KiB |
|
After Width: | Height: | Size: 582 KiB |
|
After Width: | Height: | Size: 510 KiB |
|
After Width: | Height: | Size: 90 KiB |
|
After Width: | Height: | Size: 579 B |
|
After Width: | Height: | Size: 58 KiB |
|
After Width: | Height: | Size: 450 KiB |
|
After Width: | Height: | Size: 400 KiB |
|
After Width: | Height: | Size: 370 KiB |
|
After Width: | Height: | Size: 406 KiB |
|
After Width: | Height: | Size: 404 KiB |
|
After Width: | Height: | Size: 222 B |
|
After Width: | Height: | Size: 222 B |
|
After Width: | Height: | Size: 222 B |
|
After Width: | Height: | Size: 1.2 MiB |
|
After Width: | Height: | Size: 639 B |
|
After Width: | Height: | Size: 345 B |
|
After Width: | Height: | Size: 334 KiB |
|
After Width: | Height: | Size: 176 KiB |
|
After Width: | Height: | Size: 515 KiB |
|
After Width: | Height: | Size: 235 B |
|
After Width: | Height: | Size: 11 KiB |
|
After Width: | Height: | Size: 257 KiB |