isaac/cv/fast-foundationstereo/cpp

Fast-FoundationStereo C++ Inference

C++ runtime for Fast-FoundationStereo stereo depth inference on TensorRT.

Run from inside the C++ Docker container

All commands in this README are intended to be run from inside the container built from docker/dockerfile_cpp. That image carries the CUDA toolkit, TensorRT 10 runtime + ONNX parser + C++ headers, OpenCV development package, trtexec, and a Python environment with PyTorch / ONNX export tooling. The plain docker/dockerfile image does not ship the C++ TRT headers and will fail to build the C++ targets.

Environment setup:

docker build --network host -t ffs -f docker/dockerfile_cpp .
bash docker/run_container.sh

run_container.sh mounts the parent of docker/ (the repo root) into /workspace, so cd /workspace/<repo-name>/cpp lands you in this folder. Every cd cpp, cpp/build/..., python3 scripts/..., and trtexec ... example below assumes that working directory.

cpp/
+-- CMakeLists.txt
+-- README.md
+-- app/
|   +-- main.cpp                  -- ffs_depth_main: single image inference + visualization
|   +-- build_single_engine.cpp   -- ffs_build_single_engine: ONNX (with FFSGWCVolume plugin) -> .engine
|   +-- profile_speed.cpp         -- ffs_profile_speed: latency profiler for either route
+-- include/
|   +-- ffs_depth_tensorrt.hpp
|   +-- ffs_depth_single_tensorrt.hpp
|   +-- ffs_gwc_plugin.hpp
+-- src/
    +-- ffs_depth_tensorrt.cpp
    +-- ffs_depth_single_tensorrt.cpp
    +-- gwc_volume_plugin.cpp
    +-- depth_kernels.cu

Two interchangeable inference routes are supported:

• FFSSingleEngineInference loads one TensorRT engine that contains the FFSGWCVolume plugin node.
• FFSDepthInference loads two TensorRT engines (feature_runner + post_runner) and computes the GWC cost volume between them with a hand-written CUDA kernel.

ffs_depth_main auto-detects the route based on the engine directory contents:

• If the directory contains fast_foundationstereo.engine, it uses the single-engine plugin path.
• Otherwise it uses the two-engine reference path.

Inputs and outputs are the same in both cases:

• Input: a stereo image pair plus an intrinsic file (demo_data/K.txt format: 9 floats on line 1 for the 3x3 camera matrix, one float on line 2 for the stereo baseline in meters).
• Output: float32 disparity (input-pixel units), float32 depth in meters, and PNG visualizations.

Dependencies

All C++ build dependencies are provided by the docker/dockerfile_cpp image (see Run from inside the C++ Docker container for how to build and enter it):

• CUDA Toolkit (for nvcc and the CUDA runtime).
• TensorRT 10 runtime, ONNX parser, and C++ headers.
• OpenCV development package (image I/O and depth visualization in app/main.cpp).
• trtexec, used by Route B to build the two-engine TensorRT engines from ONNX.

Build

cd cpp
cmake -B build
cmake --build build -j

This produces:

• build/libffs_gwc_plugin.so -- the FFSGWCVolume plugin as a shared library (loadable by trtexec --staticPlugins=...).
• build/libffs_depth_inference.a -- the inference static library.
• build/ffs_build_single_engine -- single-engine builder.
• build/ffs_depth_main -- demo / inference CLI.
• build/ffs_profile_speed -- latency profiler.

Route A: Single Engine with FFSGWCVolume Plugin

A.1 Export ONNX with the plugin node

scripts/make_plugin_onnx.py exports one ONNX graph in which the GWC cost volume is represented by an FFSGWCVolume custom plugin node (resolved at engine-build time by libffs_gwc_plugin.so):

python3 scripts/make_plugin_onnx.py \
  --model_dir weights/23-36-37/model_best_bp2_serialize.pth \
  --save_path output_plugin_onnx \
  --height 480 \
  --width 640 \
  --valid_iters 8 \
  --max_disp 192

This writes:

output_plugin_onnx/
+-- fast_foundationstereo_plugin.onnx
+-- onnx.yaml

A.2 Build the single TensorRT engine

cpp/build/ffs_build_single_engine \
  output_plugin_onnx/fast_foundationstereo_plugin.onnx \
  output_plugin_onnx/fast_foundationstereo.engine

By default the engine is built with FP16 enabled. Pass --fp32 to disable FP16. Pass --workspace-mb N to override the workspace (default 4096 MB).

After this step output_plugin_onnx/ contains everything the runtime needs:

output_plugin_onnx/
+-- fast_foundationstereo.engine
+-- onnx.yaml

A.3 Run inference

Run from the repository root:

cpp/build/ffs_depth_main \
  output_plugin_onnx \
  demo_data/left.png \
  demo_data/right.png \
  demo_data/K.txt \
  output_plugin_onnx

The last argument is the output directory (default ffs_output). ffs_depth_main sees fast_foundationstereo.engine inside output_plugin_onnx/ and uses FFSSingleEngineInference, which deserializes the engine and registers the FFSGWCVolume plugin before inference.

A.4 Python alternative (build engine and run inference without the C++ apps)

The same plugin ONNX can be turned into an engine and executed end-to-end from Python. Only the C++ plugin shared library (libffs_gwc_plugin.so) is required from the C++ build; the C++ apps (ffs_build_single_engine, ffs_depth_main) are not.

# 1. Export plugin ONNX
python3 scripts/make_plugin_onnx.py \
  --model_dir weights/23-36-37/model_best_bp2_serialize.pth \
  --save_path output_plugin_onnx \
  --height 480 \
  --width 640

# 2. Build the C++ plugin shared library
cmake -S cpp -B cpp/build
cmake --build cpp/build -j

# 3. Build the TensorRT engine from Python
python3 scripts/build_plugin_trt.py \
  output_plugin_onnx/fast_foundationstereo_plugin.onnx \
  output_plugin_onnx/fast_foundationstereo.engine

# 4. Run inference from Python
python3 scripts/run_demo_plugin_trt.py \
  --model_dir output_plugin_onnx \
  --left_file demo_data/left.png \
  --right_file demo_data/right.png \
  --intrinsic_file demo_data/K.txt \
  --out_dir output_plugin_onnx

Both build_plugin_trt.py and run_demo_plugin_trt.py auto-discover libffs_gwc_plugin.so in cpp/build/. Pass --plugin_lib /path/to/libffs_gwc_plugin.so to override the location. build_plugin_trt.py accepts --fp32 and --workspace-mb N with the same meaning as ffs_build_single_engine. run_demo_plugin_trt.py writes the same five output files as the C++ demo (see Outputs).

Route B: Two TensorRT Engines (feature_runner + post_runner)

B.1 Build the two engines

See the Two-stage ONNX section in the top-level README for how to export feature_runner.onnx / post_runner.onnx with scripts/make_onnx.py and then build feature_runner.engine / post_runner.engine with trtexec. After running those steps your engine directory should contain:

output_two_onnx/
+-- feature_runner.engine
+-- post_runner.engine
+-- onnx.yaml

The two engines do not use the FFSGWCVolume plugin: the GWC cost volume is computed externally on GPU by cpp/src/depth_kernels.cu between the two engine calls, so a plain trtexec --fp16 build with no custom plugin library is enough.

B.2 Run inference

cpp/build/ffs_depth_main \
  output_two_onnx \
  demo_data/left.png \
  demo_data/right.png \
  demo_data/K.txt \
  output_two_onnx

ffs_depth_main sees no fast_foundationstereo.engine in the directory and falls back to FFSDepthInference, which executes the feature engine, builds the GWC volume with the CUDA kernel, then executes the post engine.

Outputs

Both routes write the same files into the output directory:

• disparity.bin -- raw float32 disparity (input-pixel units), prefixed by int32 [height, width].
• depth_meter.bin -- raw float32 depth in meters, prefixed by int32 [height, width].
• depth_meter.npy -- NumPy float32 depth in meters, shape [height, width].
• disp_vis.png -- left/right/colorized-disparity side-by-side visualization.
• depth_vis.png -- colorized depth visualization.

Profile Speed

ffs_profile_speed benchmarks either route end-to-end on a single image pair using CUDA events for GPU-side timing and steady_clock for host wall time. It auto-detects the route the same way ffs_depth_main does (presence of fast_foundationstereo.engine in the engine directory selects single-engine), and --mode can be set explicitly:

cpp/build/ffs_profile_speed <engine_dir> <left_image> <right_image> <intrinsic_file> \
    [--mode auto|two|single] [--warmup N] [--runs N] [--include-depth]

Defaults: --mode auto, --warmup 10, --runs 30. With --include-depth, the disparity-to-depth conversion is included in the timed region (otherwise only the infer() call is timed).

Profile the single engine

cpp/build/ffs_profile_speed \
  output_plugin_onnx \
  demo_data/left.png demo_data/right.png demo_data/K.txt \
  --mode single --warmup 20 --runs 100

Profile the two engines

cpp/build/ffs_profile_speed \
  output_two_onnx \
  demo_data/left.png demo_data/right.png demo_data/K.txt \
  --mode two --warmup 20 --runs 100

The output looks like:

mode=single
image=960x540
model=640x480
warmup=20 runs=100
timed_region=infer
gpu  mean_ms=... p50_ms=... p90_ms=... min_ms=... max_ms=... std_ms=...
host mean_ms=... p50_ms=... p90_ms=... min_ms=... max_ms=... std_ms=...

• gpu is cudaEventElapsedTime between start/stop events on the inference stream (pure GPU work).
• host is std::chrono::steady_clock around the same region, including the cost of cudaEventSynchronize. host >= gpu always.

For a side-by-side comparison, run the profiler twice with different engine directories and --mode, then compare the gpu mean_ms columns.