tigris-bench

Reproducible benchmarks for TiGrIS. The source of truth behind every number quoted in the TiGrIS docs and blog.

Each benchmark suite is self-contained and lives in its own subdirectory: model preparation on the host, a device harness, and scripts that collect machine-parseable results. Anyone with the matching hardware should be able to reproduce any number end-to-end.

Canonical device runs use the exact compiler and runtime commits recorded in core-versions.json. Both hardware orchestrators verify the sibling checkouts, accepted plan schema, and clean tracked state before building or flashing. A development-only run may set TIGRIS_ALLOW_UNPINNED_CORE=1, but results from that override are not canonical until their exact revisions are recorded and validated.

How it is organized

Every suite follows the same three-step shape:

1. Prepare models on the host: ONNX, quantization, TiGrIS compilation, and reference outputs from ONNX Runtime.
2. Run the device harness for each configuration under test.
3. Collect results into JSON, format them as tables, and validate device outputs against the reference to catch numerical drift.

Suites are grouped by what they are measuring (e.g. latency against a peer framework, tiling overhead across memory budgets, end-to-end demos). See each suite's own README for its benchmark matrix, hardware, and exact commands.

Quick start

# Prepare sibling compiler/runtime checkouts at the commits in core-versions.json
python scripts/check_core_versions.py

# Enter the suite you want to run, then follow its README
cd <suite>/
pip install -r requirements.txt
python models/prepare.py
./scripts/run_all.sh /dev/ttyUSB0
python scripts/results.py results/raw/ -o results/summary.json

Accuracy validation is part of the pipeline: validate_accuracy.py compares device outputs against the ORT reference, so a run is rejected if the numbers do not match within tolerance.

Common dependencies

• Host: Python 3.10+, onnx, onnxruntime, tigris-ml, plus whatever a specific suite needs
• Device: ESP-IDF 5.x for ESP32 suites; toolchains for other targets as applicable
• Hardware: varies per suite, listed in each suite's README

Further reading

• Introducing TiGrIS: context for the numbers and how tiling works
• TiGrIS and tigris-runtime: the toolchain and runtime being benchmarked