ORCA — Orchestration & Risk Control for Agents

The safety and security system for AI agents: it checks every risky action before it runs, scores whether outputs can be trusted, freezes agents for human sign-off, and keeps a record no agent can secretly edit.

LTTS Engineering Intelligence Hackathon · Problem GOH-UC-017 (Agent Control Tower)

Everyone builds dashboards that tell you an AI agent already broke. ORCA is the safety interlock that catches it before it acts — like the hard-stop on a factory machine that won't let an unsafe move begin.

Why this wins

Everyone else	ORCA
Observability: reports problems after they happen	Enforcement: the Policy Gate decides before anything executes
Trusts agent outputs	A second judge cross-checks outputs; low confidence → a human decides
Editable log files	Tamper-evident hash chain — alter one record and it visibly breaks
Works for one framework	Enforces on wrapped tools — proven inside a real LangGraph graph
Office chatbots	Built for safety-critical engineering: plant, factory, machinery

Timely: EU AI Act high-risk rules are enforceable Aug 2, 2026 — automatic event recording (Art. 12), human oversight (Art. 14), retention (Art. 19). ORCA's audit chain and compliance report are shaped around exactly that.

Architecture

  agents (ORCA harness · LangGraph via ToolGuard · any framework)
     │  every action is PROPOSED, never just executed
     ▼
┌─────────────────────────────────────────────────────────────┐
│ POLICY GATE — deterministic rules, in order                 │
│  denylist → poisoned-input guard → per-task allowlist       │
│  (least privilege, deny-by-default) → safety interlock      │
└──────┬──────────────────┬──────────────────┬────────────────┘
     ALLOW            ESCALATE             BLOCK
       │           agent FREEZES for      never runs ·
       ▼           human approve /        OWASP-tagged alert
  tool executes    give up / try another
       │
       ▼
┌─────────────────────────────────────────────────────────────┐
│ QUALITY GATE (optional, per-step rubric) — a second judge   │
│ re-checks the output; disagreement → low confidence →       │
│ FREEZE for human review (never auto-approved)               │
└──────────────────────────┬──────────────────────────────────┘
                           ▼
  TRIPWIRES  token budget (per trace) · loop detection · silent failure
                           ▼
┌─────────────────────────────────────────────────────────────┐
│ HASH-CHAINED EVENT STORE (sha256, single writer)            │
│ → REST + WebSocket API → dashboard · lineage DAG · replay   │
│ → cost attribution · alerts · compliance report (download)  │
└─────────────────────────────────────────────────────────────┘

One schema powers everything: every event carries trace_id (the run), parent_id (what triggered it — draws the DAG), policy verdicts, judge reviews, human approvals, costs, and its place in the hash chain. Full API contract: docs/API.md.

Quickstart

You need two terminals — one for the backend API, one for the website.

Terminal 1 — backend (from the repo root, NOT from a subfolder):

python3 -m venv .venv && source .venv/bin/activate
pip install fastapi 'uvicorn[standard]' pydantic websockets
pip install langgraph   # only for the cross-framework test

uvicorn orca.api:app --port 8000

Terminal 2 — the website (needs Node 20.19+ / 22.12+):

cd orca-frontend && npm install && npm run dev

Now open http://localhost:5173 — that's the site (Control Tower, approval inbox, policy gate, governance, lineage, cost analytics, audit replay — all live).

Common mistakes (404s):

• http://localhost:8000 is the API, not the website — it serves JSON. The site is at http://localhost:5173. (API docs: localhost:8000/docs.)
• The uvicorn command must run from the repo root (where orca/ is), or you'll get "Could not import module orca.api".
• If npm run dev/npm run build fails with "Cannot find native binding": rm -rf node_modules package-lock.json && npm install (known npm bug).

Optional dev pages (no frontend needed): open orca/preview.html and open orca/replay.html after the backend is up.

The plant demo scenario runs on a loop. It will freeze and wait for you — that's the point: approve or reject in the preview page and watch it resume.

Run the proofs (no server needed, all deterministic):

python -m orca.test_store        # hash chain catches tampering at the exact event
python -m orca.test_gate         # block / escalate / approve / deny flow
python -m orca.test_scope        # least privilege + poisoned-doc block (OWASP-LLM01)
python -m orca.test_approvals    # agent thread genuinely freezes until a human decides
python -m orca.test_quality      # judge disagreement → flagged + frozen
python -m orca.test_lineage      # trace-back across 3 agents to the root cause
python -m orca.test_compliance   # report says BROKEN if the log was touched
python -m orca.test_tripwires    # token budget · loop · silent failure
python -m orca.test_adapters     # enforcement inside a real LangGraph graph

The demo (4 beats)

Scenario: a plant-modernization workflow — Doc agent reads a P&ID → Safety agent runs the HAZOP check → Procurement orders the part → Plant Control actuates the valve. The plant is simulated; the guardrail is production-real.

1. Bad read, caught by the judge. The Doc agent misreads the valve's pressure rating (150 vs 1500 PSI). The judge disagrees per its rubric → confidence 0.25 → the run freezes for human review. Click any event in replay.html to trace it back to the exact source.
2. The attack (showstopper). The P&ID is poisoned — hidden instructions tell Procurement to export the vendor list. The Policy Gate sees untrusted input + a dangerous action and blocks it before execution, tagged OWASP-LLM01. The tool's code never ran.
3. The interlock. Plant Control proposes an irreversible valve command → the agent freezes until a named human signs off. "Who approved this?" is permanently in the chain.
4. The audit. Download the compliance report (button in either page): chain VERIFIED ✓, every block, every reviewer, every cost. Then tamper with one DB row and regenerate — BROKEN ✗ with the exact event named.

Bonus: click ☠ trigger rogue agent — a runaway agent loops, blows its token budget, and a second one dies silently; all three tripwires fire live.

What's real vs simulated

Real: the schema/normalization, Policy Gate + scoped permissions, the poisoned-doc block, freeze/resume human approvals, the judge cross-check, the hash-chained audit + verification + report, DAG trace-back, tripwires, LangGraph tool-level enforcement. Simulated: the plant and its tools (stubs), demo agents are scripted so the on-stage run is deterministic, judge defaults to a rubric-based mock (JUDGE_PROVIDER=llm switches to a real second LLM).

Repo layout

orca/               the product
  schema.py         the locked event contract
  store.py          hash-chained SQLite event store
  policy.py         Policy Gate + rules (deny/taint/scope/interlock)
  harness.py        agent loop: propose → gate → execute/freeze
  quality.py        judge cross-check (mock + LLM)
  approvals.py      freeze/decide/resume human-in-the-loop queue
  tripwires.py      token budget · loop · silent-failure wires
  adapters.py       ToolGuard: enforcement for LangGraph/any framework
  lineage.py        trace-back · blast radius · depth layers
  compliance.py     audit report (JSON + markdown download)
  api.py            FastAPI: REST + WebSocket (docs/API.md)
  scenario.py       deterministic plant demo + rogue-agent demo
  preview.html      dev dashboard (approvals, alerts, report button)
  replay.html       replay & lineage viewer
  test_*.py         nine runnable proofs
orca-frontend/      the real dashboard (React + Vite; src/lib/orca.js is
                    the single API/WebSocket layer every page uses)
docs/API.md         frontend integration contract
DECISIONS.md        kickoff + team decisions
backend_starter/    early prototype (pre-ORCA schema; superseded)

Built in 24 hours at the LTTS Engineering Intelligence Hackathon by team bug bytes.