Module-level design docs for Heron's data pipeline. Ordered to follow the packet-to-metric flow.
For cross-cutting terminology (TTFT/E2E/TPOT, wire_api, agent_kind, HttpExchange, LlmCall, AgentTurn, …) see ../glossary.md.
| # | Document | Crate | Description |
|---|---|---|---|
| 01 | Architecture | — | Monorepo layout, pipeline topology, crate dependency graph |
| 02 | Capture | h-capture |
libpcap + cloud-probe ZMQ packet acquisition, + eBPF SSL-uprobe on-host TLS capture (Linux) |
| 02b | eBPF static targets | h-capture |
Byte-signature offset uprobes for static, symbol-stripped TLS (Bun / Claude Code) |
| 03 | LLM | ts-llm |
Wire-API detection, registry + extractor pattern |
| 04 | Turn | ts-turn |
Agent interaction (turn) grouping state machine |
| 05 | Metrics | ts-metrics |
Sliding-window aggregation, t-digest percentiles |
| 06 | Storage | ts-storage |
Pluggable backend trait, write buffer, batch flush |
| 07 | Schema | ts-storage |
agent_turns, llm_calls, llm_metrics table definitions |
| 08 | Internal Metrics | ts-common |
Operational self-monitoring (counters, gauges) |
| 09 | Body cap | h-llm / h-common |
Stored-body head+tail sampling for 1M-token contexts |
| 10 | sglake backend | h-storage-sglake |
Append-only log-platform backend: index layout, wildcard tiers, retry/ack, measured costs |
capture → protocol (net + http) → llm → turn → metrics → storage
02 (in 01) 03 04 05 06/07
- Prefix —
XX-numeric prefix sets reading order (pipeline flow). - Scope — Each doc covers one crate or cross-cutting concern.
- Updates — Keep docs in sync with code. If you change a crate's interface, update its design doc.