Sentinel — AI-Powered Anomaly Detection Engine

Overview

Sentinel is a system metrics anomaly detection engine combining a low-level Rust collector with a Keras LSTM Autoencoder for real-time anomaly detection.

The project is built in five Agile phases. Three are complete: a Rust-based metrics collector (Phase 0), an LSTM Autoencoder trained on real system data (Phase 1), and a Detection Engine that turns reconstruction error into actionable anomaly alerts (Phase 2).

---

Architecture

Operating System (CPU, RAM)
        |
        v
Rust Collector (sysinfo, 1 sample/sec)
        |
        +--> Circular buffer (last 60s)  --> sentinel_latest.json / sentinel_buffer.json
        +--> Append-only log              --> sentinel_training.csv
        |
        v
Python Preprocessing
  segmentation -> normalization -> sliding windows (30 timesteps)
        |
        v
LSTM Autoencoder (Keras, Functional API)
  Encoder (LSTM) -> RepeatVector -> Decoder (LSTM) -> TimeDistributed(Dense)
        |
        v
Detection Engine
  reconstruction error -> percentile threshold -> severity scoring -> alert log

---

Phase 0 — Foundation & Data Collector (Rust)

A lightweight, thread-safe system metrics collector written in Rust.

What it does:

• Collects CPU and RAM metrics every second using the sysinfo crate
• Maintains a thread-safe circular buffer (Arc<Mutex<VecDeque<...>>>) holding the last 60 seconds of data
• Writes atomically to disk (.tmp → rename) to guarantee zero corruption for readers
• Appends every sample to a persistent CSV log used for model training
• Modular design split across metrics.rs (data collection) and storage.rs (persistence)

Outputs:

File	Purpose
sentinel_latest.json	Latest single snapshot — real-time inference
sentinel_buffer.json	Last 60 snapshots — short-term context
sentinel_training.csv	Append-only log — model training data

Stack: Rust, sysinfo, serde, serde_json

---

Phase 1 — ML Core (Keras LSTM Autoencoder)

An Autoencoder learns to reconstruct normal system behavior. High reconstruction error signals a deviation from that learned baseline.

Pipeline (preprocessing.py):

1. Segmentation — detects temporal gaps in the collected data (the collector was paused/resumed during collection) and assigns a segment_id so sliding windows are never built across a discontinuity
2. Normalization — MinMaxScaler on CPU/RAM, fitted once and persisted (scaler.pkl) for reuse in inference
3. Sliding windows — 30-timestep sequences built independently within each continuous segment
4. Train/validation split — 80/20, shuffled (sequence order doesn't carry temporal dependency across windows, so shuffling is safe here)

Model (model.py):

Functional API, following the official Keras autoencoder pattern:

Input (30, 2) -> LSTM Encoder (16 units) -> RepeatVector(30)
              -> LSTM Decoder (16 units, return_sequences=True)
              -> TimeDistributed(Dense(2))

Layer	Output Shape	Params
Input	(None, 30, 2)	0
LSTM Encoder	(None, 16)	1,216
RepeatVector	(None, 30, 16)	0
LSTM Decoder	(None, 30, 16)	2,112
TimeDistributed(Dense)	(None, 30, 2)	34

Total: 3,362 parameters.

Training (train.py): Adam optimizer, MSE loss, EarlyStopping on val_loss with restore_best_weights=True.

Results:

• Training data: ~7,700 real samples collected over a 2-hour work session (4 continuous segments after gap detection)
• Final loss / val_loss: 0.0039 / 0.0039 — no overfitting (train and validation loss track each other closely throughout training)
• Reconstruction error on synthetic anomalies (CPU spikes, memory leak trends, sensor saturation): 5.1x higher than on normal data

Evaluation (evaluate.py): loads the trained model, computes per-sequence reconstruction error, and generates three categories of synthetic anomalies for testing — sustained CPU spikes, monotonic memory-leak trends, and extreme saturation (simulating a stuck sensor).

---

Phase 2 — Detection Engine

Turns raw reconstruction error into a classification decision with severity levels and an alert log.

Threshold (detection.py): the anomaly threshold is the 99th percentile of reconstruction error on the normal validation set — a standard, well-documented technique in the anomaly detection literature. By construction, this means roughly 1% of normal data will sit above the threshold; this is a known statistical property of percentile-based thresholds, not a defect.

Severity scoring: classifies the ratio between observed error and threshold into low, medium, or critical, giving an operator a quick sense of how far outside normal behavior a detected anomaly is.

Alert log: every detected anomaly is written as a structured JSON Lines event (sentinel_alerts.log) with timestamp context, reconstruction error, threshold, severity, and ratio — designed to be consumed by downstream tooling rather than read as plain console output.

Evaluation results (mixed set: validation normals + synthetic anomalies):

Metric	Value
Precision	0.7576
Recall	1.0000
F1 Score	0.8621
AUC-ROC	0.9971

Target (F1 > 0.80): met.

Honest limitation: a targeted test on a continuous 10-minute window of known-normal data (571 sequences) produced 8 false positives (1.4%) — slightly above the 1% implied by the 99th-percentile threshold, but consistent with it statistically. The original Definition of Done specified "zero false positives," which is not achievable by construction with a percentile-based threshold; the system trades a small, known false-positive rate for perfect recall (no missed anomalies). This is a deliberate and documented engineering trade-off, not an unaddressed defect — see Lessons Learned.

---

Project Status

[x] Phase 0 — Foundation & Data Collector (Rust)
[x] Phase 1 — ML Core (Keras LSTM Autoencoder)
[x] Phase 2 — Detection Engine
[ ] Phase 3 — Production Layer (TFLite, real-time pipeline)
[ ] Phase 4 — Portfolio Polish (documentation, benchmarking, ADRs)

---

Lessons Learned

• Percentile-based thresholds don't support "zero false positives" as a goal. A 99th-percentile threshold guarantees roughly 1% of normal data will exceed it, by definition. Specifying "zero false positives" in a Definition of Done without accounting for this is a specification error, not an implementation one — worth catching during planning rather than after measuring.
• Temporal gap detection matters even for "simple" time series. The data collection session included a multi-hour interruption (the machine was suspended). Building sliding windows naively across that gap would have produced artificial sequences mixing unrelated time periods. Segmenting on timestamp deltas before windowing was a small addition with an outsized correctness impact.
• Synthetic anomaly severity depends entirely on how aggressive the synthetic anomalies are. Early synthetic anomalies (CPU spikes near 0.9, gradual memory-leak trends) mostly landed in the "low" severity bucket relative to the threshold. Adding a more extreme saturation case (a simulated stuck sensor) was necessary to exercise the full severity range and get a more realistic F1/AUC-ROC reading.

---

Author: Michele Grimaldi — github.com/Mike014