Condition-based maintenance for a ship propulsion shaft — three unsupervised detectors tracked in MLflow, a Streamlit monitoring dashboard, and a labeled synthetic-fault benchmark (best F1 0.93)
Vibration and RPM sensors on a ship's propulsion shaft produce a continuous stream of unlabeled data. This project learns the healthy signature of that stream with three unsupervised models — Isolation Forest, a dense autoencoder, and an LSTM autoencoder — tracks every training run in MLflow, and serves the detector through a Streamlit operations dashboard with anomaly-cause attribution (which sensor deviated, by how many σ).
Because the real data is unlabeled, "X% of windows flagged" says nothing about whether the right windows were flagged. The repository therefore includes a physics-informed synthetic fault benchmark: four classic rotating-machinery faults are injected into simulated shaft vibration with known labels, and all three models are scored on precision/recall/F1 — thresholds calibrated on healthy data only, never tuned on test labels.
The original sensors are proprietary (propulsion-shaft monitoring of the Korea Maritime
& Ocean University training ship Hanbada): a tachometer (RTTN_SPDMTR) and four
vibration channels (shaft lower/upper, vertical, horizontal), summarized per 1-second
window as mean / std / RMS / peak-to-peak / kurtosis / skewness.
The raw data cannot be redistributed, so evaluation/synthetic_data_generator.py
produces a drop-in substitute with the same schema: 1×/2×/3× RPM harmonic composition,
centrifugal amplitude scaling, and sensor noise for the healthy state, plus four
injectable faults with per-channel sensitivity profiles:
| Fault | Physical signature | Feature response |
|---|---|---|
| Unbalance | 1×-RPM amplitude rise, radial-dominant | RMS / p2p ↑ |
| Misalignment | 2× (and 3×) harmonics, axial/horizontal | RMS ↑, waveform shape change |
| Bearing wear | Periodic impulses at the defect frequency | Kurtosis ↑↑ |
| Mechanical looseness | 0.5× sub-harmonic + broadband bursts | Std / skewness ↑ |
Every fault segment ramps severity from 0.15 to 1.0 (progressive degradation), so early low-severity windows are genuinely hard — the benchmark rewards early detection rather than only fully-developed failures.
2,000 healthy training windows; 2,900 test windows of which 1,440 are faults
(4 types × 3 voyages each). Threshold = 95th percentile of training scores.
Run python evaluation/synthetic_fault_eval.py to reproduce:
| Model | PR-AUC | Precision | Recall | F1 | Unbalance | Misalign | Bearing | Looseness |
|---|---|---|---|---|---|---|---|---|
| Isolation Forest | 0.845 | 0.91 | 0.75 | 0.82 | 89% | 90% | 41% | 78% |
| Dense Autoencoder | 0.975 | 0.96 | 0.91 | 0.93 | 100% | 100% | 72% | 91% |
| LSTM-AE (t=10) | 0.846 | 0.88 | 0.73 | 0.80 | 87% | 95% | 39% | 72% |
What the labels reveal:
- The dense autoencoder wins decisively (F1 0.93) — it reconstructs the joint healthy feature distribution and reacts to any deviation, catching unbalance and misalignment perfectly.
- Bearing wear is the shared bottleneck (39–72% recall): its signature lives almost entirely in kurtosis, and at low ramp severity the impulses drown in noise. This is the honest gap a "5% of windows flagged" report would never expose — and motivates envelope-spectrum features as the next step.
- The LSTM-AE does not earn its cost here: with slowly-varying cruise RPM, the temporal context adds noise rather than signal (it also has the highest false-alarm rate, 10.9%). Sequence models need faults with temporal structure — e.g. transient events — to justify themselves over point-wise models.
The rule above ("95th percentile of training scores") delivers its nominal 5% false-alarm rate on average. But shaft vibration scales with RPM², so a healthy shaft scores differently at different throttle settings. Splitting the healthy test windows by RPM quartile exposes what the average hides.
The study followed one discipline: measure the problem before proposing a fix, then make every candidate rule survive two guards that a headline F1 passes on its own.
| Threshold rule | PR-AUC | F1 | False-alarm rate by RPM quartile (low → high) | Spread |
|---|---|---|---|---|
| Static q95 (baseline) | 0.845 | 0.82 | 27.8% · 0.0% · 0.2% · 5.9% | 27.8 pt |
| POT / EVT (q=0.05) | 0.845 | 0.82 | 27.4% · 0.0% · 0.2% · 5.7% | 27.4 pt |
| Conditional on RPM | 0.905 | 0.78 | 18.5% · 3.2% · 0.7% · 4.4% | 17.8 pt |
| Rolling q95 (streaming) | 0.792 | 0.62 | 21.0% · 0.0% · 0.7% · 14.8% | 21.0 pt |
Root cause. Only 2.8% of healthy test windows have an RPM below anything seen in training (train 105–150, test 90–140) — and 97.6% of those are flagged, against 4.7% inside the trained range. The static rule spends most of its alarm budget on unfamiliar operating conditions, not on machine health.
What conditioning buys, and what it costs. Because a threshold rule is just a
score normalization (s − threshold), it can be judged threshold-free: conditioning
lifts PR-AUC 0.845 → 0.905, so this is a genuine ranking improvement, not a moved
operating point. The gain concentrates where maintenance actually operates — a
scarce alarm budget:
| Alarms allowed (of 1,460 healthy windows) | 10 | 20 | 30 | 50 | 106 | 200 |
|---|---|---|---|---|---|---|
| Static q95 — F1 | 0.00 | 0.08 | 0.30 | 0.51 | 0.82 | 0.83 |
| Conditional — F1 | 0.45 | 0.65 | 0.70 | 0.76 | 0.80 | 0.82 |
At a 10-alarm budget the static rule finds no fault at all — every one of its top alarms is an unfamiliar-RPM window. Conditioning fixes that. Honestly stated: at the default, looser operating point (106 alarms) conditioning is slightly worse (F1 0.82 → 0.80), and it only halves the regime spread rather than removing it, because the worst regime lies outside the trained RPM range where no healthy calibration data exists. The dashboard therefore warns when a reading falls there instead of silently clamping.
Two rejected alternatives, and why.
- POT / EVT reparametrizes the cutoff as a function of the target false-alarm rate, which is useful for reasoning about it — but it is still one global number, so the ranking is unchanged (PR-AUC 0.845, identical to static). It does not address regime dependence.
- Rolling / streaming quantiles are actively harmful here: the threshold recalibrates on recent data, so a slowly ramping fault drags the alarm level up behind it (right panel above) and PR-AUC drops to 0.792. Excluding flagged windows from the buffer (SPOT-style) avoids the masking but sends the false-alarm rate to 26.4%.
The deeper lesson: the Dense Autoencoder barely benefits (PR-AUC 0.975 → 0.977) because RPM is one of its input features — it reconstructs the operating point and so self-conditions already. Regime dependence is better fixed in the score than in the threshold; conditioning the threshold is the remedy when the score cannot do it itself, as with Isolation Forest.
Reproduce with python evaluation/dynamic_threshold_eval.py. The dashboard's rule is
config.THRESHOLD_MODE (conditional by default; static and pot available).
The comparison above needs labels, so it can only run on the synthetic benchmark.
Real shaft data has none — but the diagnosis that motivated the conditional rule
never needed them. Whether a flag rate holds its nominal target across operating
points is a property of healthy-baseline data plus an RPM channel, so
evaluation/threshold_field_audit.py measures it on real data directly and reports
no precision/recall/F1 at all:
python evaluation/threshold_field_audit.py \
--calibration data/field_healthy.parquet \
--audit data/field_recent.parquetIt audits the deployed MLflow model, breaks the flag rate down by operating point, compares threshold rules, and flags windows outside the calibrated range. Run against a held-out healthy period of the synthetic corpus it reproduces the pathology without using a single label — static spread 18.9% against a 5% target, and windows outside the calibrated RPM range flagged 29× more often than inside it.
One detail worth keeping in mind when reading such a report: auditing the calibration set itself shows a spread of only 6.0%, and the tool then says conditioning is probably unnecessary. The problem becomes visible only on data that visits operating points the calibration never covered — which is exactly why a pipeline validated on its own training distribution can miss it.
| Path | Contents |
|---|---|
training/train_models_mlflow.py |
Trains all three models, logs params/metrics/artifacts/models to MLflow |
training/anomaly_cause_analysis.py |
Loads the latest run and attributes each anomaly to its top-deviating sensors |
evaluation/synthetic_data_generator.py |
Physics-informed shaft-vibration simulator (healthy + 4 fault types, labeled) |
evaluation/synthetic_fault_eval.py |
Labeled benchmark: PR-AUC / P / R / F1 per model, per-fault recall |
evaluation/dynamic_threshold_eval.py |
Static vs conditional / POT / rolling thresholds, matched-alarm-budget comparison |
evaluation/threshold_method_figure.py |
Renders the method figure for how that threshold rule was chosen |
evaluation/threshold_field_audit.py |
Label-free threshold audit for real unlabeled data (flag rate by operating point, coverage) |
app.py, pages/ |
Streamlit dashboard: real-time detection, batch analysis, statistics/trends, settings |
utils/model_utils.py |
Model loading (latest MLflow run auto-resolved), prediction, cause attribution |
utils/threshold_utils.py |
Threshold rules: static quantile, RPM-conditional, POT/EVT tail fit |
docs/ |
Model selection guide, dashboard guide, visualization guide, AWS deployment notes |
pip install -r requirements.txt
# 1. Generate the synthetic data substitute (writes preprocessed_shaft_data.parquet)
python evaluation/synthetic_data_generator.py
# 2. Train all three models with MLflow tracking
python training/train_models_mlflow.py
mlflow ui --port 5000 # inspect runs at http://localhost:5000
# 3. Launch the monitoring dashboard (uses the latest Isolation Forest run)
streamlit run app.py
# 4. Reproduce the labeled benchmark (standalone — no MLflow needed)
python evaluation/synthetic_fault_eval.pyFour pages built for an operations context: Real-time Detection (window-by-window
scoring with per-sensor cause attribution), Batch Analysis (upload CSV/parquet,
score in bulk), Statistics & Trends (anomaly-rate history), and Settings
(model info, threshold tuning). A Dockerfile is included for containerized
deployment; see docs/AWS_DEPLOYMENT.md.
Main monitoring view — KPIs, score distribution against the alert threshold (shaded: its range across operating points), top anomaly-contributing sensors, the score timeline with the threshold tracking RPM, and the rolling anomaly rate. The alarms shown and the threshold drawn come from the same calibrated rule:
| Real-time detection with live verdict | Daily anomaly trend analysis |
|---|---|
 |
 |
The detection above is the conditional threshold doing its job: a +3σ vibration reading at 120 RPM scores −0.532 against a threshold of −0.476 and is flagged, while the same score at 134 RPM (threshold −0.568) would pass as normal operating noise.
- The benchmark is synthetic: fault signatures follow textbook rotating-machinery theory, but real shaft data adds sea-state loading, hull-transmitted vibration, and sensor drift that the simulator does not model. Numbers are for comparing models under identical conditions, not absolute field performance.
- The real-data pipeline remains unsupervised; deploying a model chosen on the synthetic benchmark still requires field validation against maintenance logs.
- Conditional thresholds are calibrated on RPM alone. Real voyages also vary with load, sea state, and fouling, so a field deployment would need those as conditioning variables too — and outside the calibrated range the rule clamps rather than extrapolates, which the dashboard surfaces as a warning instead of a verdict.
- Vehicle-Anomaly-Algorithm — road-CCTV trajectory anomaly detection (same evaluate-honestly methodology)
- AIS-Traffic-Model / AIS-Traffic-Ops — maritime traffic forecasting research & MLOps