Add ordinal + calibration metrics, CV reporting, and CI-built model artifact (#2)

* Update ci.yml

* Update .gitignore

* Add files via upload

Author: AmirhosseinHonardoust

.gitignore

@@ -23,3 +23,8 @@ env/
 
 # Scored outputs produced at runtime
 *_scored.csv
+
+# Trained model is a build artifact (produced by `python -m src.train_model`
+# and by CI), not committed. Small reporting figures/metrics are kept for the
+# README to render.
+models/*.joblib

README.md

@@ -39,14 +39,37 @@ The pipeline performs:
 
 Trained on 1,200 sessions, evaluated on a held-out 300-session test set:
 
-| Metric | Value |
-| ------ | ----- |
-| Test accuracy | **0.483** |
-| Majority-class baseline | 0.200 |
-| Lift over baseline | **+0.283** |
-| Macro F1 | ~0.47 |
+| Metric | Value | What it tells you |
+| ------ | ----- | ----------------- |
+| Test accuracy | **0.483** | Exact-rating hit rate (single split) |
+| 5-fold CV accuracy | **0.429 ± 0.036** | Stable across folds — the split above isn't a lucky one |
+| Majority-class baseline | 0.200 | Chance for a balanced 5-class target |
+| Lift over baseline | **+0.283** | ~2.4× baseline |
+| Quadratic-weighted kappa | **0.726** | Agreement that rewards *near* predictions (ordinal quality) |
+| Mean absolute error | **0.71** | Average rating distance — most misses are off-by-one |
+| Log loss | 1.25 | Probability quality |
+| Expected calibration error | 0.091 | Confidence-vs-accuracy gap (see reliability diagram) |
 
-Chance for a balanced 5-class problem is 0.20, so the model reaches roughly **2.4×** the baseline. This is a deliberately honest, moderate result: satisfaction is noisy by construction, and adjacent ratings are genuinely hard to separate — which is what a real ordinal satisfaction signal looks like. Metrics are written to [`reports/metrics/metrics.json`](reports/metrics/metrics.json) on every training run.
+The single-split accuracy (0.483) and the cross-validated mean (0.429 ± 0.036) are reported together so the headline isn't cherry-picked. The high **quadratic-weighted kappa (0.73)** is the important number for an ordered target: the model rarely makes large errors — most mistakes are adjacent ratings. All values are written to [`reports/metrics/metrics.json`](reports/metrics/metrics.json) on every training run.
+
+### Is ordinal regression better here?
+
+The 1–5 target is ordered, so a natural question is whether a dedicated **ordinal** model beats flat multi-class. This is tested reproducibly in [`src/evaluate.py`](src/evaluate.py) (`compare_ordinal_vs_flat`), which fits a Frank & Hall ordinal wrapper ([`src/ordinal.py`](src/ordinal.py)) on the same split and writes [`reports/metrics/ordinal_comparison.json`](reports/metrics/ordinal_comparison.json):
+
+| Model | Accuracy | MAE | QWK |
+| ----- | -------- | --- | --- |
+| Flat RandomForest (default) | **0.483** | **0.71** | **0.726** |
+| Frank & Hall ordinal RF | 0.397 | 1.01 | 0.640 |
+
+The flat forest wins on every metric — it already respects the ordering via `class_weight="balanced"` and the SHAP-visible structure — so it stays the default. The ordinal model is kept as a documented, tested alternative rather than dropped, so the conclusion is checkable, not asserted.
+
+### Calibration
+
+`predict_proba` reliability is measured with a reliability diagram and Expected Calibration Error (ECE):
+
+<img src="reports/figures/reliability_diagram.png" width="480" alt="reliability diagram" />
+
+The RandomForest is moderately calibrated (ECE ≈ 0.09). Post-hoc **sigmoid (Platt) recalibration** reduces ECE to ≈ 0.06 but costs ~3 points of accuracy, so it is reported rather than applied by default; enable it downstream with `sklearn.calibration.CalibratedClassifierCV` if trustworthy probabilities matter more than exact-rating accuracy for your use case.
 
 ---
 
@@ -64,17 +87,19 @@ AI-Assistant-Satisfaction-Prediction-Engine/
 │       ├── sessions_train.csv
 │       └── sessions_test.csv
 │
-├── models/
-│   └── satisfaction_pipeline.joblib        # Trained ML pipeline
+├── models/                                 # satisfaction_pipeline.joblib is a
+│   └── .gitkeep                            #   build artifact (git-ignored, see below)
 │
 ├── reports/
 │   ├── metrics/
 │   │   ├── metrics.json
-│   │   └── classification_report.json
+│   │   ├── classification_report.json
+│   │   └── ordinal_comparison.json         # flat-RF vs ordinal experiment
 │   └── figures/
 │       ├── confusion_matrix.png
 │       ├── satisfaction_distribution.png
 │       ├── per_class_f1.png
+│       ├── reliability_diagram.png         # calibration curve
 │       └── shap_summary.png
 │
 ├── src/
@@ -85,13 +110,15 @@ AI-Assistant-Satisfaction-Prediction-Engine/
 │   ├── data_prep.py
 │   ├── features.py
 │   ├── shap_utils.py                       # Version-robust SHAP shape helpers
+│   ├── model_metrics.py                    # Ordinal + calibration metric helpers
+│   ├── ordinal.py                          # Frank & Hall ordinal classifier
 │   ├── train_model.py
 │   ├── evaluate.py
 │   ├── explain.py
 │   └── score_new_sessions.py
 │
-├── tests/                                  # pytest suite
-├── .github/workflows/ci.yml                # ruff + black + mypy + pytest
+├── tests/                                  # pytest suite (20 tests)
+├── .github/workflows/ci.yml                # gate + full-pipeline job w/ artifacts
 ├── pyproject.toml                          # Tooling configuration
 ├── requirements.txt
 ├── LICENSE
@@ -222,12 +249,17 @@ pip install -r requirements.txt
 
 python -m src.generate_data   # 1) create the synthetic raw dataset
 python -m src.data_prep       # 2) time features + stratified train/test split
-python -m src.train_model     # 3) train, evaluate, save model + metrics
-python -m src.evaluate        # 4) confusion matrix, distribution, per-class F1
+python -m src.train_model     # 3) train, save model, write accuracy/CV/QWK/MAE metrics
+python -m src.evaluate        # 4) confusion matrix, distribution, per-class F1,
+                              #    reliability diagram, ordinal-vs-flat comparison
 python -m src.explain         # 5) SHAP global summary
 streamlit run app.py          # 6) launch the dashboard
 ```
 
+## Model artifact
+
+`models/satisfaction_pipeline.joblib` is a **build artifact**, not committed to git (it is `.gitignore`d). Because the whole pipeline is deterministic, you reproduce it exactly with `python -m src.train_model`. CI also trains it on every run and publishes it (plus `reports/`) as a downloadable **GitHub Actions artifact**. Run steps 1–3 above before launching the dashboard or scoring new sessions.
+
 ---
 
 # **Quality Gate**
@@ -238,10 +270,12 @@ This repo ships a CI-enforced quality gate (see [`.github/workflows/ci.yml`](.gi
 ruff check .          # lint (E,F,I,B,SIM,UP), line-length 100
 black --check .       # formatting, line-length 100
 mypy src app.py       # static type checking
-pytest                # unit tests (heavy shap/streamlit tests skip cleanly)
+pytest                # 20 unit tests (heavy shap/streamlit tests skip cleanly)
 ```
 
-Tests cover time-feature engineering, the training pipeline, the stratified split, batch scoring round-trips, the documented data signal, and the version-robust SHAP shape helpers.
+CI runs two jobs: a **quality-gate** job (the four commands above, on Python 3.10 and 3.12) and a **pipeline** job that executes the full `generate_data → data_prep → train_model → evaluate → explain` chain end-to-end and uploads the trained model and reports as artifacts — so the training path itself is exercised in CI, not just the unit tests.
+
+The 20 tests cover time-feature engineering, the training pipeline, the stratified split, batch scoring round-trips, the documented data signal, the version-robust SHAP shape helpers, the ordinal (Frank & Hall) classifier, and the calibration/ordinal metric helpers (ECE, MAE, QWK).
 
 ---
 
@@ -273,19 +307,18 @@ Python 3.10+, pandas, numpy, scikit-learn, matplotlib, seaborn, SHAP, Streamlit,
 
 # **Limitations**
 
-* Synthetic data: results reflect a *designed* signal, not real users. Real satisfaction would be noisier and confounded.
-* Moderate accuracy (~0.48) is intentional — the noise term keeps the task realistic.
-* Treated as flat multi-class rather than ordinal.
+* Synthetic data: results reflect a *designed* signal, not real users. Real satisfaction would be noisier and confounded — this is the main bound on the findings.
+* Moderate accuracy (~0.48, CV 0.43) is intentional — the noise term keeps the task realistic; quadratic-weighted kappa (0.73) shows errors are mostly adjacent.
+* The target is ordered; a Frank & Hall ordinal model was evaluated and did **not** beat the flat forest here (see Results), so flat multi-class is retained.
 * No per-user personalization or temporal modeling.
 
 ---
 
 # **Future Work**
 
-* Reframe as **ordinal regression** to exploit the natural ordering.
-* Add calibration plots and probability calibration.
-* Add temporal / sequence features per user.
-* Deploy the scorer as a FastAPI microservice.
+* Validate on a **real** (non-synthetic) dataset — the single biggest open item.
+* Add temporal / sequence features per user and per-user personalization.
+* Deploy the scorer as a FastAPI microservice with the CI-built model artifact.
 
 ---
 

reports/figures/reliability_diagram.png

@@ -2,5 +2,11 @@
   "accuracy": 0.48333333333333334,
   "baseline_accuracy": 0.2,
   "lift_over_baseline": 0.2833333333333333,
-  "n_test_samples": 300
+  "cv_accuracy_mean": 0.4293333333333333,
+  "cv_accuracy_std": 0.03561522770451489,
+  "mae": 0.7133333333333334,
+  "quadratic_weighted_kappa": 0.725609756097561,
+  "log_loss": 1.2513373472029916,
+  "n_test_samples": 300,
+  "random_state": 42
 }

reports/figures/shap_summary.png

@@ -0,0 +1,12 @@
+{
+  "flat_rf": {
+    "accuracy": 0.48333333333333334,
+    "mae": 0.7133333333333334,
+    "quadratic_weighted_kappa": 0.725609756097561
+  },
+  "ordinal_frank_hall": {
+    "accuracy": 0.39666666666666667,
+    "mae": 1.0066666666666666,
+    "quadratic_weighted_kappa": 0.6404109589041096
+  }
+}

reports/metrics/metrics.json

@@ -7,7 +7,7 @@
 import numpy as np
 import pandas as pd
 import seaborn as sns
-from sklearn.metrics import classification_report, confusion_matrix
+from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
 
 from .config import (
     FIGURES_DIR,
@@ -16,6 +16,13 @@
     PROCESSED_DATA_DIR,
     TARGET_COL,
 )
+from .features import build_model, build_preprocessor
+from .model_metrics import (
+    calibration_curve_points,
+    expected_calibration_error,
+    ordinal_metrics,
+)
+from .ordinal import OrdinalClassifier
 
 
 def load_model_and_data():
@@ -99,12 +106,76 @@ def plot_per_class_f1(report_dict) -> None:
     print(f"Saved per-class F1 plot to: {f1_path}")
 
 
+def plot_reliability_diagram(y_true, proba, classes) -> float:
+    """Save a reliability diagram and return the Expected Calibration Error."""
+    conf, acc, counts = calibration_curve_points(y_true, proba, classes)
+    ece = expected_calibration_error(y_true, proba, classes)
+
+    fig, ax = plt.subplots()
+    ax.plot([0, 1], [0, 1], "--", color="grey", label="Perfectly calibrated")
+    ax.plot(conf, acc, "o-", label="Model")
+    ax.set_xlabel("Mean predicted confidence")
+    ax.set_ylabel("Empirical accuracy")
+    ax.set_title(f"Reliability Diagram (ECE = {ece:.3f})")
+    ax.set_xlim(0, 1)
+    ax.set_ylim(0, 1)
+    ax.legend()
+
+    path = FIGURES_DIR / "reliability_diagram.png"
+    plt.tight_layout()
+    plt.savefig(path)
+    plt.close()
+    print(f"Saved reliability diagram to: {path}  (ECE={ece:.3f})")
+    return ece
+
+
+def compare_ordinal_vs_flat() -> dict:
+    """Fit a flat RF and a Frank-Hall ordinal RF on the same split and compare.
+
+    Makes the "should this be ordinal regression?" question reproducible. On the
+    synthetic data the flat forest wins on every metric, so it stays the default.
+    """
+    train_df = pd.read_csv(PROCESSED_DATA_DIR / "sessions_train.csv")
+    test_df = pd.read_csv(PROCESSED_DATA_DIR / "sessions_test.csv")
+
+    X_train = train_df.drop(columns=[TARGET_COL])
+    y_train = train_df[TARGET_COL]
+    X_test = test_df.drop(columns=[TARGET_COL])
+    y_test = test_df[TARGET_COL]
+
+    pre = build_preprocessor().fit(X_train)
+    Xtr, Xte = pre.transform(X_train), pre.transform(X_test)
+
+    flat = build_model().fit(Xtr, y_train)
+    flat_pred = flat.predict(Xte)
+
+    ordinal = OrdinalClassifier(build_model()).fit(Xtr, y_train.values)
+    ord_pred = ordinal.predict(Xte)
+
+    def _summary(pred) -> dict:
+        return {"accuracy": float(accuracy_score(y_test, pred)), **ordinal_metrics(y_test, pred)}
+
+    result = {
+        "flat_rf": _summary(flat_pred),
+        "ordinal_frank_hall": _summary(ord_pred),
+    }
+    out = METRICS_DIR / "ordinal_comparison.json"
+    with out.open("w") as f:
+        json.dump(result, f, indent=2)
+    print(f"Saved ordinal-vs-flat comparison to: {out}")
+    print(f"  flat RF   : {result['flat_rf']}")
+    print(f"  ordinal   : {result['ordinal_frank_hall']}")
+    return result
+
+
 def main() -> None:
     model, X_test, y_test = load_model_and_data()
     y_pred = model.predict(X_test)
+    proba = model.predict_proba(X_test)
 
     plot_confusion_matrix(y_test, y_pred)
     plot_satisfaction_distribution(y_test, y_pred)
+    plot_reliability_diagram(y_test, proba, model.classes_)
 
     report = classification_report(y_test, y_pred, output_dict=True, digits=3)
     report_path = METRICS_DIR / "classification_report.json"
@@ -113,6 +184,7 @@ def main() -> None:
     print(f"Saved updated classification report to: {report_path}")
 
     plot_per_class_f1(report)
+    compare_ordinal_vs_flat()
 
 
 if __name__ == "__main__":

reports/metrics/ordinal_comparison.json

@@ -0,0 +1,76 @@
+"""Evaluation metric helpers.
+
+Small, dependency-light functions so the extra reporting (ordinal quality,
+probability calibration, cross-validated stability) is unit-testable and shared
+between ``train_model`` and ``evaluate``.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+from sklearn.metrics import cohen_kappa_score, log_loss, mean_absolute_error
+
+
+def ordinal_metrics(y_true, y_pred) -> dict:
+    """Mean absolute error and quadratic-weighted kappa.
+
+    Both reward predictions that land *near* the true rating, which is what we
+    care about for an ordered 1-5 target -- plain accuracy treats "off by one"
+    and "off by four" identically.
+    """
+    return {
+        "mae": float(mean_absolute_error(y_true, y_pred)),
+        "quadratic_weighted_kappa": float(cohen_kappa_score(y_true, y_pred, weights="quadratic")),
+    }
+
+
+def expected_calibration_error(y_true, proba, classes, n_bins: int = 10) -> float:
+    """Expected Calibration Error of the top-1 prediction.
+
+    Bins predictions by confidence and averages |accuracy - confidence| weighted
+    by bin population. 0 = perfectly calibrated.
+    """
+    y_true = np.asarray(y_true)
+    proba = np.asarray(proba)
+    classes = np.asarray(classes)
+
+    confidence = proba.max(axis=1)
+    predictions = classes[proba.argmax(axis=1)]
+    correct = (predictions == y_true).astype(float)
+
+    edges = np.linspace(0.0, 1.0, n_bins + 1)
+    ece = 0.0
+    n = len(y_true)
+    for i in range(n_bins):
+        lo, hi = edges[i], edges[i + 1]
+        mask = (confidence > lo) & (confidence <= hi)
+        count = int(mask.sum())
+        if count:
+            ece += abs(correct[mask].mean() - confidence[mask].mean()) * count / n
+    return float(ece)
+
+
+def calibration_curve_points(y_true, proba, classes, n_bins: int = 10):
+    """Return (mean_confidence, accuracy, count) per confidence bin for plotting."""
+    y_true = np.asarray(y_true)
+    proba = np.asarray(proba)
+    classes = np.asarray(classes)
+
+    confidence = proba.max(axis=1)
+    predictions = classes[proba.argmax(axis=1)]
+    correct = (predictions == y_true).astype(float)
+
+    edges = np.linspace(0.0, 1.0, n_bins + 1)
+    conf_pts, acc_pts, counts = [], [], []
+    for i in range(n_bins):
+        lo, hi = edges[i], edges[i + 1]
+        mask = (confidence > lo) & (confidence <= hi)
+        if mask.sum():
+            conf_pts.append(float(confidence[mask].mean()))
+            acc_pts.append(float(correct[mask].mean()))
+            counts.append(int(mask.sum()))
+    return np.array(conf_pts), np.array(acc_pts), np.array(counts)
+
+
+def multiclass_log_loss(y_true, proba, classes) -> float:
+    return float(log_loss(y_true, proba, labels=list(classes)))