Enterprise Data Science & MLOps Preprocessing Framework

A production-ready data science reference blueprint establishing a standardized, highly rigorous workflow for Exploratory Data Analysis (EDA), statistical data sanitization, feature engineering, and real-time model serving.

This framework scales seamless analytical workflows into a cloud-ready, high-performance FastAPI microservice protected by explicit structural request schema contracts.

System & End-to-End MLOps Architecture

[Raw Ingestion] ──> [Sanitization (IQR)] ──> [Statistical EDA] ──> [Visual Telemetry]
                                                                        │
[Live HTTP POST] ──> [FastAPI Server Core] ──> [Predictive Inference] ◄── [Serialized Model Assets]

Repository Architecture & Project Lifecycle

The codebase is mapped into a predictable, modular timeline layout that isolates experimental research domains from production-serving environments:

• 01_Data_Loading/ : Standardizes tabular data ingestion vectors, structural shapes, memory footprints, and early schema label sanity gates.
• 02_Data_Cleaning/ : Handles primitive data type casting, vectorized string trimming, missingness metrics, and mathematical outlier removal via Interquartile Range (IQR) fencing.
• 03_Exploratory_Data_Analysis/ : Calculates continuous descriptive matrices, checks nominal class imbalances, and tracks structural multi-collinearity via Pearson correlation tracking.
• 04_Visualization/ : Transforms complex numerical statistical tables into high-resolution visual layouts using Seaborn masked heatmaps, kernel density charts, and pairwise distribution grids.
• 05_Regression/ : Separates feature spaces, trains an enterprise predictive Linear Regression machine learning engine, extracts optimization feature weights, and freezes model states.
• Models/ : The immutable serialization storage vault hosting the frozen mathematical states of the trained model engine (.pkl), data scaling vectors, and categorical schemas.
• Api/ : A production-ready, high-throughput backend application layer exposing stateless RESTful endpoints for real-time model inference.

---

Technology Ecosystem & Dependencies

• Core Environment Tools: Virtualenv (venv), Python 3.10+
• Data Engineering Matrix: Pandas, NumPy
• Visual Telemetry Engines: Matplotlib, Seaborn
• Statistical ML Frameworks: Scikit-Learn (Pipelines, Estimators, Metrics)
• Production Infrastructure: FastAPI Core, Pydantic (Data Validation Enforcer), Uvicorn Engine

---

Local Installation & Execution Strategy

1. Environment Initialization & Dependency Isolation

Navigate into your local workspace directory and establish a clean, virtual execution boundary to avoid global library fragmentation:

# Generate the virtual environment footprint
python -m venv .venv

# Activate the isolated environment gate
# (For macOS/Linux)
source .venv/bin/activate
# (For Windows PowerShell)
.venv\Scripts\Activate.ps1

# Upgrade package manager and install frozen requirements matrix
python -m pip install --upgrade pip
pip install -r requirements.txt

2. Operationalizing the Inference Microservice

Once the virtual requirements are successfully installed and your notebooks have compiled and saved the models into the Models/ folder, steer your console into the application module to launch the server:

cd Api
uvicorn main:app --reload

3. Interactive Contract Verification

FastAPI automatically parses the operational Pydantic code logic into live, interactive web documentation footprints. Open your browser and navigate to:

👉 http://127.0.0.1:8000/docs

Use the interactive Swagger UI payload simulator to submit high-density sample property vectors:

{
  "Size_sqft": 1500.0,
  "Bedrooms": 3,
  "Age_years": 4.5,
  "Location": "Urban"
}

The server will automatically map the input data categories, scale the numerical variables, run the inference parameters against hot RAM cache layers, and instantly respond with an explicit HTTP 200 JSON valuation payload:

{
  "Inference_Telemetry": {
    "Predicted_Valuation_Lakhs": 122.1272,
    "Units": "INR Lakhs",
    "Input_Processed": {
      "Size_sqft": 1500.0,
      "Bedrooms": 3,
      "Age_years": 4.5,
      "Location": "Urban"
    }
  }
}

---

Defensive Engineering Standards

• Zero Ingestion Leakage: Always works on isolated memory deep-copies to preserve raw, original source dataset integrity.
• Dynamic Category Handling: Category vectors are transformed live using serialized OneHotEncoder parameters to protect the system against structural column shape mismatches.
• Input Guardrails: Pydantic schema constraints automatically block invalid structural input models (e.g., negative room properties or incorrect type bounds) at the API gate before they ever touch the machine learning layers.