Click the banner to view the full analysis report

End-to-End Data Science Project (Linux + MySQL + Python)

📌 Overview

This project demonstrates a full-stack data science workflow entirely on the Ubuntu Linux command line.

Instead of relying solely on notebooks, I built an automated pipeline that:

1. Ingests raw CSV data using Linux CLI tools (awk, sed).
2. Cleans & Normalizes data using Python scripts.
3. Loads structured data into a MySQL database.
4. Analyzes business KPIs using complex SQL queries.
5. Visualizes results using matplotlib.

📂 Project Structure

linux-data-science-project/
│
├── data/
│   ├── raw/                  # Original CSV file
│   └── processed/            # (Generated) Cleaned artifacts
│
├── scripts/
│   ├── 01_setup_env.sh       # Virtual env automation
│   ├── 02_etl_to_mysql.ipynb    # Python ETL (CSV -> MySQL)
│   
│
├── notebooks/
│   └── 03_analysis.ipynb        # Generates charts from SQL data
│
├── sql/
│   ├── schema.sql            # Database creation scripts
│   ├── queries.sql           # 10+ Business analytical queries
│
├── output_plots/             # Generated visualizations
├── LINUX_COMMANDS.md         # Documentation of CLI data exploration
├── SQL_SCENARIOS.md          # Business questions & SQL results
├── Makefile                  # Build automation commands
├── README.md                 # Project documentation

🛠 Architecture & Skills Demonstrated

Component	Tools Used	Skills Demonstrated
Data Exploration	Linux Terminal (grep, awk, wc)	CLI proficiency, stream processing
ETL Pipeline	Python (pandas, sqlalchemy)	Data cleaning, database connectors, automation
Database	MySQL	Schema design, relational modeling
Analytics	SQL	Window functions, aggregations, subqueries
Automation	GNU Make (Makefile)	Build automation, reproducible workflows

🚀 How to Run

1. Prerequisites

   • Ubuntu/Linux OS (or WSL)

   • MySQL Server installed and running

   • Python 3.8+

2. Setup
   1. Create a MySQL database named superstore_db:

   CREATE DATABASE superstore_db;

   2. Update database credentials in scripts/02_etl_to_mysql.ipynb:

   DB_USER = 'your_username'
   DB_PASS = 'your_password'

   3. Run the Pipeline I have set up a Makefile to automate the entire process. Simply run:

   # Sets up environment, cleans data, loads DB, and runs analysis
   make pipeline

📊 Key Insights

Full analysis can be found in SQL_SCENARIOS.md.

• Top Profit Center: Technology category yields the highest profit margins (17%).

• Shipping Efficiency: "Standard Class" shipping averages 5.0 days vs 0.04 days for "Same Day".

• Seasonality: Sales volume consistently spikes by 30% in November/December.

📄 Documentation

• Linux Command Logs: How I explored the data using only the terminal.

• SQL Business Scenarios: Detailed breakdown of 10 business questions and queries.

Citation

If you use this project in academic research, publications, educational materials, or derivative works, please cite the project.

This repository includes a CITATION.cff file, so GitHub provides a "Cite this repository" button in the repository sidebar. You can use it to obtain citations in BibTeX, APA, and other supported formats.

Suggested citation:

Arain, S. U. R. (2026). End-to-End-Data-Science-Pipeline-Linux-Python-MySQL (Version 1.0) [Software]. https://github.com/sanaurrehmanarain/End-to-End-Data-Science-Pipeline-Linux-Python-MySQL

Author: Sana Ur Rehman Arain

Profession: Data Scientist

GitHub: https://github.com/sanaurrehmanarain

Contact: sana.arain.work@gmail.com

If you build upon this work, attribution is appreciated and helps others discover the original project.

Note: The MIT License requires that the original copyright notice be retained in copies of the Software.

---

License

This project is licensed under the MIT License. See the LICENSE file for details.