A causal inference pipeline for evaluating dynamic surge pricing policies using NYC taxi trip data.
This project investigates how surge pricing affects ride availability across New York City taxi zones while accounting for the fact that surge pricing is not assigned randomly. Rather than treating surge as an independent variable, the project models the probability that a particular surge level would have been assigned under existing market conditions and uses Inverse Probability Weighting (IPW) to estimate less biased treatment effects.
The repository combines feature engineering, propensity score modeling, simulation, and visualization to analyze how alternative surge pricing policies influence ride availability across space and time.
Dynamic pricing systems are inherently observational: surge pricing is introduced because demand already exceeds supply. This creates selection bias when estimating the effect of surge on marketplace outcomes.
This project addresses that challenge by:
- engineering demand and supply features from NYC taxi trip records
- constructing a treatment assignment model for surge pricing
- estimating propensity scores
- applying inverse probability weighting
- comparing naïve and IPW-adjusted models
- simulating zone-level marketplace outcomes
NYC TLC Trip Data
│
▼
Feature Engineering
│
├── Demand / Supply Ratio
├── Lag Features
├── Rolling Statistics
└── Time Features
│
▼
Synthetic Surge Assignment
│
▼
Propensity Score Model
│
▼
Inverse Probability Weights
│
▼
Weighted Outcome Models
│
▼
Policy Simulation
│
▼
Zone-Level Analysis
surge-pricing-simulator
│
├── data/
│ ├── taxi_zone_lookup.csv
│ └── taxi_zones/
│
├── notebooks/
│ ├── 01_surge_eda.ipynb
│ ├── 03_train_ipw.ipynb
│ └── 05_post_exporation.ipynb
│
├── outputs/
│ ├── simulation.csv
│ ├── zone_summary.csv
│ └── taxi_zone_lookup.csv
│
├── scripts/
│ ├── build_features_table.py
│ ├── features_lag.py
│ ├── init_db.py
│ ├── parquet_to_csv.py
│ └── shp_to_parquet.py
│
├── sql/
│ └── feature_lags.sql
│
├── requirements.txt
└── README.md
The feature pipeline derives marketplace state from raw trip records using 15-minute time windows.
Features include:
- demand counts
- supply counts
- demand-to-supply ratio
- lagged demand/supply ratios
- rolling historical averages
- hour of day
- weekend indicator
These variables characterize market conditions prior to surge assignment.
To emulate a production pricing system, surge multipliers are assigned as a function of the demand-to-supply ratio with added stochastic variation.
This creates an observational dataset where surge is correlated with market conditions, introducing selection bias similar to real-world ride-sharing platforms.
Rather than directly modeling outcomes from observed surge values, the project estimates the probability that each observation received its assigned surge level.
A multinomial logistic regression predicts treatment assignment using pre-treatment covariates.
Inverse probability weights are then computed as
and incorporated into weighted logistic regression models for downstream outcome estimation.
The repository includes a comparison between naïve and IPW-adjusted surge coefficients to illustrate the effect of correcting for non-random treatment assignment.
The simulation generates aggregated metrics including:
Zone-level simulation summaries including:
- median lift
- 5th percentile lift
- 95th percentile lift
- most common surge multiplier
- surge variability
Per-zone statistics such as:
- average surge
- average rides lift
- number of simulated windows
These outputs are intended for downstream visualization and policy analysis.
The repository includes a collection of visualizations that summarize marketplace dynamics, rider and driver behavior, and simulation outcomes. All figures are located in the dashboard/ directory.
A choropleth map showing the average simulated surge multiplier across NYC taxi zones. The visualization highlights geographic patterns in pricing, identifying areas that consistently experience higher demand relative to available supply.
Displays how the average demand-to-supply ratio changes throughout the day. This chart illustrates recurring marketplace imbalances and identifies peak periods where surge pricing is most likely to be triggered.
Compares modeled rider cancellation probability and driver acceptance probability across surge levels. The visualization highlights the central trade-off in dynamic pricing: higher surge attracts additional drivers while simultaneously increasing rider attrition.
Ranks the twenty taxi zones with the largest simulated increase in completed rides under the evaluated surge pricing policy. This provides a quick comparison of where dynamic pricing delivers the greatest operational benefit.
Shows the distribution of simulated ride lift across taxi zones using median estimates and uncertainty intervals. This visualization communicates not only the expected policy impact but also the variability of those estimates across repeated simulation runs.
- Python
- Pandas
- NumPy
- DuckDB
- Scikit-learn
- SciPy
- PuLP
- PyArrow
The project uses publicly available NYC Taxi & Limousine Commission trip data that can be found at https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page
Large raw datasets and generated model artifacts are not included in the repository due to size constraints.
Potential extensions include:
- replacing heuristic surge assignment with learned pricing policies
- evaluating additional causal estimators (matching, doubly robust estimation, causal forests)
- incorporating weather and special event data
- reinforcement learning for adaptive pricing
- agent-based marketplace simulation
This repository is intended for academic and research use. Ensure that you have the necessary rights and approvals for all datasets and API services used with it.