A discrete-event simulator for SLO-aware autoscaling of multi-instance LLM serving. Compares four autoscaling policies across five workload patterns and four instance startup delays.
For full architecture, design decisions, and methodology see design.md.
How many instances do you need to guarantee p99 TTFT under 500ms?
The answer depends on:
- the workload pattern (steady, burst, spike)
- the autoscaling policy (reactive, predictive, conservative)
- the instance startup delay (5s to 60s)
This simulator measures the tradeoff between SLO compliance and cost.
reactive Scale up when utilization or TTFT exceeds threshold. Arrives late to bursts. Consistently the worst dynamic policy.
predictive_headroom Predict demand from recent trends. Add explicit headroom. Best policy when startup delay is long (30s+).
conservative Always maintain headroom above current need. Simple and effective. Best when startup delay is short (5-15s).
fixed_4 Fixed 4 instances. No scaling. Baseline for cost comparison.
Overall by policy:
conservative: SLO_viol=0.106 TTFT_p99=17893ms inst=3.3 predictive_headroom: SLO_viol=0.104 TTFT_p99=16820ms inst=3.7 reactive: SLO_viol=0.139 TTFT_p99=17686ms inst=4.5 fixed_4: SLO_viol=0.227 TTFT_p99=29206ms inst=3.9
By startup delay:
startup=5s: conservative: SLO=0.050 inst=3.4 <- BEST predictive_headroom: SLO=0.066 inst=3.8 reactive: SLO=0.101 inst=4.5
startup=15s: conservative: SLO=0.086 inst=3.3 <- BEST predictive_headroom: SLO=0.087 inst=3.7 reactive: SLO=0.126 inst=4.5
startup=30s: predictive_headroom: SLO=0.103 inst=3.7 <- BEST conservative: SLO=0.111 inst=3.3 reactive: SLO=0.147 inst=4.6
startup=60s: predictive_headroom: SLO=0.159 inst=3.5 <- BEST conservative: SLO=0.176 inst=3.1 reactive: SLO=0.181 inst=4.5
Startup delay is the dominant factor for SLO compliance. The same policy achieves SLO_viol=0.050 at 5s startup but 0.176 at 60s. Reducing startup delay matters more than improving autoscaling logic.
Conservative is best when startup is fast. At 5s and 15s startup delay, simple headroom absorbs bursts effectively because new instances arrive before queues build up.
Predictive + headroom is best when startup is slow. At 30s and 60s, trend detection gives the scaler a head start. Combined with headroom, it handles slow startup better than conservative.
Reactive scaling is consistently the worst dynamic policy. It responds after SLO violations have already occurred. With non-trivial startup delay, the damage accumulates during the wait.
Pure prediction without headroom fails. Forecasting average demand is not enough to absorb bursts. Explicit headroom is necessary regardless of prediction quality.
The practical lesson is operational, not algorithmic. Invest in fast instance startup (warm pools, cached weights, pre-warmed containers) before investing in sophisticated autoscaling algorithms.
slo-aware-autoscaling-sim/ ├── src/ │ ├── init.py │ ├── config.py │ ├── workload.py │ ├── autoscaler.py │ ├── simulator.py │ ├── metrics.py │ └── analysis.py ├── results/ ├── plots/ ├── LICENSE ├── design.md ├── README.md ├── requirements.txt └── run.py
python3 -m venv venv source venv/bin/activate pip install -r requirements.txt python run.py
Outputs:
results/sweep_summary.csv results/summary.txt plots/slo_vs_startup.png plots/slo_by_pattern.png plots/slo_vs_cost.png plots/startup_heatmap.png
This project closes the cluster-level view of LLM serving:
llm-inference-scheduler: which request runs inside one instance disaggregated-prefill-decode-sim: how to split phases across nodes request-routing-sim: how to distribute requests across instances slo-aware-autoscaling-sim: how many instances to run
MIT License. See LICENSE for details.
Joao Felipe De Souza 2026