Skip to content

Fix sentiment 502 and cut per-request latency across slow endpoints - #2

Open
nabindev3 wants to merge 1 commit into
mainfrom
fix/render-cold-start-502
Open

Fix sentiment 502 and cut per-request latency across slow endpoints#2
nabindev3 wants to merge 1 commit into
mainfrom
fix/render-cold-start-502

Conversation

@nabindev3

Copy link
Copy Markdown
Owner

The /api/v1/sentiment/analyse endpoint was returning 502 Bad Gateway from Render's proxy because the engine could spend up to ~400s per request in the worst case. Several other endpoints had silent slowness from re-reading CSVs and rewriting cache files on every request.

Sentiment engine (src/hf_sentiment_engine.py, src/sentiment_engine.py):

  • Replace per-request disk cache (full JSON read + full JSON rewrite on every call) with a thread-safe in-memory LRU. Disk cache was O(N) I/O per request and burned Render's ephemeral disk.
  • Cap HFInferenceClient TIMEOUT 25s->8s, RETRY_ATTEMPTS 3->1, COLD_START_WAIT 20s->3s, and disable wait_for_model so HF returns 503 fast on a cold model instead of blocking up to 20s.
  • Fan out the 3 HF model calls (sentiment, irony, zero-shot aspect) concurrently with a ThreadPoolExecutor. Latency is now bounded by the slowest of the three, not their sum.
  • Drop the 0.3-0.5s sleep between batch items; replace sequential batch with bounded concurrent execution.
  • Don't cache failure results so the next request can retry the warm engine.
  • Include engine identifier in cache key so a TextBlob fallback doesn't poison later warm-engine lookups.

Sentiment router (backend/routers/sentiment.py):

  • Hard endpoint deadline (12s single, scaled+capped 45s batch). If the underlying engine doesn't return in time we serve a TextBlob result instead of letting the request hang until Render's proxy emits 502.

Hot-path I/O caching:

  • backend/routers/forecast.py: add @lru_cache to _load_daily and _load_ext (were re-parsing 2192-row + 6-year CSVs on every /forecast call).
  • backend/routers/pricing.py: same fix for daily_kpis and external_regs.
  • backend/routers/xai.py: cache the 180k-row bookings DataFrame used by /global-importance instead of re-reading it per request.

All 25 existing tests pass; suite runtime dropped from 5.0s to 3.9s.

The /api/v1/sentiment/analyse endpoint was returning 502 Bad Gateway from
Render's proxy because the engine could spend up to ~400s per request in
the worst case. Several other endpoints had silent slowness from
re-reading CSVs and rewriting cache files on every request.

Sentiment engine (src/hf_sentiment_engine.py, src/sentiment_engine.py):
- Replace per-request disk cache (full JSON read + full JSON rewrite on
  every call) with a thread-safe in-memory LRU. Disk cache was O(N) I/O
  per request and burned Render's ephemeral disk.
- Cap HFInferenceClient TIMEOUT 25s->8s, RETRY_ATTEMPTS 3->1,
  COLD_START_WAIT 20s->3s, and disable wait_for_model so HF returns 503
  fast on a cold model instead of blocking up to 20s.
- Fan out the 3 HF model calls (sentiment, irony, zero-shot aspect)
  concurrently with a ThreadPoolExecutor. Latency is now bounded by the
  slowest of the three, not their sum.
- Drop the 0.3-0.5s sleep between batch items; replace sequential batch
  with bounded concurrent execution.
- Don't cache failure results so the next request can retry the warm engine.
- Include engine identifier in cache key so a TextBlob fallback doesn't
  poison later warm-engine lookups.

Sentiment router (backend/routers/sentiment.py):
- Hard endpoint deadline (12s single, scaled+capped 45s batch). If the
  underlying engine doesn't return in time we serve a TextBlob result
  instead of letting the request hang until Render's proxy emits 502.

Hot-path I/O caching:
- backend/routers/forecast.py: add @lru_cache to _load_daily and _load_ext
  (were re-parsing 2192-row + 6-year CSVs on every /forecast call).
- backend/routers/pricing.py: same fix for daily_kpis and external_regs.
- backend/routers/xai.py: cache the 180k-row bookings DataFrame used by
  /global-importance instead of re-reading it per request.

All 25 existing tests pass; suite runtime dropped from 5.0s to 3.9s.
nabindev3 added a commit that referenced this pull request Jun 13, 2026
- api.cached_get(): short-lived (API_CACHE_TTL, default 60s) cache keyed on
  path+params, so reruns from slider moves / tab switches don't re-fire the same
  GETs. Sidebar KPIs/health/data-quality and every tab's data GET now use it;
  get_schema uses it too (front-end review #1, #3).
- The sidebar no longer pings http://localhost:5001 on every render (#2): the
  MLflow status row is gated behind MLFLOW_UI_URL (off by default) and its health
  check is cached 30s, so cloud deploys don't eat a 2s blocking timeout per load.
- Dropped the hardcoded localhost:8501 dashboard link (#18); /health 'degraded'
  now counts as up.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant