feat: materialized query snapshots - #1238
Conversation
Adds an opt-in "Materialize Results" mode for a query: its result is computed once in a background job and stored as a table in the DuckDB warehouse, and charts/downstream queries read the stored result instead of re-running the query live on every view. Because the stored artifact is the query *result* (already aggregated), each refresh recomputes and replaces it wholesale — there is no incremental state to corrupt, so source updates/deletes can't leave stale rows behind. - New Insights Query v3 fields: is_materialized, snapshot_refresh_frequency and read-only snapshot state (status, last_refreshed_at, row_count, error, digest). - Read path: build() swaps the operation chain for a snapshot read when materialized; dashboard filters still apply on top, previews and refreshes build live. A failed refresh keeps serving the old snapshot (never falls back to a live full scan under load). - Refresh honours the per-query frequency via an hourly scheduler; editing the operations or toggling the flag re-syncs the snapshot on save. - Stock template opts its two Stock Ledger-backed queries into daily materialization. Verified end-to-end against demo data and covered by tests.
Query editor gains a "Materialize Results" toggle with a refresh-frequency select, a manual refresh button and a freshness line. Dashboard charts backed by a materialized query show an unobtrusive "as of <time>" badge so a stored snapshot never reads as live data.
|
Tick the box to add this pull request to the merge queue (same as
|
Confidence Score: 3/5Merge with caution: a lost background job can leave a query permanently stuck in Queued status, silently stopping scheduled refreshes, plus two open issues from prior threads (DuckDB timeout no-op, frontend save-before-refresh race) remain unaddressed. The core read/write path and permission model are sound, but three independent defects affect production reliability: the stuck-Queued scheduler dead zone (new), the silent DuckDB timeout no-op (prior thread), and the unsaved-state race on manual refresh (prior thread). snapshots.py (refresh_due_snapshots stuck-Queued logic, _disable_source_timeout DuckDB no-op); frontend query.ts (refreshSnapshot save-before-call race) Reviews (6): Last reviewed commit: "fix(src2): anchor template preview image..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
frappe.scrub folds distinct docnames (hyphen vs underscore) onto the same DuckDB table, so one query's refresh could overwrite another's snapshot and drop_snapshot could drop both. The docname is already a unique system hash and DuckDB tolerates it as an identifier, so use it directly.
Two CI fixes: - Snapshot refresh was forced to run inline in tests (now or in_test), so importing the stock template (a materialized query) executed the query against tabStock Ledger Entry — absent on a bare test site — and broke the import. The refresh is a background job; a save/import must never run or fail on it. Callers needing a synchronous refresh call refresh_snapshot() directly. - Wrap the two user-facing frappe.throw messages in frappe._() for the linter.
Codecov Report❌ Patch coverage is ❌ Your patch status has failed because the patch coverage (84.61%) is below the target coverage (85.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## develop #1238 +/- ##
===========================================
+ Coverage 47.91% 48.96% +1.05%
===========================================
Files 76 77 +1
Lines 5998 6165 +167
===========================================
+ Hits 2874 3019 +145
- Misses 3124 3146 +22
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Materialization is an infrastructure setting, so surface it behind the query toolbar's More menu (admin-only) in its own dialog instead of the inline Details panel. Enforce admin server-side too: refresh_snapshot and toggling is_materialized now require an Insights admin.
A materialized query stores its result in the DuckDB warehouse, so joining it with live tables spans two backends and ibis aborts with a bare "Multiple backends found for this expression". Two fixes: - Stock template: derive the ageing report from the Stock Ledger snapshot instead of joining live Bin. Bin.actual_qty / stock_value are by definition the SLE sums per item+warehouse, so the enriched last-movement snapshot carries everything ageing needs and the join disappears. As a bonus the quantities are now consistent with the snapshot date instead of mixing live Bin values with a stored date. - Translate the opaque multi-backend ibis error into an actionable one that names what got mixed (stored vs live, or two live data sources) and points to the data store. Covers cross-data-source live joins too, not just snapshots.
| ) | ||
| for q in queries: | ||
| if q.snapshot_status in ("Queued", "Running"): | ||
| continue | ||
| if _is_snapshot_due(q, now): | ||
| enqueue_snapshot_refresh(q.name) |
There was a problem hiding this comment.
refresh_due_snapshots permanently skips queries with snapshot_status in ('Queued', 'Running'). If a job is lost (worker restart, enqueue failure after db_set) the status stays "Queued" forever and the hourly scheduler never retries — the snapshot silently stops refreshing until an admin manually clicks "Refresh Now". Adding a staleness threshold to reset long-stuck Queued/Running statuses would close this gap.
2 participants
Why
The shipped ERPNext templates run their charts live against the site's OLTP database. On a large site the heavy aggregated queries (stock movement over the full Stock Ledger, AR/AP over GL) scan tens of millions of rows on every dashboard view. Lowering the timeout and slimming the queries helps, but the real fix for the heaviest ones is to stop running them live.
What
An opt-in Materialize Results mode for a query. The query's result is computed once in a background job and stored as a table in the DuckDB warehouse; charts and downstream queries then read the stored result instead of re-running the query live.
The key design choice is that we store the query result — already aggregated to its final grain — not the source rows. So each refresh recomputes and replaces the whole table, and there is no incremental sync state that source updates or deletes could corrupt. The trade-off is staleness, surfaced honestly as an "as of <time>" label.
Notes for reviewers
IbisQueryBuilder: a materialized query swaps its operation chain for a single snapshot read, and dashboard filters still apply on top (they target the query's output columns, which the snapshot preserves). Previews (active_operation_idx) and refreshes build live viaresolve_snapshot=False.Testing
Backend covered by
test_query_snapshots.py(refresh, snapshot read, preview bypass, adhoc filters, failed-refresh survival, toggle-off cleanup, row cap, scheduler due-logic) and verified end-to-end against demo ERPNext data. Frontend is type-checked; the dashboard badge/query-editor panel were not browser-verified.🤖 Generated with Claude Code