Skip to content

[spark] Support time-range incremental batch reads - #3883

Open
Yohahaha wants to merge 2 commits into
apache:mainfrom
Yohahaha:spark/time-range-incremental-batch-read
Open

[spark] Support time-range incremental batch reads#3883
Yohahaha wants to merge 2 commits into
apache:mainfrom
Yohahaha:spark/time-range-incremental-batch-read

Conversation

@Yohahaha

@Yohahaha Yohahaha commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Support time-range incremental batch reads in the Spark connector, so pipelines can read only the data written within a [start, end) window (start inclusive, end exclusive).

New options (per-query read options, only read from scan options / TVF args — never from session configuration, so a window can't leak into later reads; batch-only, streaming ignores them):

Option Default Meaning
scan.incremental.start.timestamp (none) Enables the incremental read; inclusive lower bound. Epoch millis or yyyy-MM-dd HH:mm:ss in the Spark session time zone.
scan.incremental.end.timestamp latest Exclusive upper bound; latest stops at the latest committed data captured at planning time.
scan.incremental.timestamp.out-of-range error If the start predates the earliest data retained by Fluss (bounded by table.log.ttl): error fails fast (default), adjust clamps to the earliest retained offset.

New TVF fluss_incremental_between_timestamp(table, start[, end]) for pure SQL — sugar over the options above, registered via FlussSparkSessionExtensions; arguments accept string/integral/TIMESTAMP constant expressions (e.g. rolling past hour windows computed in SQL). Note: unlike Paimon's similarly named function, the window is start-inclusive/end-exclusive.

Read semantics per table type:

  • Log table: raw records appended within the window.
  • Primary key table: keys inserted/updated in the window, folded to their latest value as of the window end (reads only the changelog range, no kv snapshot; keys deleted in the window are excluded).
  • Lake-enabled table: same as above, but always reads from Fluss only — an incremental read never unions the lake snapshot.

Also:

  • scan.startup.mode is clarified to affect streaming reads only; plain batch reads remain full-table regardless, so default behavior is unchanged.
  • Out-of-range end handling: an end timestamp in the future is rejected by the server; empty [start, stop) buckets emit no partitions.
  • Docs: website/docs/engine-spark/reads.md (time-range batch read section) and options.md.

Fixes #3842

Test Plan

  • New unit tests: FlussOffsetInitializersTest
  • New integration tests: SparkTimeRangeTvfTest, SparkLakeTimeRangeReadTest
  • mvn spotless:check passes on affected modules

🤖 AI-assisted changes - reviewed by human developer

Yohahaha and others added 2 commits August 6, 2026 23:28
Add timestamp-bounded batch reads to the Spark connector so downstream
pipelines can incrementally read rows written within a [t1, t2) window:

- scan.startup.mode=timestamp + scan.startup.timestamp (inclusive start)
- scan.bounded.mode=timestamp + scan.bounded.timestamp (exclusive end,
  defaults to latest committed data at planning time)
- Log tables return raw records in the window; primary key tables return
  keys inserted/updated in the window folded to their latest value
- Out-of-range start fails fast by default;
  scan.startup.timestamp.out-of-range=adjust clamps to earliest retained data
- Default behavior unchanged (scan.startup.mode=full)
…-range tests

Blank scan.incremental.* values now count as unset, so a whitespace-only
start timestamp no longer enables an incremental read. Test cleanups:
merge the redundant datetime-expression TVF case into the timestamp
arguments case, drop the future-end case (server-side validation), slim
the retention-guard message test, and replace the weak option-scoping
case with a session-configuration negative test.

Co-Authored-By: Qoder <noreply@qoder.com>
AI-Model: Qoder Auto
AI-Contributed/Feature: 7/7
AI-Contributed/UT: 81/81
@Yohahaha
Yohahaha force-pushed the spark/time-range-incremental-batch-read branch from 80165a8 to be893fe Compare August 6, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Spark] Support time-range (incremental) batch reads for log and primary key tables

1 participant