[spark] Support time-range incremental batch reads - #3883
Open
Yohahaha wants to merge 2 commits into
Open
Conversation
Add timestamp-bounded batch reads to the Spark connector so downstream pipelines can incrementally read rows written within a [t1, t2) window: - scan.startup.mode=timestamp + scan.startup.timestamp (inclusive start) - scan.bounded.mode=timestamp + scan.bounded.timestamp (exclusive end, defaults to latest committed data at planning time) - Log tables return raw records in the window; primary key tables return keys inserted/updated in the window folded to their latest value - Out-of-range start fails fast by default; scan.startup.timestamp.out-of-range=adjust clamps to earliest retained data - Default behavior unchanged (scan.startup.mode=full)
…-range tests Blank scan.incremental.* values now count as unset, so a whitespace-only start timestamp no longer enables an incremental read. Test cleanups: merge the redundant datetime-expression TVF case into the timestamp arguments case, drop the future-end case (server-side validation), slim the retention-guard message test, and replace the weak option-scoping case with a session-configuration negative test. Co-Authored-By: Qoder <noreply@qoder.com> AI-Model: Qoder Auto AI-Contributed/Feature: 7/7 AI-Contributed/UT: 81/81
Yohahaha
force-pushed
the
spark/time-range-incremental-batch-read
branch
from
August 6, 2026 15:28
80165a8 to
be893fe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Support time-range incremental batch reads in the Spark connector, so pipelines can read only the data written within a
[start, end)window (start inclusive, end exclusive).New options (per-query read options, only read from scan options / TVF args — never from session configuration, so a window can't leak into later reads; batch-only, streaming ignores them):
scan.incremental.start.timestampyyyy-MM-dd HH:mm:ssin the Spark session time zone.scan.incremental.end.timestamplatestlateststops at the latest committed data captured at planning time.scan.incremental.timestamp.out-of-rangeerrortable.log.ttl):errorfails fast (default),adjustclamps to the earliest retained offset.New TVF
fluss_incremental_between_timestamp(table, start[, end])for pure SQL — sugar over the options above, registered viaFlussSparkSessionExtensions; arguments accept string/integral/TIMESTAMPconstant expressions (e.g. rollingpast hourwindows computed in SQL). Note: unlike Paimon's similarly named function, the window is start-inclusive/end-exclusive.Read semantics per table type:
Also:
scan.startup.modeis clarified to affect streaming reads only; plain batch reads remain full-table regardless, so default behavior is unchanged.[start, stop)buckets emit no partitions.website/docs/engine-spark/reads.md(time-range batch read section) andoptions.md.Fixes #3842
Test Plan
FlussOffsetInitializersTestSparkTimeRangeTvfTest,SparkLakeTimeRangeReadTestmvn spotless:checkpasses on affected modules🤖 AI-assisted changes - reviewed by human developer