[spark] Guard lake split partition pruning against malformed splits - #3836
Open
Yohahaha wants to merge 2 commits into
Open
[spark] Guard lake split partition pruning against malformed splits#3836Yohahaha wants to merge 2 commits into
Yohahaha wants to merge 2 commits into
Conversation
…ion keys SparkPartitionPredicate.matchesPartition previously admitted any split with empty or partial partition values when a partition predicate was pushed, because the scan builder removes the partition predicate from the post-scan filters handed back to Spark. Such splits were never re-filtered, so rows from non-matching partitions could leak into the query result. A longer value tuple even crashed with an unreadable IndexOutOfBoundsException. Now validate that the split reports exactly one value per partition key and fail fast with a descriptive IllegalArgumentException otherwise, since the arity mismatch means the lake plugin broke the LakeSplit#partition contract. Add SparkPartitionPredicateTest covering extraction and matching, including guard tests that fail on the pre-fix behavior.
The new SparkPartitionPredicate tests and the existing SparkPredicateConverter tests are both unit tests for predicate utilities in the same package and share the same Spark DSv2 predicate helpers. Consolidate them into a single SparkPredicateUtilsTest suite so that ref/lit/pred helpers are not duplicated and the test surface is easier to maintain.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contribution Checklist
The pull request title follows the format "[component] Title of the pull request".
The change addresses a single issue/bug.
Automated tests were run and the relevant test suite passed.
This PR corresponds to a GitHub issue (issue number to be added before final review).
Generative AI disclosure:
Purpose
Linked issue: N/A
Harden partition pruning for Spark lake splits so that a split whose reported partition values do not match the table's partition key arity is rejected instead of being admitted silently. This prevents rows from non-matching partitions from leaking into the query result, and avoids an unreadable
IndexOutOfBoundsExceptionwhen the value tuple is too long.Also consolidate the predicate utility tests to remove duplicated Spark DSv2 predicate helpers.
Brief change log
SparkPartitionPredicate.matchesPartitionnow validates that the partition value tuple size equals the number of partition keys, and throws a clearIllegalArgumentExceptionon mismatch.SparkPredicateConverterTestandSparkPartitionPredicateTestinto a singleSparkPredicateUtilsTestsuite.Tests
fluss-spark/fluss-spark-ut/src/test/scala/org/apache/fluss/spark/utils/SparkPredicateUtilsTest(44 tests, all passed).API and Format
No public API or storage format changes.
Documentation
No new feature or user-facing documentation change.